A/B testing is dead — Part 1: Audience-Content Heatmap
I claimed it before, A/B testing is dead. Why?
- It’s too long and most likely makes the conversion rate drop during the test
- It takes more time and resources than it brings value
- It’s inaccurate and biased
(Read: Don’t accept these results from a testing or conversion optimization campaign)
I was challenged quite a bit on that claim(fair enough) so I decided to start a series on why I believe.
The notion of “content-audience matrix”
Let’s look at real-life examples (Cauzal customer)
This customer has 20 variations of content for multiple elements on many pages (CTA, H1, Links, etc…). I’m only looking here at 2 variants of the main CTA: “Start for free” vs “Start Now”
From the visitors, we’re picking up 18 predictive signals (behavior, demographics, referral, etc..).
After running for 4 days, here is the overall performance
Easy! “Start Now” is better so let’s turn off “Start for free” and we’re good to go…
Not so fast…let’s look at more sensitivity reports…
Now when we look at more data, it’s not that simple… People don't think uniformly.
There are a lot of nuances in audiences. A/B testing is making marketers make a decision based on a snapshot…This is wrong and not necessary. There are ways to provide relevant content at a visitor level. Without losing any conversions in the process.
Overall, an element might be underperforming, but the data showing that 30% of your visitors want to see that is enough to justify the shift to smart content optimization. Why pick. Let a smart engine personalize the experience of all your visitors.
Interested in getting your site’s sensitivity matrix for free? Just comment here!
