A/B testing is dead — Part 1: Audience-Content Heatmap

I claimed it before, A/B testing is dead. Why?

  • It’s too long and most likely makes the conversion rate drop during the test
  • It takes more time and resources than it brings value
  • It’s inaccurate and biased

(Read: Don’t accept these results from a testing or conversion optimization campaign)

I was challenged quite a bit on that claim(fair enough) so I decided to start a series on why I believe.

The notion of “content-audience matrix”

Let’s look at real-life examples (Cauzal customer)

This customer has 20 variations of content for multiple elements on many pages (CTA, H1, Links, etc…). I’m only looking here at 2 variants of the main CTA: “Start for free” vs “Start Now”

From the visitors, we’re picking up 18 predictive signals (behavior, demographics, referral, etc..).

Image for post
Image for post
Some of the visitor's data

After running for 4 days, here is the overall performance

Image for post
Image for post

Easy! “Start Now” is better so let’s turn off “Start for free” and we’re good to go…

Not so fast…let’s look at more sensitivity reports…

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

Now when we look at more data, it’s not that simple… People don't think uniformly.

There are a lot of nuances in audiences. A/B testing is making marketers make a decision based on a snapshot…This is wrong and not necessary. There are ways to provide relevant content at a visitor level. Without losing any conversions in the process.

Overall, an element might be underperforming, but the data showing that 30% of your visitors want to see that is enough to justify the shift to smart content optimization. Why pick. Let a smart engine personalize the experience of all your visitors.

Interested in getting your site’s sensitivity matrix for free? Just comment here!

Building Cauzal.ai. We predict the content that will convert your audiences

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store