159: Mastering A/B Testing: Strategies and Insights from Sascha Bossen and Yours Truly

A/B testing can be fun, but there are some things to watch out for

Sascha Bossen has a good article on A/B testing that I’ll link below. Today I’ll review his four points, and add three of my own.

Pay attention to the novelty effect

One panel might outperform your control simply because it’s new. To avoid this mistake, first, make sure to watch performance over time. The novelty effect will wear off. Second, check to see if first-time visitors behave differently than returning visitors.

Make sure your test groups are statistically significant.

Three is 150% more than two. But that’s statistically meaningless. You need a larger sample size before you can make any conclusions.

When you do an A/B split test, you end up with some numbers. Variant A got X clicks out of Y view, etc. Before you make any decisions, you need to check to see if the numbers are significant, and there are some good online tools to help with that. Sascha provides links in his article.

Only change one KPI per test

That doesn’t mean only test one thing. Some people say every test should only test one change on the page. I don’t agree with that. Sometimes there’s a need to try a completely different design.

But Sascha’s addressing a different point, which is what action you’re using to measure the test.

For example, let’s say you have an article template. You could have several KPIs for that page, such as time on site, clicking through to sign up for an e-newsletter, or posting a comment. Your A/B test should focus on one of those because errors multiply as you increase the number of KPIs.

Try usability testing if you can’t do A/B tests

Sometimes you don’t have enough traffic to do a reasonable test in a reasonable amount of time. In those cases, Sascha recommends usability testing as a proxy.

I’m going to add three more things to Sascha’s list.

Abandon tests that take too long to finish.

Sascha hints at this, but I want to make it explicit. If a test takes too long to get a significant answer, you really can’t trust that test. It’s like the old “you can’t step into the same river twice” argument. If the test goes on for too long, are you really testing what you think you’re testing, or are you testing changing moods and currents in your market.

For example, let’s say I want to sell a bicycle, and I test two product images: one of the bicycle, and one of a beautiful person on the bicycle. If that test goes on for a while with no clear winner – going back and forth between the two – I’d start to wonder about external factors, like maybe the beautiful person I picked looks too much like a particular actor.

That leads to my next point.

Avoid testing psychosis.

Testing can sound so logical and mathematical, but the more you get into testing the more you realize how many variables you can’t control, and how many extraneous things could be affecting your results. What if your landing page suddenly gets an influx of traffic from a completely different source? What if a new version of a popular browser changes the way your page looks? And when we get into custom audiences, it gets even more complicated. Does one audience prefer variant A and another prefer variant B?

It goes on and on, and while you should have an eye out for those things, don’t go crazy about it.

Which leads to my final point.

When in doubt follow Amazon.

They have tons of traffic and they’re constantly testing. What they’re doing is probably as close to good as you’re going to find. So the cheapest way to do A/B testing is to avoid it altogether and mimic Amazon wherever you can.

Links

4 learnings about A/B testing after 5 years working on Paid Content at ZEIT ONLINE
https://theaudiencers.com/operations/4-learnings-about-a-b-testing-after-5-years-working-on-paid-content-at-zeit-online/