Have a Clear Hypothesis and Focus Area Before You Start A/B Testing

I’m continuing with my video series on A/B testing. I added a few new questions this week and the transcription is below. Here are links to part 1 and part 2.

Is there anything that should be done before you can start A/B testing?

Before you can start A/B testing, it’s really important that you have a clear understanding of what you’re actually testing. What is the hypothesis that you’re trying to prove? Why do you think that your test is more likely to win than what is already on the site? Do you have a good understanding of the customer and why you’re running the test that you’re planning to run, versus just testing for the sake of testing. I think that’s incredibly important. So if you can’t clearly articulate the hypothesis that you’re testing, if you can’t clearly articulate the customer reason why the test is going to be run, I think then it is really difficult to justify running an A/B test. You should probably go back and spend more time getting a sense for who your customer is and what your product is.

Secondarily, you need to make sure you have enough traffic to successfully run an A/B test. If you don’t have enough users hitting a particular page or a particular part of your product where you want to run a test, you may not actually be able to get to a statistically significant result. You might need to do other forms of testing to validate that whatever you’re interested in testing is something that users want versus just assuming you can prove it out with data. It’s easy to assume that everything can run through an A/B test, but you need to also validate that you have enough data to actually get the results you’re looking for.

What is the single most important thing to A/B test?

I think it’s hard to answer what is the single most important thing to A/B test? I think it really depends what part of the product growth cycle you’re in. For some products, if you’re really struggling to get more users from the top of your funnel through your funnel to convert, it might be your conversion funnel. For other products, it might be something like the tactics that you’re using in retention, like testing your email, your other types of channels and different things like that.

It really will come down to understanding where your product is at and what’s the most important metric for you to move. I think some of the most common examples are

  • Conversion rate optimization. Looking at how you’re doing at the top of the funnel and figuring out if you can convert more customers.

  • Core User Experience. Looking at your ongoing user experience and making sure that users are engaging with the most important parts of your product. If users are actively using those parts of the product that tends to be a key indicator of retention.

  • Customer Retention Strategy: What are your key strategies and tactics for customer retention? Is it push notifications, emails, or something else? There are a lot of easy, testable elements in each one of those different topics.

I think those are some of the areas where you could easily start A/B testing. However, you shouldn’t just start there because that’s what everyone else does. You should take some time to figure out where you have the biggest opportunity within your product.