This article outlines a practical way for setting up your A/B testing at an early stage business.
Run extreme tests
It’s important to make sure your tests are bold enough. This is especially true if you’re in a low traffic environment, because time is a precious commodity. It may take you 4+ weeks to get to statistical significance.
If you’re only able to run 1 a month, you want to make sure that those tests are going to set you off in a new direction that drives growth. Make sure that there is considerable variation between the ideas you’re testing.
Don’t be distracted by stories of people testing something as moderate as button colours. Although that can be powerful - it requires an extremely large traffic volume for those insights to be uncovered in your testing programme. The effect on growth from something like button colour changes is unlikely to be significant enough to hit your challenging targets.
Save space in your testing programme for those ideas that are going to make a step-change in your business.
Make sure that result has a predefined action plan, ask yourself: ‘What exactly will we do if this test succeeds / fails?’
Setting up your A/B testing tool
There’s lot written on different A/B testing tools. Optimizely is one of the more popular tools as it’s user friendly but feature-rich. It’s likely that you’ll hire people that have used it before.
There is now a Pay-As-You-Go pricing model ($49 for 1,000 uniques per month).
To start testing, add the Optimizely snippet to every page on your site. You’ll do this just once, but then you can run the experiment across your site. You can read more about how the set-up works on the Optimizely site here.
Integrate Optimizely with Mixpanel
There’s a lot of benefit with integrating Optimizely directly with Mixpanel (Kissmetrics or similar would also work).
This joins up the ‘event tracking’ in your product analytics, with the tests that you’re running. The Optimizely experiment ID & variation ID become ‘properties’ or ‘attributes’ of the events that you’re tracking, they will be linked directly to the email address of that particuar user.
You’ll now be able to ask more sophisticated questions like:
What’s the difference in average revenue per customer from variation 1 compared with variation 2?
Show me the user journeys of the top 10 customers from variation 1. How do these compare to variation 2?
What are the biggest drop-off points in the conversion funnel for variation 1? How do these compare to variation 2?
What’s the difference in customer retention between variation 1 and variation 2 after 3 months?
Optimizely provide their own ‘Stats Engine’, you can read more about that here. But by taking the data out of Optimizely, it means you have the flexibility to use whatever model of statistical significance you prefer.
We’re more likely to use the p-value test, rather than the Optimizely Stats Engine - as it’s useful to have full visibility of the numbers and associated calculations. As a broad rule of thumb for early stage businesses, you’re looking for a minimum sample of 100 conversions for your test. Once you reach your sample size - then you can use an online statistical significance calculator. Over time, you can get more sophisticated in how you're thinking about sample size and significance thresholds, but for the moment - keep it simple.
The Optimizely Stats Engine does not require a pre-defined sample size, for more detail on this subject Optimizely released a paper which outlines their methodology; created in conjunction with Stanford University.
Get the team onboard
Finally, this is inherently a cross-functional project. It's important to have buy-in from across the team. To conduct a successful growth programme - you'll need a combination of creative ideas, design, UX soultions, technical execution, and much more. If you're leading the project, speak to the team regularly and create a central hub to share the results.
I'd like to thank Tom Evans, COO @ Email Octopus for teaching me valuable lessons about A/B testing.