Rapid Growth

Reading Time Time to read: 4 minutes

A practical guide to measuring A/B tests

Jake Higgins

Head of Growth

In an ideal world, your technology team would build an in-house A/B testing tool that has a deep integration with your product, however, most startup companies don’t live in an ideal world.

This article outlines a practical way for setting up your A/B testing at an early stage business.

Run extreme tests

It’s important to make sure your tests are bold enough. This is especially true if you’re in a low traffic environment, because time is a precious commodity. It may take you 4+ weeks to get to statistical significance.

If you’re only able to run 1 a month, you want to make sure that those tests are going to set you off in a new direction that drives growth. Make sure that there is considerable variation between the ideas you’re testing.

Don’t be distracted by stories of people testing something as moderate as button colours. Although that can be powerful - it requires an extremely large traffic volume for those insights to be uncovered in your testing programme. The effect on growth from something like button colour changes is unlikely to be significant enough to hit your challenging targets.

Save space in your testing programme for those ideas that are going to make a step-change in your business.

Make sure that result has a predefined action plan, ask yourself: ‘What exactly will we do if this test succeeds / fails?’

Setting up your A/B testing tool

There’s lot written on different A/B testing tools. Optimizely is one of the more popular tools as it’s user friendly but feature-rich. It’s likely that you’ll hire people that have used it before.

There is now a Pay-As-You-Go pricing model ($49 for 1,000 uniques per month).

To start testing, add the Optimizely snippet to every page on your site. You’ll do this just once, but then you can run the experiment across your site. You can read more about how the set-up works on the Optimizely site here.

One word of warning: it’s not best practice to add the ‘javascript snippet’ into into Google Tag Manager, but instead add it into the <head> tag of your site. Otherwise you might find it takes longer to load, and the user may be flashing between different versions of your site. Google’s official line is that it doesn’t support A/B testing tags.

Integrate Optimizely with Mixpanel

There’s a lot of benefit with integrating Optimizely directly with Mixpanel (Kissmetrics or similar would also work).

This joins up the ‘event tracking’ in your product analytics, with the tests that you’re running. The Optimizely experiment ID & variation ID become ‘properties’ or ‘attributes’ of the events that you’re tracking, they will be linked directly to the email address of that particuar user. 

You’ll now be able to ask more sophisticated questions like:

What’s the difference in average revenue per customer from variation 1 compared with variation 2?

Show me the user journeys of the top 10 customers from variation 1. How do these compare to variation 2?

What are the biggest drop-off points in the conversion funnel for variation 1? How do these compare to variation 2?

What’s the difference in customer retention between variation 1 and variation 2 after 3 months?

Advanced technical tip: For complex queries you can use the JQL language to directly query the data and bypass the Mixpanel segmentation and funnel tools. JQL is a mixture of javascript and SQL that was created by Mixpanel specifically for this purpose.

Measuring success

Optimizely provide their own ‘Stats Engine’, you can read more about that here. But by taking the data out of Optimizely, it means you have the flexibility to use whatever model of statistical significance you prefer.

We’re more likely to use the p-value test, rather than the Optimizely Stats Engine - as it’s useful to have full visibility of the numbers and associated calculations. As a broad rule of thumb for early stage businesses, you’re looking for a minimum sample of 100 conversions for your test. Once you reach your sample size - then you can use an online statistical significance calculator. Over time, you can get more sophisticated in how you're thinking about sample size and significance thresholds, but for the moment - keep it simple.

The Optimizely Stats Engine does not require a pre-defined sample size, for more detail on this subject Optimizely released a paper which outlines their methodology; created in conjunction with Stanford University.

Get the team onboard 

Finally, this is inherently a cross-functional project. It's important to have buy-in from across the team. To conduct a successful growth programme - you'll need a combination of creative ideas, design, UX soultions, technical execution, and much more. If you're leading the project, speak to the team regularly and create a central hub to share the results. 


I'd like to thank Tom Evans, COO @ Email Octopus for teaching me valuable lessons about A/B testing. 

Jake Higgins

Head of Growth

Jake was the first employee at Property Partner, which he helped grow from 0 - 10,000+ customers. That was following numerous growth roles at startup and large technology companies, taking time in-between those to teach Digital Marketing & Business Fundamentals at General Assembly. Jake is a fellow at the New Entrepreneur Foundation.

Apply for Office Hours

We’re looking for great entrepreneurs with great ideas.

Apply here

Similar Guides