The biggest success factor in split testing

Posted on June 18, 2015 at 2:13 pm Written by

split testingTim Watson is an email marketing consultant with over 10 years of 100% focus on email marketing. Tim is also an elected member of the UK DMA Email Council, supporting the email marketing industry and Chairs the Legal and Best Practice hub of the Email Council, authoring and reviewing DMA whitepapers and best practice documentation. Below he shares with us the biggest success factor in split testing. 

Big uplifts and exciting results, you’ve seen the many case studies and blog posts that show just how valuable split testing is, right? Perhaps you’ve tried a few tests and have been disappointed with results?

All the super results case studies leave out one thing and without this one factor you can’t be successful.

They all left out that many tests were run and most produced uninteresting results, the only result published was the one big hit.

What does this mean? The key ingredient of split testing is simply doing enough of it.

As a rule of thumb expect out of ten test treatments for one to make a big shift to the needle, 3 provide some learning and improvement, 3 no winner and 3 under perform.

On this basis running one A/B test a month means you’ll get one exciting gain per year. Nobody wants to wait a year to improve their results.

I normally incorporate split testing as part of strategy with my consulting clients; except where there is insufficient data to make testing feasible.

As an example, in a recent email split test I ran 11 treatments. Of those 4 produced uplifts, 2 had the same performance as the control and 4 under performed. Two of the 4 winners gave over 30% improvement in click through rate. I simply couldn’t have got this result without incorporating quantity of testing into the test plan.

If you are running A/B subject line tests then be prepared for several tests showing no performance improvement. Don’t get despondent but keep persevering, quantity and consistency are key to making split test strategy work.

Don’t forget that your test cells must be large enough for validity; you won’t get a valid result by running 10 split tests on a list of 5000. Use this free split test calculator to work out test cell sizes and check your results.

Also take a look at this advice if you run two tests and get the same split test results. It provides more tips as to what to do next.

This article was written by Tim Watson.


Get the latest Email Marketing updates & insights