Things to Consider When Using A/B Testing

Size matters

When planning A/B testing you should always consider the sample size you are going to need. For example, if you don’t anticipate the difference between variation A and variation B is going to be substantially different, you’re going to need a larger survey to reliably indicate whether the margin of difference is statistically significant enough to justify the survey results.

One thing at a time

It’s always best to have just one variation for each A/B test. If you change multiple parameters between variant A and variant B, it’s far too easy to lose sight of exactly what it was that created any difference in perception or performance in the results. The aim is to reduce complexity and make crystal clear what it is about your product or service that will sell, not increase the amount of guesswork involved.

Go the extra mile

The further down the customer journey path you go, the more insights you are going to be able to measure. Let’s say you are promoting an eBook about a new product and measuring clicks to gauge its effectiveness. Sorry, but counting clicks in isolation is not a particularly good metric for measuring success, largely because of infestations of spam and accidental clicks.

Measuring downloads, on the other hand, is a much better yardstick because they are almost certainly intentional, and by actual human beings. However, the best measurement is sales. Why? Closing sales deals not only takes the monetary value of sales into account, it factors in the quality of the account in terms of its viability for longer term sales deals. Granted, this measurement can be tricky to assess and must be acquired over a significantly longer period of time than strictly digital methods, but it is nevertheless a good window into what’s working, and what’s not. Identifying which is which by using the appropriate measurement tools is the key to capitalise on what’s working well, and scrapping market failures before it’s too late to recover.

What’s old is new again, and again

Believe it or not, email of some description has been around for more than 50 years and is almost as common as indoor plumbing. However, do not take it for granted as a marketing tool. Email is still one of the most effective marketing tools in your arsenal, if created – and tested – properly.

Email marketing messages, as highly effective as they can be, require as equally rigorous A/B testing as any other marketing tool. However, when you’re doing your emails testing, make sure you’re testing the right things and in the right way. For example, sending version A of your marketing email to the UK and version B to Germany is not AB testing. Language and meaning differences alone will invalidate the results.

The subject line (and From address!) will significantly impact the open rate. The content won’t because recipients won’t see the content until they open it. The layout of the email will also impact the click-to-open rate, so it must be equally easy to observe, interpret and intrigue and collectively culminate in a succinct, compelling call to action.

It’s also worth remembering malware bots can impact click and open rates, so email testing is not an exact science. Again, a confirmed sale is often a better yardstick than a click.

Half the money I spend on advertising is wasted; the trouble is I don't know which half.”

US marketing pioneer John Wanamaker (1838-1922)

Well John, we still don’t know for sure which half, but we can now vastly improve our odds of getting it right by focussing our advertising with far more precision through A/B testing.

Testing advertising is more important than ever. Napier’s view is that it can be more difficult to test advertising in online trade journals because you can’t make frequent changes to the digital ads that are running. Yes, you can ask a publication to run more than one version of an ad at some point to help with your testing, but it’s not an ideal way to acquire the data you need.

Programmatic advertising (e.g., search ads, Google display, LinkedIn/paid social) on the other hand, is much easier and more definitive. Using programmatic channels to test and then enhance your display ads enables you to iterate them far more quickly through those channels and, therefore, achieve more immediate, actionable results. It’s important to point out that Google is now attempting to bypass A/B testing with “responsive” ads, i.e., where you upload multiple headlines/body copy/images that Google will “optimise” for the best combination. To be fair, it generally works reasonably well. The downside is that it can be hit and miss in providing results that give you the level of focussed information you could achieve with a true A/B test. Google takes into account just too many parameters in its optimisation process that are probably irrelevant for what you need and therefore provides false positive - or negative – results.

Again, counting clicks isn’t necessarily the best measure of performance, but the practice has tended to remain a popular choice.

Similar to email A/B testing, you should run two variations of an ad simultaneously. Running one version in December and the other in January is not a fair AB test. (Running variants of an ad can be done easily with tools like Creatopy, which enables you to create multiple versions of ads quickly and can be very useful for ad A/B testing.)

A web of receipt

Yes, if you expect to generate revenue via your web site, whether as a shop window or ecommerce portal, you need to A/B test it, too.

Your aim in A/B testing a website should be three-fold:

To reduce your bounce rate
Secure more downloads (pdfs and the like) or form fills
Increase the amount of time spent on your site and/or pages visited

However, be aware that it can be difficult to A/B test web pages on your own. To do it correctly you typically need special software, although many marketing automation systems now include built-in A/B testing modules for landing pages. Such software enables you to optimise landing page headlines, body copy, bullets, the call to action, layout (including where you place the form) and the images you plan to use.