Glossary /

A/B/n Testing

A/B/n testing is the process of A/B testing with more than two different versions. The little “n” doesn’t refer to a third test, but to any number of additional tests: A/B/n encompasses A/B/C, A/B/C/D, or any other type of extended A/B test.

Despite these additional variations, though, A/B/n testing works the same way as standard A/B testing: split users into groups, assign variations (typically of landing pages or other webpages) to groups, check the change of a key metric (typically conversion rate), check the test results for statistical significance, deploy the winning version.

A/B/n testing vs multivariate testing

Though they’re often confused, A/B/n testing is not the same as multivariate testing. The key difference lies in how the variations are controlled. Let’s use a webpage as an example. Say we have an image and a call to action (CTA) button, and we have three variations of each. If we run a multivariate test, it will automatically test all possible combinations – in this case, 6. However, if we run an A/B/n test, we hand-select which variations we want to test, which is frequently less than every possible combination. If we had a large number of different resources we wanted to test, the number of different variations in a multivariate test would grow exponentially – quickly requiring massive amounts of traffic and time it would take to get statistically significant results – but in an A/B/n test, we can manually choose how many variations to deploy.

A/B/n testing is more helpful in situations where getting results is more important than learning or generalizing from them. Multivariate testing, because of its granularity, is more helpful where knowing the precise cause of an increase or decrease in traffic is worth waiting for.

A/B/n testing vs multi-armed bandit testing

Another experimentation method, which happens to be more commonly used in machine learning than marketing, is the multi-armed bandit algorithm (MAB). Pardoning the esoteric, gambling-inspired name, multi-armed bandits basically use a different set of assumptions on how long an experimentation algorithm should spend on exploring possible alternatives versus how long it should spend exploiting those it has already found.

The process of A/B testing in general, and A/B/n testing, in particular, explores possible alternatives and their effectiveness for the test period before spitting out an answer and letting the user exploit the opportunity it has decided is best. By contrast, MABs dynamically explore and exploit in much shorter phases, relying on the past effectiveness of explored opportunities to decide on their next actions.

MAB testing is applicable to a broader range of problems than A/B testing. A MAB can produce significant results more quickly than an A/B test, and it can also automatically adapt to a changing environment and provide the best alternative in each, where several sequential A/B tests would need to be run manually to achieve the same result. However MABs are not perfect: if there is any significant time between a change and its result – like an email campaign taking a few days to convert a prospect – A/B testing is far superior. Not to mention, MABs are more computationally difficult than A/B tests.