The Difference Between A/B Testing and Multivariate Testing

A/B testing, otherwise known as split testing, is the process of testing two different versions of a web page or other resource in order to optimize for a metric or set of metrics (often conversion rate). Multivariate testing (or MVT) is similar: it involves testing three or more versions.

What is A/B Testing?

Most examples of A/B testing involve the front end – testing different landing pages for purposes of conversion rate optimization – but all that’s really required for A/B testing is testing different ideas for a feature. It can just as easily be testing different features in software or mobile applications as testing different pages or different design elements for user experience optimization.

A/B testing goes like this. Gather or find data on your current resources (homepage, feature, etc.) and figure out which ones aren’t performing as well as they should. Then, hypothesize different possible combinations of changes which will fix the problem. Segment your userbase into two parts, showing the original page or feature to one part and the new version to the other. How quickly you get significant results depends on the amount of traffic you have, since for the test results to have statistical significance, they must come from a large population. After you have your results, if there’s a significant positive change, you can implement the new version for all users.

A/B testing can occur on more than two different variations (this is called A/B/n testing), but not much more than two. The mechanic of A/B testing doesn’t lend itself well to testing more than around four variations at once: testing a higher number of variations is slow, and even A/B/n tests won’t give you a granular view of which subtle changes actually produced the results.

What is Multivariate Testing?

MVT is a testing method that is similar to A/B testing in roughly the same way as a cube is similar to a square: same shape, higher dimensions. You can think of a multivariate test as multiple nested A/B tests.

Say you have two features that you’d like to test variations on instead of just one. If Feature A has two possible states (on and off) and Feature B also has two possible states, in a full factorial scheme, you end up with 4 different combinations: (A on, B off), (A off, B on), (A on, B on), and (A off, B off). Segmenting your userbase into four parts this way will let you know which individual elements produced the change, and further, how much each one contributed.

The biggest problem with multivariate testing is that you need a high-traffic platform to have enough users coming through each group to get significant results in a reasonable amount of time. Because the number of groups grows exponentially with the number of tested variations, you should be careful to make sure you have enough traffic to make a multivariate test worthwhile.