A/B testing, otherwise known as split testing, is the process of testing two different versions of a web page, feature, user flow, or other resource in order to optimize for a metric or set of metrics (often conversion rate). Multivariate tests are run with a higher number of variables and generally provide deeper insight on how to optimize your page. In multivariate testing, your feature implementation becomes a combination of elements which can be decomposed and tested simultaneously.
Advantages of A/B Testing
Most examples of A/B testing involve the front end – testing different landing pages for purposes of conversion rate optimization – but all that’s really required for A/B testing is testing different ideas for a feature. It can just as easily be testing different features in software or mobile applications as testing different pages or different design elements for user experience optimization.
Because there are only two versions of your software when conducting an A/B test, your tests will deliver dependable data quickly. The more variations you have, the longer it will take to gather significant data – the traffic required for A/B testing is much less than the traffic required for multivariate testing.
The first thing that needs to happen when you run an A/B test is to gather data on your current resources (homepage, feature, user flow, etc.) and figure out which ones aren’t performing as well as they should. This should be a list of the features that you would like to optimize.Once you have this list, you can prioritize them in order of the potential value that they could bring. You should work with both your data analyst and product person to gather this information.
Hypothesize different possible combinations of changes that will fix the problem. This is usually done by the product person. Segment your user base into two parts, showing the original page or feature to one part and the new version to the other. How quickly you get significant results depends on the amount of traffic you have, since for the test results to have statistical significance, they must come from a large population. After you have your results, if there’s a significant positive change, you can implement the new version for all users.
Testing Multiple Features
A/B testing can occur on more than two different variations (this is called A/B/n testing), but not much more than two. The mechanic of A/B testing doesn’t lend itself well to testing more than around four variations at once: testing a higher number of variations is slow, and even A/B/n tests won’t give you a granular view of which subtle changes actually produced the results.
What is Multivariate Testing?
Multivariate testing is a testing method that is similar to A/B testing in roughly the same way as a cube is similar to a square: same shape, higher dimensions. You can think of a multivariate test as multiple nested A/B tests.
Say you have two features that you’d like to test variations on instead of just one. If Feature A has two possible states (on and off) and Feature B also has two possible states, in a full factorial scheme, you end up with 4 different combinations: (A on, B off), (A off, B on), (A on, B on), and (A off, B off). Segmenting your user base into four parts this way will let you know which individual elements produced the change, and further, how much each one contributed.
How Do You Know Which Type of Test to Run?
The biggest problem with multivariate testing is that you need a high-traffic platform to have enough users coming through each group to get significant results in a reasonable amount of time. Because the number of groups grows exponentially with the number of tested variations, you should be careful to make sure you have enough traffic to make a multivariate test worthwhile. So for a lower-traffic platform, it’s best to run an A/B test.
Learn More About A/B Testing
Interested in digging deeper on A/B Testing and Experimentation? We’ve got you covered:
- Learn about more about A/B testing and A/B/n testing
- Dig deeper into Multivariate testing
- And finally, check out the state of feature delivery in 2020
Stay up to date
Don’t miss out! Subscribe to our digest to get the latest about feature flags, continuous delivery, experimentation, and more.
The marriage of feature flags with Split and security via Spring Security makes testing in production so easy it might blow your mind! Get building today!
Product experimentation is the most effective way to determine what works best for end-users. You can measure engagement with a feature and hypothesize ways to improve that engagement, which will ultimately impact one or more critical business metrics. If you’re setting up an experiment, it’s mission-critical to implement A/B testing…