Sequential Testing is a statistical testing method where sample sizes are not fixed in advance. In Sequential Testing, boundaries are set, large at first, and smaller as more data comes in. This means that at the beginning, you would have to see very extreme notable changes come through.
With more data, we get a higher precision so that we can detect smaller effects. This allows you to make conclusions at a much earlier stage of the experimentation in comparison to the Fixed Horizon approach. With Sequential Testing, you can confidently make informed decisions about your releases, faster since results are always valid.
Why Implement Sequential Testing?
Sequential Testing is perfect for those teams that are looking to move faster without breaking anything in production. Results are always valid, which means you can peek at your data as many times as you’d like without waiting for your experiment review period to complete. This enables faster monitored rollouts, faster release decisions, and quicker iterations of experiments when big, notable changes are detected.
Sequential Testing vs Fixed Horizons
Both Sequential Testing and Fixed Horizons address very different use cases. With Sequential Testing, results are always valid, which means you run a low risk of interpreting a false positive. This method is the most optimal when you’re releasing net new features and looking for big, notable changes within your data to determine whether it’s safe to move forward (or backward) with your releases.
With Fixed Horizons, you have to wait for your experiment review period to complete before peeking at your data to avoid interpreting any false positives. Fixed Horizons is great when you’re trying to make small optimizations within your product such as improving the page load time by 0.10%. You should use both approaches interchangeably, depending on your team’s needs. Learn more about setting up Sequential Testing here.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.
Deliver Features That Matter, Faster. And Exhale.
Split is a feature management platform that attributes insightful data to everything you release. Whether your team is looking to test in production, perform gradual rollouts, or experiment with new features–Split ensures your efforts are safe, visible, and highly impactful. What a Release. Get going with a free account, schedule a demo to learn more, or contact us for more information.