Flagship 2024 – Day 2 is live! Click here to register and watch now.

Glossary

Sequential Testing

With Sequential Testing, you can confidently make informed decisions about your releases, faster since results are always valid.

Sequential Testing is a statistical testing method where sample sizes are not fixed in advance. In Sequential Testing, boundaries are set, large at first, and smaller as more data comes in. This means that at the beginning, you would have to see very extreme notable changes come through.

With more data, we get a higher precision so that we can detect smaller effects. This allows you to make conclusions at a much earlier stage of the experimentation in comparison to the Fixed Horizon approach.

Why Implement Sequential Testing?

Sequential Testing is perfect for those teams that are looking to move faster without breaking anything in production. Results are always valid, which means you can peek at your data as many times as you’d like without waiting for your experiment review period to complete. This enables faster monitored rollouts, faster release decisions, and quicker iterations of experiments when big, notable changes are detected.

Sequential Testing vs. Fixed Horizons

Both Sequential Testing and Fixed Horizons address very different use cases. With Sequential Testing, results are always valid, which means you run a low risk of interpreting a false positive. This method is the most optimal when you’re releasing net new features and looking for big, notable changes within your data to determine whether it’s safe to move forward (or backward) with your releases.

With Fixed Horizons, you have to wait for your experiment review period to complete before peeking at your data to avoid interpreting any false positives. Fixed Horizons is great when you’re trying to make small optimizations within your product such as improving the page load time by 0.10%. You should use both approaches interchangeably, depending on your team’s needs. Learn more about setting up Sequential Testing here.

Want to Dive Deeper?

We have a lot to explore that can help you understand feature flags. Learn more about benefits, use cases, and real world applications that you can try.

Create Impact With Everything You Build

We’re excited to accompany you on your journey as you build faster, release safer, and launch impactful products.

Want to see how Split can measure impact and reduce release risk? 

Book a demo