Fixed Horizon is a statistical testing method where sample sizes and your experiment goals are defined in advance. In this method, you can do a single statistical test and draw conclusions only once your review period has completed. Since sample sizes are defined in advance, this is the perfect testing method when you’re looking to make small improvements to a relatively low traffic site or page.
Why Implement Fixed Horizon?
Fixed Horizon is perfect for those teams that are looking to make tiny optimization “tweaks” within their product. Results are only valid once your experiment has reached statistical significance and your review period has been met. This enables the ability to track even the slightest movement in your data (i.e 0.10% increase in your page load times).
Fixed Horizon vs. Sequential Testing
With Sequential Testing, results are always valid the minute you launch an experiment. However, since you don’t need to define your sample size in advance, this method is optimal when you’re trying to detect big notable changes in your data. With Fixed Horizons, you have to wait for your experiment review period to complete before peeking at your data to avoid interpreting any false positives. However, since you define your sample size in advance, this method is great when making slight tweaks within your KPIs. If you were to use the Sequential Testing method when making small improvements, it would take you a longer time to reach a conclusion in comparison to the Fixed Horizon approach. Both methods should be used interchangeably, depending on your team’s needs. Learn more about setting up Fixed Horizons here.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.
Deliver Features That Matter, Faster. And Exhale.
Split is a feature management platform that attributes insightful data to everything you release. Whether your team is looking to test in production, perform gradual rollouts, or experiment with new features–Split ensures your efforts are safe, visible, and highly impactful. What a Release. Get going with a free account, schedule a demo to learn more, or contact us for more information.