With fixed horizon, you can do a single statistical test and draw conclusions only once your review period has completed. Since sample sizes are defined in advance, this is the perfect testing method when you’re looking to make small improvements to a relatively low traffic site or page.
Why Implement Fixed Horizon?
Fixed Horizon is perfect for those teams that are looking to make tiny optimization “tweaks” within their product. Results are only valid once your experiment has reached statistical significance and your review period has been met. This enables the ability to track even the slightest movement in your data (i.e. 0.10% increase in your page load times).
Fixed Horizon vs. Sequential Testing
With Sequential Testing, results are always valid the minute you launch an experiment. However, since you don’t need to define your sample size in advance, this method is optimal when you’re trying to detect big notable changes in your data. With Fixed Horizons, you have to wait for your experiment review period to complete before peeking at your data to avoid interpreting any false positives. However, since you define your sample size in advance, this method is great when making slight tweaks within your KPIs. If you were to use the Sequential Testing method when making small improvements, it would take you a longer time to reach a conclusion in comparison to the Fixed Horizon approach. Both methods should be used interchangeably, depending on your team’s needs. Learn more about setting up Fixed Horizons here.
