Drive smarter product decisions through online controlled product experiments
Schedule a Split demo tailored to your needs
Speed up development cycles, reduce release risk, and focus your team on DevOps best practices that create maximum impact.
Building a Product Experimentation Platform
Product experimentation platforms consist of a robust targeting engine, a telemetry system, a statistics engine, and a management console.
- The targeting engine is responsible for dividing users across variants.
- Telemetry is the automatic capture of user interactions within the system.
- A statistics engine determines what feature caused a change in your metrics.
- The management console is where experiments are configured, metrics are created, and results of the statistics engine are consumed and visualized.
Designing Metrics
Four types of metrics are important to experiments:
- The Overall Evaluation Criteria is a measure of long-term business value or user satisfaction.
- Feature Metrics are specific metrics that are important to the team in charge of the experiment.
- Guardrail Metrics should be directional and sensitive but not necessarily tie back to business value.
- Debugging Metrics must be sensitive but need not be directional or understandable.
DevOps Best Practices
Running A/A Tests
In an A/A Test, both the treatment and control variants are served the same feature, confirming that the engine is statistically fair and that the implementation of the targeting and telemetry systems are unbiased.
Understanding Power Dynamics
Power Dynamics measures an experiment’s ability to detect an effect when there is an effect there to be detected.
Executing an Optimal DevOps Ramp Strategy
During a ramp up process, taking too many steps or taking too long at any step can slow down innovation. But taking big jumps or not spending enough time at each step can lead to suboptimal outcomes.
Building Alerts and Automation
By building in metrics thresholds, you can set limits within which the experimentation platform will detect anomalies and alert key stakeholders, not only identifying issues but attributing them to their source.