O'Reilly eBook

Understanding Experimentation Platforms

Drive Smarter Product Decisions Through Online Controlled Product Experiments

A software developer’s playbook for mitigating unintended consequences in financial services, and how feature flags can help.

Learn Why Agile Businesses Use Split

Deliver software features that matter, fast!

renderContent(deleteTreatment) {
  const allowDelete = deleteTreatment.treatment === "on";
  return (
    <div className="todoListMain">
      <div className="header">
        <form onSubmit={this.addItem}>
          <input
            ref={(a) => (this._inputElement = a)}
            placeholder="Enter Task"
          ></input>
          <button type="submit">Add</button>
        </form>
      </div>
      <TodoItems
        entries={this.state.items}
        allowDelete={allowDelete}
        delete={this.deleteItem}
      />
      <TodoItems
        entries={this.state.items}
        allowDelete={allowDelete}
        delete={this.deleteItem}
      />
    </div>
  );
}

Building a Product Experimentation Platform

Product experimentation platforms consist of a robust targeting engine, a telemetry system, a statistics engine, and a management console.

  • The targeting engine is responsible for dividing users across variants.
  • Telemetry is the automatic capture of user interactions within the system.
  • A statistics engine determines what feature caused a change in your metrics.
  • The management console is where experiments are configured, metrics are created, and results of the statistics engine are consumed and visualized.

DevOps Best Practices

Whether you’re merging conflicts, managing silos, or working on the weekend, Engineering Managers can make life better for their teams.

Download eBook
  • Running A/A Tests

    In an A/A Test, both the treatment and control variants are served the same feature, confirming that the engine is statistically fair and that the implementation of the targeting and telemetry systems are unbiased.
  • Understanding Power Dynamics

    Power Dynamics measures an experiment’s ability to detect an effect when there is an effect there to be detected.
  • Executing an Optimal DevOps Ramp Strategy

    During a ramp up process, taking too many steps or taking too long at any step can slow down innovation. But taking big jumps or not spending enough time at each step can lead to suboptimal outcomes.
  • Building Alerts and Automation

    By building in metrics thresholds, you can set limits within which the experimentation platform will detect anomalies and alert key stakeholders, not only identifying issues but attributing them to their source.

Designing Metrics

Four types of metrics are important to experiments:

  • Overall Evaluation Criteria measures business value or user satisfaction
  • Feature Metrics are important to the team in charge of the experiment
  • Guardrail Metrics should be directional and sensitive
  • Debugging Metrics must be sensitive but need not be directional

Schedule a Split Demo Tailored to Your Needs

Speed up development cycles, reduce release risk, and focus your team on DevOps best practices that create maximum impact.

Book a Demo

Create Impact With Everything You Build

We’re excited to accompany you on your journey as you build faster, release safer, and launch impactful products.