O'Reilly eBook

Understanding Experimentation Platforms

Drive Smarter Product Decisions Through Online Controlled Product Experiments

A software developer’s playbook for mitigating unintended consequences in financial services, and how feature flags can help.

Learn Why Agile Businesses Use Split

Deliver software features that matter, fast!

renderContent(deleteTreatment) {
  const allowDelete = deleteTreatment.treatment === "on";
  return (
    <div className="todoListMain">
      <div className="header">
        <form onSubmit={this.addItem}>
          <input
            ref={(a) => (this._inputElement = a)}
            placeholder="Enter Task"
          ></input>
          <button type="submit">Add</button>
        </form>
      </div>
      <TodoItems
        entries={this.state.items}
        allowDelete={allowDelete}
        delete={this.deleteItem}
      />
      <TodoItems
        entries={this.state.items}
        allowDelete={allowDelete}
        delete={this.deleteItem}
      />
    </div>
  );
}

Building a Product Experimentation Platform

Product experimentation platforms consist of a robust targeting engine, a telemetry system, a statistics engine, and a management console.

The targeting engine is responsible for dividing users across variants.
Telemetry is the automatic capture of user interactions within the system.
A statistics engine determines what feature caused a change in your metrics.
The management console is where experiments are configured, metrics are created, and results of the statistics engine are consumed and visualized.

Download eBook

DevOps Best Practices

Whether you’re merging conflicts, managing silos, or working on the weekend, Engineering Managers can make life better for their teams.

Download eBook

Running A/A Tests
In an A/A Test, both the treatment and control variants are served the same feature, confirming that the engine is statistically fair and that the implementation of the targeting and telemetry systems are unbiased.
Understanding Power Dynamics
Power Dynamics measures an experiment’s ability to detect an effect when there is an effect there to be detected.
Executing an Optimal DevOps Ramp Strategy
During a ramp up process, taking too many steps or taking too long at any step can slow down innovation. But taking big jumps or not spending enough time at each step can lead to suboptimal outcomes.
Building Alerts and Automation
By building in metrics thresholds, you can set limits within which the experimentation platform will detect anomalies and alert key stakeholders, not only identifying issues but attributing them to their source.