Glossary

Testing in Production

Testing in production is the process of testing your features in the environment that your features will live in. It does not mean releasing untested code to users and hoping it works, but rather using feature flags to test the different treatments. This is best implemented in addition to pre-production…

Client Side Testing

Client-side testing refers to any type of testing – commonly A/B testing, but also multivariate testing or multi-armed-bandit testing – that occurs in the user’s browser. This is contrasted with server-side testing, where the test cases are decided on the back-end (in the web server) before they’re served to the…

Server Side Testing

Server-side testing refers to any type of testing – commonly A/B testing, but also multivariate testing or multi-armed bandit testing – that occurs on the web server instead of in the user’s browser. This is contrasted with client-side testing, where the test cases are rendered (typically using some type of…

Statistical Significance

When people discuss A/B testing, they commonly throw around the term “statistical significance” a lot. But what does this mean? In short, getting a statistically significant result means that the result is highly unlikely to be the product of random noise in the data, and more likely to be the…

Dark Launch

Dark launching is the term for releasing features to a subset of your users, seeing how they respond, and making updates to your features accordingly. It’s somewhat like what every project manager does to monitor application health, but focused entirely on a single new feature. In this modern age of…

Hypothesis Driven Development

If a software engineer wakes up in the morning, and hears something that sounds like rain outside her window, she thinks it might be raining. Her hypothesis, that it’s raining, drives her decision to look out her window. She knows ahead of time that if she sees rain, it’s actually…

Feature Experimentation

Ordinary software development is widespread with guesswork. Some amount of data, plus some amount of the product manager’s hunches, drives the decisions of what new features the development team will create. Without some fairly major shifts in how development is done, this is almost how it has to be: you…

Customer Experience Metrics

Customer experience (abbreviated CX) is the experience your customers have with your brand. Customer experience metrics are organizational KPIs that help you monitor your customer journey to see if there are touch points where you are letting customer churn rate increase. Today, we’ll discuss some of the most commonly-used customer…

Controlled Rollout

A controlled rollout is a feature rollout with highly granular user targeting. It allows you to release new features gradually, ensuring a good user experience for smaller groups of users before releasing them to larger groups. Controlled Rollout Use Cases  There are two basic types of controlled rollouts: those where…

Observability

Observability is defined as the ability of the internal states of a system to be determined by its external outputs. With the unknown unknowns of our software’s failure modes, we want to be able to figure out what’s going on just by looking at the outputs: we want observability. “Observability”…