Glossary

Type II Error

A type II error (type 2 error) is one of two types of statistical errors that can result from a hypothesis test (the other being a type I error). Technically speaking, a type II error occurs when a false null hypothesis is accepted, also known as a false negative. In…

CI/CD – Continuous Integration/Continuous Delivery

CI/CD is the acronym in software development for the combination of continuous integration (CI) and continuous delivery (CD). It can also be expanded to include continuous deployment, but to avoid confusion in this article, we’ll be using CD to refer to continuous delivery. It’s an extremely useful agile process for…

Trunk-Based Development

Trunk-based development (TBD) is a branching model for software development where developers merge every new feature, bug fix, or other code change to one central branch in the version control system. This branch is called “trunk”, “mainline”, or in Git, the “master branch”. Trunk-based development enables continuous integration – and,…

Feature Flag Management

Feature flag management is the process of, well, managing feature flags. While feature flags are extremely useful for many purposes, they require an organized process to manage. In what specific ways, we’ll discuss shortly. There are two main types of feature flag management systems: those which are built in-house and…

False Positive Rate

False positive rate (FPR) is a measure of accuracy for a test: be it a medical diagnostic test, a machine learning model, or something else. In technical terms, the false positive rate is defined as the probability of falsely rejecting the null hypothesis. False Positive Definition Imagine you have an…

Testing in Production

Testing in production is the process of testing your features in the environment that your features will live in. It does not mean releasing untested code to users and hoping it works, but rather using feature flags to test the different treatments. This is best implemented in addition to pre-production…

Client Side Testing

Client-side testing refers to any type of testing – commonly A/B testing, but also multivariate testing or multi-armed-bandit testing – that occurs in the user’s browser. This is contrasted with server-side testing, where the test cases are decided on the back-end (in the web server) before they’re served to the…

Server Side Testing

Server-side testing refers to any type of testing – commonly A/B testing, but also multivariate testing or multi-armed bandit testing – that occurs on the web server instead of in the user’s browser. This is contrasted with client-side testing, where the test cases are rendered (typically using some type of…

Statistical Significance

When people discuss A/B testing, they commonly throw around the term “statistical significance” a lot. But what does this mean? In short, getting a statistically significant result means that the result is highly unlikely to be the product of random noise in the data, and more likely to be the…

Dark Launch

Dark launching is the term for releasing features to a subset of your users, seeing how they respond, and making updates to your features accordingly. It’s somewhat like what every project manager does to monitor application health but focused entirely on a single new feature. In this modern age of…