A/A Testing
A/B testing is the process of split testing two different variations of a web page or feature by serving different versions of the feature to …
Glossary
Your guide to progressive delivery and experimentation concepts in software development.
A
A/B Testing
A/B testing, otherwise known as split testing, is the process of testing two different versions of a web page or product feature in order to …
A/B/n Testing
A/B/n testing is the process of A/B testing with more than two different versions. The little “n” doesn’t refer to a third test, but to …
B
Blue/Green Deployment
Blue/green deployment is a continuous deployment process that reduces downtime and risk by having two identical production environments, called blue and green. (The names blue …
C
Canary Deployment
A canary deployment, or canary release, is a deployment pattern that allows you to roll out new code/features to a subset of users as an …
Change Advisory Board
A change advisory board (or CAB) is a collective of representatives from different departments within the company who run that company’s formal change management processes. …
Chaos Engineering
In any complex system, failure is inevitable. Chaos engineering, or chaos testing, embraces this reality, allowing you to introduce failures in a controlled way.
CI/CD – Continuous Integration/Continuous Delivery
CI/CD is the acronym in software development for the combination of continuous integration (CI) and continuous delivery (CD). It can also be expanded to include …
Client Side Testing
Client-side testing refers to any type of testing – commonly A/B testing, but also multivariate testing or multi-armed-bandit testing – that occurs in the user’s …
Configuration Drift
More and more companies are beginning to understand that using a staging environment to test features causes more harm than good. Because this process separates …
Continuous Delivery
Continuous delivery is a software delivery process centered around improving the speed with which development teams release new features to end-users.
Continuous Delivery Tools
Continuous delivery is the process of systematically keeping code deploy-ready at all times. This shift in release process requires several changes in the operations of …
Continuous Delivery vs Deployment
Software development has changed a lot in the last few years. Now, instead of monolithic software delivery and long-lived feature branches, new software features can …
Continuous Deployment
Continuous deployment is the practice of automatically promoting code changes to prod after they pass all automated tests in a continuous delivery pipeline.
Continuous Integration
Continuous integration is the practice of merging the code of all developers on a project and testing for conflicts continuously, as changes are made.
Controlled Rollout
A controlled rollout is a feature rollout with highly granular user targeting. It allows you to release new features gradually, ensuring a good user experience …
Customer Experience Metrics
Customer experience (abbreviated CX) is the experience your customers have with your brand. Customer experience metrics are organizational KPIs that help you monitor your customer …
D
Dark Launch
Dark launching is the term for releasing features to a subset of your users, seeing how they respond, and making updates to your features accordingly. …
Data Pipeline
Data pipelines automate the flow of data from one point to another. In a data pipeline, you start with defining how data is collected, and …
Do No Harm Metrics
Do no harm metrics are metrics that teams use to ensure nothing bad is happening to your teams metrics due to a feature rollout. Many …
E
Event Stream
An event stream is a series of data points that flow into or out of a system continuously, rather than in batches. Event stream processing …
F
False Discovery Rate
False Discovery Rate (FDR) is a measure of accuracy when multiple hypotheses are being tested at once. Ex: multiple metrics measured in the same experiment.
False Negative
A false negative result would indicate that “the change being tested has not improved the key metric significantly when in fact, the change generally has a positive impact on the underlying behavior.”
False Positive Rate
False positive rate (FPR) is a measure of accuracy for a test: be it a medical diagnostic test, a machine learning model, or something else. …
Feature Branch
A feature branch is a copy of the main codebase, where an individual or team of software developers work on a new feature until it …
Feature Experimentation
Ordinary software development is widespread with guesswork. Some amount of data, plus some amount of the product manager’s hunches, drives the decisions of what new …
Feature Flag Management
Feature flag management is the process of, well, managing feature flags. While feature flags are extremely useful for many purposes, they require an organized process …
Feature Flags
A feature flag, or feature toggle, is a software development tool used to safely activate or deactivate features for testing in production, gradual release, experimentation, and operations.
Feature Flags Framework
The feature flags framework is a revolutionary method of software development that allows individual features of a software product to be individually enabled or disabled. Feature flags allow features to be centrally managed from outside of the application, meaning they can even be turned on and off after they’ve already been rolled out to end users.
Feature Flags in DevOps
DevOps is a software development philosophy that combines the Agile development approach focusing on rapid delivery with a collaborative effort between the development and operations staff. This collaboration is key to creating a more efficient and adaptive product lifecycle, and formalizes the increased role that operations teams play in the service-focused landscape of today.
Feature Rollout Plan
A feature rollout plan is a process that allows the introduction of a set of new features to a group of your user base. A …
Feature Toggles
Feature toggles, also known as feature flags, are components of software development that allow specific features of an application to be activated or deactivated at will. This allows developers to safely “toggle” new features on or off for testing. Feature toggles have many uses beyond just testing, such as the targeted release of new features, experimentation, operations, or activation of premium features.
G
Guardrail Metrics
Guardrail metrics are business metrics designed to indirectly measure business value and provide alerts […]
H
Hypothesis Driven Development
If a software engineer wakes up in the morning, and hears something that sounds like rain outside her window, she thinks it might be raining. …
I
J
Jira Feature Flags
One of the biggest challenges that software development teams encounter when adopting feature flags is feature management, or how best to track new features across …
K
Kill Switch
In software development, a kill switch is a button that disables a feature if needed. This enables stakeholders to turn off broken features in production …
M
Mobile A/B Testing
A/B testing in general is the process of testing two variations of a resource by showing different versions to different users, then comparing the test …
Multi-Armed Bandit
In general, a multi-armed bandit problem is any problem where a limited set of resources need to be allocated between multiple options, where the benefits …
Multivariate Testing
Multivariate Testing is a method of experimenting with different variations of particular elements in a feature implementation, such as the headline, images, copy, etc in …
O
Observability
Observability is defined as the ability of the internal states of a system to be determined by its external outputs. With the unknown unknowns of …
P
Power Analysis
Statistical power is the probability of detecting an effect, given that the effect is really there. Power has a strong relationship with the number of …
Progressive Delivery
Progressive delivery is the logical next step for teams who have already implemented agile development, scrums, a CI/CD pipeline, and DevOps. It includes many modern …
R
S
Server Side Testing
Server-side testing refers to any type of testing – commonly A/B testing, but also multivariate testing or multi-armed bandit testing – that occurs on the …
Simpson’s Paradox
Simpson’s Paradox, otherwise known as the Yule-Simpson Effect, is a reversal paradox where the correlation found in each of several groups either disappears or even …
Smoke Testing
Smoke tests are a type of regression test which ensure that your most important, critical functional flows work. These tests should not encompass your entire …
Statistical Significance
When people discuss A/B testing, they commonly throw around the term “statistical significance” a lot. But what does this mean? In short, getting a statistically …
T
T-Test
A t-test is a type of hypothesis test which assumes the test statistic follows the t-distribution. It is determines if there is a statistically significant difference between two groups.
Testing in Production
Testing in production is the process of testing your features in the environment that your features will live in. It does not mean releasing untested …
Trunk-Based Development
Trunk-based development (TBD) is a branching model for software development where developers merge every new feature, bug fix, or other code change to one central …
Type I Error
A type I error (or type 1 error), also called a false positive, is a type of statistical error where the test wrongly gives a …
Type II Error
A type II error (type 2 error) is one of two types of statistical errors that can result from a hypothesis test (the other being …
U
Usability Testing
A key part of the software development process, usability testing provides invaluable feedback on the user experience of a product. Usability testing means conducting real-world …