Big News! Split to be acquired by Harness. Read More


Feature Delivery Glossary

Your guide to progressive delivery and experimentation concepts in software development.


A/A Testing

A/A tests can help you to ensure that your A/B testing process is working properly and your A/B test results are telling you exactly what you think they are.

A/B Testing

A/B testing can be very similar, with only a change in button color, or very different, with a total change in the way a feature behaves.

Application Versions

Application versions refer to distinct releases or iterations of a software application, each representing a specific stage of development or enhancement. These versions typically feature updates, bug fixes, and new features, and are identified by unique version numbers or names.

Bayesian Statistics

Bayesian statistics employs Bayesian probability theory to model and update uncertainties about hypotheses. It involves combining prior beliefs with new evidence, using Bayes' theorem, to obtain updated and more informed probability distributions.

Canary Deployment

A canary deployment, or canary release, is a deployment pattern that allows you to roll out new code/features to a subset of users as an initial test.

Change Advisory Board

A change advisory board is a collective of representatives from different departments within a company who runs formal change management processes.

Chaos Engineering

Chaos engineering, also known as chaos testing, provides a method and tool-set to deliberately introduce failures and outages in a system.


CI/CD is the acronym in software development for the combination of continuous integration (CI) and continuous delivery (CD).

Client-Side Testing

Client-side testing refers to any type of testing, commonly A/B testing, multivariate testing or multi-armed-bandit testing occurring in the user’s browser.

Cloud Deployment Model

The choice of a cloud deployment model is a strategic decision that influences an organization's flexibility, scalability, costs, and security posture. Whether opting for a public, private, hybrid, or community cloud, it is essential to align the chosen model.

Configuration Flag

By incorporating configuration flags into their development workflow, teams can improve their deployment strategies, enhance testing capabilities, and deliver a better user experience. However, it's important to manage these flags carefully to avoid complexity.

Continuous Delivery

Continuous delivery is a software delivery process that allows devs to release software updates to the production environment and end users at any time.

Continuous Deployment

Continuous deployment is the practice of automatically promoting code changes to prod after they pass all automated tests in a continuous delivery pipeline.

Controlled Rollout

A controlled rollout allows the release of new features gradually, ensuring a good user experience before releasing them to larger groups.

Dark Launch

Every product manager dark launches (or should). In a dark launch, features are released to a subset of your users to enable testing before a wider deployment.

Data Pipeline

A data pipeline automates the flow of data from one point to another. Defining how data is collected, and in what schema it should be collected.

Database Migration

Database migration is the process of transferring data from one database system to another, often involving changes in database schema, platform, or storage format while ensuring data integrity and consistency.

Deployment Automation

Deployment automation enables teams to deploy faster, more frequently, and with greater confidence. By automating the deployment pipeline, organizations can achieve more predictable outcomes, reduce the risk of errors, and free up valuable resources to focus on developing new features.

Edge Computing

One observable benefit of edge computing is an improvement in page loading time, but actually any type of internet communication can benefit.

Event Stream

An event stream is a series of data points that flow into or out of a system continuously, rather than in batches.

Fallback Value

In various contexts such as programming, design, or decision-making, fallback values serve as backups or substitutes to ensure continuity or functionality when the preferred option is not accessible or applicable.

False Discovery Rate

False Discovery Rate (FDR) is a measure of accuracy when multiple hypotheses are being tested at once. Ex: multiple metrics measured in the same experiment.

False Negative

A false negative result would indicate that “the change being tested has not improved the key metric significantly when in fact, the change generally has a positive impact on the underlying behavior.”

False Positive Rate

What is a false positive rate, and how is it calculated? How does it compare to other measures of test accuracy, like sensitivity and specificity?

Feature Branch

A feature branch is a copy of the main codebase, where an individual or team of software developers work on a new feature until it is complete.

Feature Development

By adhering to feature development best practices and staying user-focused, development teams can successfully introduce new features that enhance the software product and provide value to users. Continuous learning, adaptation, and feedback integration are key to refining feature development processes.

Feature Experimentation

With Feature experimentation, when anyone can run experiments and test product features, this creates a culture of experimentation, inspiring everyone from all teams to test their ideas and gather data.

Feature Flag Management

Feature flag management systems are the infrastructure for implementing feature flags, also known as feature toggles. They can be built in-house or purchased. This glossary introduces the functionality that effective feature flag management systems deliver to software development organizations.

Feature Flags

A feature flag is a software development tool used to safely activate or deactivate features without modifying the source code or performing a new deployment. They decouple deploy from release, which enables trunk-based development, testing in production and progressive delivery.

Feature Flags Framework

A feature flags framework is a tool for software development that allows individual features of a software product to be individually enabled or disabled.

Feature Rollout Plan

A feature rollout plan allows the introduction of a set of new features to a group of your user base, often to a limited user set initially.

Feature Toggles

Feature toggles let developers “toggle” features on and off without releasing new code. Often used as an alternative to feature branching, toggles have a variety of use cases.

Fixed Horizon

Fixed Horizon is a statistical testing method where sample sizes and your experiment goals are defined in advance.

Flag Dependencies

Flag dependencies are criteria that must be met before a particular flag can be set or activated within a system, software, or process. These dependencies are often utilized in software development to control the flow of actions.

Flag Trigger (Auto-Kill Webhook)

An Auto-Kill Webhook is designed to automatically terminate or disable certain processes, tasks, or resources when triggered by predefined conditions. This functionality is used for managing and controlling system resources, enforcing security measures, or responding to errors in real-time.

Guardrail Metrics

Guardrail metrics are business metrics designed to indirectly measure business value and provide alerts about any potentially misleading or erroneous results and analysis.

Jira Feature Flags

Using Jira can enhance the feature release process even further, providing real-time feature status from ideation to code deployment.

Kill Switch

With a kill switch, when a feature breaks in production, you can turn it off immediately while your team analyzes the issue.

Merge Conflict

A merge conflict occurs in version control when changes made in different branches of code cannot be automatically merged due to conflicting modifications. Manual intervention is required to reconcile the differences and successfully integrate the changes into the codebase.

Mobile A/B Testing

A/B testing for mobile apps is about as similar to standard A/B testing as mobile app development is to standard software development.

Mock Testing

Mock testing is created to replicate the behavior of real components, enabling controlled testing of individual units and their interactions.

Modern Technology Stack

A modern technology stack integrates various technologies and practices to address the complete lifecycle of an application, from development to deployment and maintenance. By carefully selecting and managing these technologies, organizations can build applications that meet current needs.

Monthly Active Users

Monthly Active Users is a KPI that quantifies the number of unique users engaging with a digital product, platform, or service within a specified monthly time frame. It provides insights into user participation, platform popularity, and overall user engagement.

Multi-Armed Bandit

A multi-armed bandit is a problem to which limited resources need to be allocated between multiple options, and the benefits of each are not yet fully known.

Multivariate Testing

Multivariate testing is a method of experimenting with different variations of elements in a feature to discover which variations will drive user behavior.

Observability Tools

Observability tools are software solutions designed to monitor, analyze, and provide insights into the performance, health, and behavior of systems and applications.

Power Analysis

Power analysis is the process of estimating how many users you will need in order to detect an effect of a given size, or how small an effect you can detect.

Production Environment

The production environment serves as the platform where users interact with the product in real-world scenarios. Its management is a critical aspect of IT operations, demanding attention to detail, rigorous planning, and adherence to best practices to ensure reliability.

Progressive Delivery

Progressive delivery uses feature flags to increase speed and decrease deployment risk, and uses a gradual process for rollout and ownership.

Pull Request

A pull request is when you are requesting for someone in your org, possibly a senior engineer, to look at your code, review it, and pull the changes in.

Release Management

Release management is the strategic process of planning, coordinating, and controlling the deployment of software releases, ensuring they are delivered efficiently, reliably, and with minimal disruption to users or systems.

Release Testing

Release testing refers to the phase in software development where a software product undergoes thorough testing to ensure its readiness for deployment or release to end-users. This process validates that the software meets quality standards and functions as intended.

Rolling Deployment

A rolling deployment is a software deployment strategy that updates components or instances of a system incrementally, one at a time, to ensure continuous availability and minimal downtime. This approach allows the system to remain operational throughout the update process.


Segments are used to organize targeted users into groups to facilitate setting up incremental product releases or experimentation.

Server-Side Testing

Server-side testing refers to any type of testing, commonly A/B testing, multivariate testing or multi-armed bandit testing occurring on the web server.

Simpson’s Paradox

When finding out if an admissions program is biased, a surgery is more successful, or an A/B test variant is superior, Simpson’s Paradox may come into play.

Smoke Testing

Smoke testing is a type of regression test which ensures that your most important, critical functional flows work as intended.

Software Delivery Lifecycle

The software delivery lifecycle (SDLC) encompasses the entire process of planning, creating, testing, deploying, and maintaining software applications or systems. It involves stages such as requirements gathering, design, development, testing, deployment, and ongoing maintenance.


Staging refers to the process of deploying and testing software changes in an environment that closely resembles the production environment but is separate from it. This allows developers and testers to validate changes, assess their impact, and identify potential issues.


A t-test is a type of hypothesis test which assumes the test statistic follows the t-distribution. It is determined if there is a statistically significant difference between two groups.

Targeting Rules

Targeting rules are predefined criteria or conditions used to direct specific actions or resources towards particular targets within a system. These rules help determine how software functionality, such as advertising, content delivery, or user interactions, should behave.

Technical Debt

Technical debt is a metaphorical concept in software development that refers to the cumulative cost of choosing expedient, suboptimal solutions during the coding process. These shortcuts may lead to increased complexities, maintenance challenges, and future inefficiencies.

Test Environment

A test environment is a controlled setup or system used to conduct software testing, providing a simulated platform where software or systems can be evaluated under specific conditions to assess functionality, performance, and reliability before deployment.


A treatment is the version of the UI (user interface) that the user sees. You can set a treatment in the UI to be on or off.

Trunk-Based Development

Trunk-based development is a software development model prioritizing continuous integration by committing to a central branch, aiding continuous delivery used by experienced teams.

Type I Error

A type I error is a type of statistical error where the test gives a false positive result, when a perfect test would report a negative. It is one of four possible results from a hypothesis test.

Usability Testing

A key part of the software development process, usability testing provides invaluable feedback on the user experience of a product.

Winning Variation

Winning variation refers to the version of an asset that outperforms other variations in achieving a specific goal, such as higher click-through rates, increased conversions, or greater user engagement. It is identified through statistical analysis of experimental data.

Workflow Management

The systematic organization, coordination, and optimization of tasks, processes, and activities within an organization to achieve efficiency, collaboration, and streamlined productivity.

Create Impact With Everything You Build

We’re excited to accompany you on your journey as you build faster, release safer, and launch impactful products.