We have updated our Privacy Policy as of September 22, 2023, for more information click here

Glossary

Feature Delivery Glossary

Your guide to progressive delivery and experimentation concepts in software development.

GLossary

A/A Testing

A/A tests can help you to ensure that your A/B testing process is working properly and your A/B test results are telling you exactly what you think they are.

A/B Testing

A/B testing can be very similar, with only a change in button color, or very different, with a total change in the way a feature behaves.

Canary Deployment

A canary deployment, or canary release, is a deployment pattern that allows you to roll out new code/features to a subset of users as an initial test.

Change Advisory Board

A change advisory board is a collective of representatives from different departments within a company who runs formal change management processes.

Chaos Engineering

Chaos engineering, also known as chaos testing, provides a method and tool-set to deliberately introduce failures and outages in a system.

CI/CD

CI/CD is the acronym in software development for the combination of continuous integration (CI) and continuous delivery (CD).

Client-Side Testing

Client-side testing refers to any type of testing, commonly A/B testing, multivariate testing or multi-armed-bandit testing occurring in the user’s browser.

Continuous Delivery

Continuous delivery is a software delivery process allows devs to release software updates to the production environment and to end users at any time.

Continuous Deployment

Continuous deployment is the practice of automatically promoting code changes to prod after they pass all automated tests in a continuous delivery pipeline.

Controlled Rollout

A controlled rollout allows the release of new features gradually, ensuring a good user experience before releasing them to larger groups.

Dark Launch

Every product manager dark launches (or should). In a dark launch, features are released to a subset of your users to enable testing before a wider deploy.

Data Pipeline

A data pipeline automates the flow of data from one point to another. Defining how data is collected, and in what schema it should be collected.

Edge Computing

One observable benefit of edge computing is an improvement in page loading time, but actually any type of internet communication can benefit.

Event Stream

An event stream is a series of data points that flow into or out of a system continuously, rather than in batches.

False Discovery Rate

False Discovery Rate (FDR) is a measure of accuracy when multiple hypotheses are being tested at once. Ex: multiple metrics measured in the same experiment.

False Negative

A false negative result would indicate that “the change being tested has not improved the key metric significantly when in fact, the change generally has a positive impact on the underlying behavior.”

False Positive Rate

What is false positive rate, and how is it calculated? How does it compare to other measures of test accuracy, like sensitivity and specificity?

Feature Branch

A feature branch is a copy of the main codebase, where an individual or team of software developers work on a new feature until it is complete.

Feature Experimentation

With Feature experimentation, when anyone can run experiments and test product features, this creates a culture of experimentation, inspiring everyone from all teams to test their ideas and gather data.

Feature Flags

A feature flags is a software development tool used to safely activate or deactivate features for testing in production, experimentation, and operations.

Feature Flags Framework

A feature flags framework is a tool for software development that allows individual features of a software product to be individually enabled or disabled.

Feature Rollout Plan

A feature rollout plan allows the introduction of a set of new features to a group of your user base, often to a limited user set initially.

Feature Toggles

Feature toggles let developers “toggle” features on and off without releasing new code. They’re an alternative to feature branches with many use cases.

Fixed Horizon

Fixed Horizon is a statistical testing method where sample sizes and your experiment goals are defined in advance.

Guardrail Metrics

Guardrail metrics are business metrics designed to indirectly measure business value and provide alerts about any potentially misleading or erroneous results and analysis.

Jira Feature Flags

Using Jira can enhance the feature release process even further, providing real-time feature status from ideation to code deployment.

Kill Switch

With a kill switch, when a feature breaks in production, you can turn it off immediately while your team analyzes the issue.

Mobile A/B Testing

A/B testing for mobile apps is about as similar to standard A/B testing as mobile app development is to standard software development.

Multi-Armed Bandit

A multi-armed bandit is a problem to which limited resources need to be allocated between multiple options, and the benefits of each are not yet fully known.

Multivariate Testing

Multivariate testing is a method of experimenting with different variations of elements in a feature to discover which variations will drive user behavior.

Observability

With the unknowns of your software's failure modes, you want to be able to figure out what's going on just by looking at the outputs: you want observability.

Power Analysis

Power analysis is the process of estimating how many users you will need in order to detect an effect of a given size, or how small an effect you can detect.

Progressive Delivery

Progressive delivery uses feature flags to increase speed and decrease deployment risk, and uses a gradual process for rollout and ownership.

Pull Request

A pull request is when you are requesting for someone in your org, possibly a senior engineer, to look at your code, review it, and pull the changes in.

Segment

Segments are used to organize targeted users into groups to facilitate setting up incremental product releases or experimentation.

Server-Side Testing

Server-side testing refers to any type of testing, commonly A/B testing, multivariate testing or multi-armed bandit testing occurring on the web server.

Simpson’s Paradox

When finding out if an admissions program is biased, a surgery is more successful, or an A/B test variant is superior, Simpson’s Paradox may come into play.

Smoke Testing

Smoke testing is a type of regression test which ensure that your most important, critical functional flows work as intended.

T-Test

A t-test is a type of hypothesis test which assumes the test statistic follows the t-distribution. It is determines if there is a statistically significant difference between two groups.

Treatment

A treatment is the version of the UI (user interface) that the user sees. You can set a treatment in the UI to be on or off.

Trunk-Based Development

Trunk-based development enables continuous integration and, by extension, continuous delivery by creating an environment where commits to trunk naturally occur multiple times daily for each programmer.

Type I Error

A type I error is a type of statistical error where the test gives a false positive result, when a perfect test would report a negative. It is one of four possible results from a hypothesis test.

Usability Testing

A key part of the software development process, usability testing provides invaluable feedback on the user experience of a product.

Create Impact With Everything You Build

We’re excited to accompany you on your journey as you build faster, release safer, and launch impactful products.