Feature flag management systems are the infrastructure for implementing feature flags, also known as feature toggles. They can be built in-house or purchased. This glossary introduces the functionality that effective feature flag management systems deliver to software development organizations.
Feature Flag Management Vs. Feature Management
Before we discuss what feature flag management systems do, let’s clarify the difference between them and the underlying concept of feature management.
A feature flag management system is the infrastructure for implementing feature flags. This typically consists of an in-house feature flag platform or SaaS-based feature flag service, where SDKs, APIs, and other functionality allow development teams, product managers, and anyone else involved in DevOps and software delivery to efficiently decouple code deployment from feature release; make real-time changes at runtime; and observe feature-specific metrics as they go. These same systems may have advanced functionality such as A/B testing, user experience monitoring, and the ability to identify specific features causing user experience or system performance degradation. This is the focus of this glossary post.
On the other hand, feature management is the reason why teams implement feature flags in the first place. Feature management is a set of underlying software delivery lifecycle practices that hide code changes behind feature flags to decouple code deployment from feature releases and shift the unit of work from application versions to individual features. These practices come from the principles of agile and DevOps, where smaller work units that are easier to reason about and test are delivered more often. Feature management also makes continuous delivery practices such as trunk-based development possible because code committed to the main branch can be “off” by default until it’s ready for testing or release. Learn more about feature management in our glossary post, What is Feature Management?
Organize and Centralize Feature Flags
The most fundamental job of a feature flag management system is to provide organization and centralization, a consistent and well-understood structure for implementing and keeping track of feature flags. If you have one feature flag or a dozen, you probably don’t need much help with this, but as usage inevitably grows, you need something beyond a config file or a flat database table to manage the lifecycle of feature flags, observe metrics related to rollouts, and ensure that the risk of increasing technical debt by having too many flags doesn’t outweigh the benefits of using them. Let’s start with the organizational patterns that form the foundation of a well-built feature flag management system.
Projects or Workspaces Are Used for Separation, Not Meaning
Most feature flag management systems have the concept of projects or workspaces. This top-level organizational functionality deliberately enforces separation, isolating one set of feature flags from all others. Public workspaces are visible to all users in your organization, while private workspaces are only seen by those with a “need to know” they exist.
Projects or workspaces differ significantly from labels or tags, which we’ll cover in the next section. For now, resist the urge to create a long list of workspaces as if they were just an indentation level in an outline. Tags, or even a flag naming convention, are a far better way to convey grouping and add meaning.
The most important thing to know about projects or workspaces is that what’s in one workspace cannot be shared with or easily moved to another workspace. For that reason, avoid spreading feature flags for a single application across multiple workspaces. We’ll talk more about managing dependencies between feature flags later, but it’s important to know that dependency management can’t occur across workspace boundaries.
Pro tip: Don’t confuse what some feature flag management systems call a project with what you are used to calling a “project” in a planning process. That is better captured as a tag, which is our next topic.
Tags and Ownership: the What, Why and Who of Feature Flags
In feature flag management, tags or labels add meaning to flags beyond what can fit into the feature flag’s name and description. They help you find and filter groups of flags quickly. The most common uses are describing what part of your app the flag operates in and which initiative or purpose it is related to. For example, a flag might have the tags [frontend] and [rebranding] or [auth-layer] and [ops-toggle].
Ownership refers to the person or team that maintains and is responsible for a feature flag. Ownership can be a single user, several users, or one or more groups. Groups provide more resilience as team members come and go or shift to other initiatives. Because ownership is closely related to permissions and notifications, resist the urge to make everyone an owner of everything.
Environments: Where Rollout Rules and Permissions Live
Within each workspace, you will have multiple environments. These correspond to the lower and upper execution environments where your code will execute during your software delivery lifecycle. Environments allow flags to enforce different rules in staging and production, for instance.
Before feature flags, environments often corresponded to a different branch in the source code repository. Looking in GitHub, you would have multiple copies of the same release, forked off on different dates. Plus, there was a high cost to keep track of how they differed, decide what code to merge or cherry-pick between branches, and ensure that all functionality was genuinely ready to roll out to the user base in production.
Environments in feature flag management simplify this with separate feature flag targeting rules for each environment. Rather than using different branches in GitHub to control what code executes in each environment, feature flags control which parts of a single main branch are exposed in each environment, and even which config parameters that code executes with.
With fewer code branches to reason about (let alone a separate feature branch for each code change), it becomes much easier to do “real” Continuous Integration/Continuous Delivery. Once the continuous delivery pipeline begins, the same artifact can be promoted sequentially through various environments. Work in progress can execute in dev and test but be kept “dark” or off, in production.
Environments are thus the final big organizing unit under a workspace. Think of them as the container where the targeting rules and permissions for altering flags live. If you have a dev or integration environment, you might allow anyone to turn on a feature flag there so work can move with the least friction. For staging, maybe you only enable flags for code changes already marked ready for release by a regression testing suite. For production, you would do well to require approvals and/or limit who can change flag rules.
Feature Delivery and Control
While organization is the foundation of feature flag management, the next layer of functionality is feature delivery and control. This spans from the management tool UI and APIs that let you create and change targeting rules to the SDKs or OpenFeature providers that are embedded in your applications that deliver decisions and capture outcomes at runtime. In the middle of that span, the most performant and reliable systems rely upon a combination of content delivery network (CDN) for fast SDK rules initialization at the edge and real-time streaming to push instant rule changes to every server or client-side SDK in less than a second.
Let’s briefly consider each of the components of feature delivery and control.
Intuitive Management UI and APIs
Intuitive management user interfaces and APIs are a big part of the reason why feature flag management has gone mainstream. Whether you call them feature flags, feature toggles, feature flippers or feature gates, some sort of feature switches evaluated at runtime have existed for decades. What didn’t exist were intuitive and reliable tools you could just pick up and start using.
Any feature flag management system worth buying (or building!) must have an intuitive UI and API so that nobody (except perhaps someone who works for LaunchDarkly or Split) wakes up in the morning to be a feature flag specialist. We all use feature flag management to accomplish something else. We live in exciting times! Continuous delivery and trunk-based development are making it easier and safer to move fast. What’s more, our teams are often smaller and more diverse, intentionally blending product managers, designers, and software developers into a single team that is given both greater responsibility and autonomy. Think of an intuitive management interface as the opposite of opening a ticket for a specialist to configure something you aren’t qualified to touch. When things go wrong, you don’t want to file a ticket, you want a member of your own team to quickly find and press the right kill switch.
SDKs and OpenFeature Providers
In feature flag management, a good SDK isn’t just a faster way to integrate with an API. A good SDK also improves speed and safety. Depending on implementation and whether you are more worried about leaking rules or customer data, it can improve security as well. Feature flag SDKs typically follow three patterns: 1) server-side SDKs that resolve rules locally, 2) client-side SDKs that resolve rules locally, and 3) client-side SDKs that send user attributes upstream to be resolved by rules held at the edge or origin. The first two patterns eliminate the need to send customer data across the network, though some systems still do so. The third pattern eliminates the need to send rulesets down to the client-side, which keeps the rules more private in exchange for allowing customer data to leave your app. OpenFeature is an open-source project that promotes a standard SDK per programming language. That SDK is then supported by a provider specific to the feature flag platform of your choice.
The Role of CDNs and Streaming in Feature Flag Management
For feature flags to deliver real-time control at runtime, there must be a way for applications to initialize SDKs quickly and be updated instantly if the rules change while the app is running. A content delivery network (known as CDN) is purpose-built to keep a fresh copy of a digital asset very close to each consuming user. In this case, the user is actually the feature flag SDK.
The CDN picks up changes at the “origin” (i.e. the backend or core infrastructure of the feature flagging platform) and distributes them globally to points of presence (POPs). When an SDK initializes, it pulls that data from a nearby POP, thus reducing the latency of that fetch so initialization can finish faster.
If your feature flag platform uses a CDN but does not have a streaming service for pushing changes in real time, you are limited to polling for changes at an interval. Polling is costing someone money (you, the provider, or both), so be aware that choosing a polling-only approach will force a constant trade-off between responsiveness and operational cost.
Feature Measurement and Learning
Feature measurement and learning is a term coined by Split for the capabilities that are available to engineering teams and product managers when feature-flag specific metrics are calculated by the same system that controls rollouts of your new features (or any new config or code change you perform during day-to-day operations or system migration).
With a few notable exceptions in very large and well-funded organizations, in-house feature flag implementations don’t usually tackle the measurement and learning use cases. Historically, only those with large in-house experimentation teams that built their own A/B testing infrastructure could calculate metrics for feature flags in real-time. Most other organizations would run an A/B test or even a multi-stage rollout with flags, and then turn to the data scientists to perform analysis after the fact.
Feature Rollout Monitoring Without Writing Feature-specific Instrumentation Code
While Split initially launched feature and measurement capabilities in 2018 to support feature experimentation (aka A/B testing focused on product and engineering teams rather than marketing use cases), they realized that these same teams were more interested in rollout monitoring and assisted triage than traditional experimentation functionality. Why? In any given week, it’s far more likely that an engineering team will be rolling out several new features that need to get to market quickly and safely than they are to be rolling out several versions of the same feature to see which one works best.
When a feature flag management platform can automatically monitor each feature rollout for feature-specific system performance degradations, it frees up development teams to build new features faster for their end users without requiring them to write feature-specific instrumentation to detect and resolve performance issues in the early stages of rollouts.
Getting Value From Feature Flag Management
Progressive Delivery
Having a fully-featured feature flag management system enables you to go beyond a simple on/off feature rollout to adopt ring deployments, canary releases, and other gradual rollout patterns. Progressive delivery is effective when the feedback loop (i.e. health checking or a guardrails function) is built-in so it doesn’t add additional overhead to the team. If everyone has to stop what they are doing to manually check the health of a gradual rollout at 5% and then again at 20% and 50% (etc), then the “cost” of gradual rollouts can eat away at the benefits of limiting the blast radius in the first place. Feature rollout monitoring makes efficient progressive delivery possible.
Dynamic Configuration (aka Remote Config)
While it is sometimes overlooked, the ability of feature flags to decide not just “on” and “off” or A/B/n, but rather to also carry config parameters with each flag treatment or variation, opens up a ton of value by letting engineering get more done with less effort. And, by letting engineering hand off certain changes to be made directly by product with no ongoing engineering effort at all. The use cases for dynamic config all stem from the benefits of a one-time implementation that yields an unlimited number of (what would have been) code changes without a single additional code deployment. The one-time effort is to write your code and your SDK calls to expect parameters to arrive as a payload from the feature flag. Once that is done, those parameters can be edited in the feature flag management UI or via API.
Staying in Control Despite Mobile App Store Approval Delays
If you are shipping mobile apps for Android or iOS, you can retain greater control of your end users’ experience and maintain the ability to instantly roll back a problematic new feature by leveraging feature flags. Remember that app stores expect you to be transparent about using feature flags. As a general rule, they expect you to have all of the features “on” for them which you might turn on for users before your next app store submission.
Taming Technical Debt
Technical debt is inevitable to some extent, but the best teams keep it at bay with a mix of solid day-to-day workflow practices and periodic cleanups. When it comes to feature flags, the last thing you want is an ever-growing collection of them in front of an ever-aging collection of no longer active code that you are forced to reason about each time you visit a code module. The beauty of a real feature flag management solution is that you get things like flag status and traffic metrics that tell you whether a flag has been in a particular state for a prolonged period, where traffic no longer passes through parts of your code.
One idea that initially seemed promising was to scan code repos and capture all “code references” containing feature flags. Those references could then be imported into the management UI so that flag-to-code relationships can be easily looked up. Several feature flag platform providers added this capability, only to find that teams were generally uncomfortable exposing chunks of their source code to a third party’s SaaS system (other than GitHub, of course, where many of them expect their source code to be). They preferred to leverage an existing separation of concerns boundary: feature flags managed by feature flag platforms <> work items managed by task systems. Most feature flag management platforms support an association between feature flags and work items in Jira orAzure DevOps. From there, every flag is already just a click or two away from existing links between those work items and the commits that put the feature flag code into the repo (or took it out if it’s already been cleaned up). Other metadata, such as the developer who made the commits and the CI/CD pipeline runs that included the code, are also found as fields or links in those work items.
As Much or as Little Governance as You Need
If you are on a lean team in an unregulated industry, you’ll still like the approvals flow of a feature flag management platform, because you can use it to ask for an optional peer review. You can easily route your proposed flag rules to a colleague along with a title and comment and they can approve them and put them into effect, or reply back with feedback.
If you need to enforce restrictions on who can see which workspaces, who can change the state of a flag, or even which API keys can perform which actions, you’ll find all of that in a mature feature flag management system.
Regardless of your appetite (or legal obligation) for strict governance, you’ll probably appreciate the way feature flag management platforms keep audit trails for changes to objects (feature flags, user segments, metric definitions) and administrative actions, such as adding or deactivating a user, creating or rotating an API key or configuring an integration.
Summary
We covered a lot of material in this glossary post, so it’s quite a bit longer than many of the others here. Our goal was to provide an in-depth overview of feature flag management systems so you’ll get more value out of any time you spend specifying, implementing or adopting one.
Here’s a quick review of what we covered:
- Organization and Centralization: Describes how these systems provide structure for managing and tracking feature flags, including the use of projects or workspaces for organization.
- Tags and Ownership: Explains the role of tags in adding meaning to feature flags and discusses ownership responsibilities.
- Environments: Details the use of environments for managing rollout rules and permissions in different execution environments.
- Feature Delivery and Control: Explores the functionalities of intuitive management UI, SDKs, and OpenFeature providers in facilitating feature delivery and control.
- Feature Measurement and Learning: Discusses the importance of measuring feature-specific metrics and learning from feature rollouts for engineering teams and product managers.
- Progressive Delivery and Dynamic Configuration: Highlights the capabilities of feature flags in enabling progressive delivery and dynamic configuration, which contribute to efficient software development practices.
- Taming Technical Debt: Explains how feature flag management systems help mitigate technical debt by providing insights into flag usage and code dependencies.
- Governance and Compliance: Touches upon the governance features of feature flag management platforms, including approvals workflows, access restrictions, and audit trails.
