The Unpredictability of Releasing Software
Releasing features is a risky job. Every time you do it, you enter a coal mine of unpredictability. There’s no sure way to know what you’ll uncover until you dig in. Without the right approach and tools, software developers are left crossing their fingers, hoping their rollouts won’t end with a crash. They can release late at night when there’s little site traffic, but that doesn’t eliminate the chance of a costly, stressful rollback. And, the last thing anybody wants to do is dig their way out of disaster on little sleep.
Does it sound as if releasing software is like working in a coal mine? No, software engineering is nowhere near as dangerous. However, the age-old method that miners once used to avoid risk now somehow applies to modern software development. The technique is called a canary release. If you’re not already doing it, it’s about to change your life.
What’s a Canary Release?
“The canary in a coal mine” is a commonly used phrase with a long history of risk mitigation. It originates from the past when miners used to carry birds into underground tunnels. If deadly gasses such as carbon monoxide were present, the canary would be the first to die. Pretty gruesome but it gave miners a clear warning, so they had enough time to escape catastrophe.
Canary releases for software teams work in a similar manner, except no birds are harmed in the process. Instead, they’re used to test features on just a tiny percentage of customers. In a canary release, features are rolled out to a small subset of users (usually 5%) in order to assess how they behave within an overall application system. While these “canaries” interact in the wild, developers can carefully note any early indication of danger. And If there’s a performance issue, the problem-causing feature is isolated and quickly turned off with feature flags. It’s a simple flip of a kill switch, and there won’t be a dead bird in the middle of your online store to send your customers fleeing.
When canary releases are successful and everything safely checks out, the next step is to slowly roll out the features to more and more users. This happens little by little, until everyone gets the new experience. The term for this percentage release is called gradual rollout.
Feature Flags Help You Safely Contain Your Canaries
The easy-to-use on/off switch that makes canary releases and gradual rollouts possible is a feature flag. How do feature flags work? In the application of a canary release, developers attach these “switches” (sometimes referred to as feature toggles) to new features which give them the ability to toggle between deploy and release states.
When in a deployed state, or dark launch, a feature is introduced to the software’s underlying infrastructure without being turned on to the end user. Once deployed, the feature can be turned on for developers, testers, and other internal stakeholders to validate in production without exposing it to customers.
Then, by turning on the feature flag for a small percentage of users, features can gradually transition to a release state while you watch for performance degradation or bugs that only show up in production. Now your canary can fly, but not without a short leash. The instant it takes your canary to reach customers, it can take the same amount of time to pull it right back if it breaks the experience.
Not only do feature flags require minimal coding for on/off switches, they can be easily set up to reach a specified percentage of users.
By using a feature flag to test canaries and do gradual rollouts, a release is exactly how it sounds—healthy and stress free. Here’s how these tools and techniques can benefit your software team.
Limit the Blast Radius
Always dip your toes in to test the water, rather than make waves that could put your software team in a precarious situation. Canary releases and gradual rollouts eliminate the need for late-nights and big bang batches. Constantly release little by little, at any hour of the day without worry. If there’s a blow up, it’ll be small and manageable.
Isolate the Problem Children
If you’re a teacher, you wouldn’t let one problem child disrupt the whole classroom. You’d send them to the principal’s office. Same applies to feature releases. Contain your problem-causing features, so they don’t spoil the entire experience. If there’s a bug, the entire application shouldn’t have to suffer. With feature flags, they won’t.
Cut the Cost of Catastrophe
Small, isolated problems are easy to fix. It’s just “flipping a toggle,” or hitting a kill switch. The opposite is a mess of entangled code, where you’re unable to locate the problem fast enough, and it causes an outage. This will churn customers and cost companies thousands by the minute. Canary releases with feature flags prevent this from happening.
Learn Fast, Refine Faster
A feature management and experimentation solution like Split will attribute data to every feature flag. This means you can canary test, follow contextual performance metrics at the feature level, and know where the problem lies quickly. As a result, failure isn’t a bad thing. It’s part of the experimentation process, where you can learn, hypothesize, and rapidly innovate from there.