6 minute read
With Halloween behind us and the holiday shopping season officially kicked off, we all know what that means: code freezes! If you’re an engineering or product leader, you know that also means the annual code freeze is upon us. It’s very common for engineering teams to start a code freeze prior to a time of known high traffic. While a code freeze is a traditional way to ensure site or app stability during this high-traffic period, it also has significant impacts on development velocity and roadmap progression. What if I told you there was a better way? That in fact, you never need to live through another code freeze, but you could still avoid the risk to your organization of continuing to release over the holidays. Follow along and I’ll explain how.
Code Freezes Contribute to Less Stability in the Long Run
When you are in a code freeze, code keeps being written but it doesn’t get integrated or tested with the main branch. It just waits until the freeze is over, which means the longer the freeze, the bigger the build-up. This process introduces a lot of risks and instabilities which slow down integrations well beyond the freeze. Teams need to figure out without integration if things are broken because of their change or someone else’s change.
Backlogged Commits Actually Increase Risk
Another problem with code freezes is the more commits you try to release at a time, the more risky the release is. If you have a code freeze, unless you give your developers something else to work on (for example planning or design work), which is rare, they’re going to keep creating commits that just aren’t going to get released. This means that the next time you do try to release, you’ve built up a large backlog making that next release very risky. On top of this, when something does go wrong, it’s much harder to point to a single part of the code as the culprit. Because there are 50 commits instead of 10, you have 50 changes your teams have to look through them all, and the chances that the problem is actually an interaction between two different commits is also higher.
Your Dev Teams May Ship Risky Code Under the Wire
The other major problem is what we call “racing the train”, which is what happens when your developers are working on something and they know there’s a code freeze coming. This is particularly relevant if a deadline is approaching and your team really needs that something to get released before the end of the code freeze. Developers are then incentivized to cut a few corners to make sure it gets out. Maybe they add fewer tests, maybe they skip the manual testing, maybe they just don’t fully think through if there are any greater implications in those merge conflicts you just resolved. Whatever the case, any developer rushing to get something into that last release is much more likely to have caused a bug than they normally would be.
Code Freezes Can Be Beneficial to Your CI/CD Pipeline
CI/CD is about creating a continuous flow. Code freezes break that flow. These are opposing approaches. However, there are cases where the risks outlined above may be worth the trade-off. If you have a big customer conference where you’re hoping to demo new features, it may not be worth having more risk at a time when it’s critical that your demos work, and while more eyes are on your product. Another example might be if you’re running some sort of maintenance that will put a lot of strain on one part of your system, it may be worth trying to make sure nothing is happening in another part. There are better ways to mitigate the risks, and the best one is feature flags.
Feature Flags Reduce the Need for Code Freezes
Code freezes were created to reduce the risk of a bad release. Companies that don’t embrace constant changes end up slowing down over time compared to the companies that consistently invest in tooling and processing to increase the speed of delivery and remove friction from it. Feature flags are one of the pieces of the deployment puzzle (along with leaner deployment tools, etc) to make sure engineers are safely pushing to production and decoupling pushes from releases.
The challenge for most organizations is to deal with the rate of changes and the risk of changes. Developers write code. Each code change is a risk. So organizations create testing processes to reduce the risk. The challenge is to manage the flow through this. Oftentimes, tests take a certain amount of time and so we have to group code changes to adjust to the speed of the testing part of the pipeline. As each code change presents a risk, the more changes we group together, the bigger the risk of something breaking. And once something breaks, then the pipeline is affected and everything slows down.
Feature flags reduce the risk of each code change. By putting the code change behind a feature flag, it essentially becomes independently managed. It reduces the code change risk to a much much smaller amount. It also avoids breaking and stopping the CI/CD pipeline. If a test fails with the flag on, it can be addressed on its own time as it doesn’t affect the user experience in production even if it is deployed. This is the cheapest, and most effective way of enabling flow at scale, while also accelerating product development and releases.
Kill the Code Freeze with Feature Flags
Using feature flags not only in times of high volume and high sensitivity, but year-round promotes a strong engineering culture. When your team writes code with feature flags, you are mitigating the risk in case something goes wrong and you need to roll back. You can do this easily with a kill switch, which allows you to instantly turn off a feature while your team troubleshoots the issue.
Learn More About Feature Flags and Continuous Deployment
Feature flags give you the ability to roll out your app in a controlled fashion to only the selected population, as well as mitigate risk in times of high sensitivity. Here at Split, we don’t write code without them!
- How to Implement Testing in Production
- Testing a Feature Flagged Change
- Continuous Integration vs. Continuous Delivery vs. Continuous Deployment