DORA continues to be the benchmark for evaluating performance and efficiency of software engineering teams. But let’s be real for minute: The way we work has evolved since DORA was initially established ten years ago, and it might not be capturing the full picture.
Yesterday, the unit of measurement was the application. Today, it’s the feature. Back then, deploy and release were the same thing and teams were just looking to ship as fast as possible. Now, rather than test features on a long-lived branch, feature flags allow us to validate in a live production environment without fully releasing to customers. It’s a totally different world.
Because feature flags have changed the way we deliver software from the ground up, we should rethink a few aspects of the DORA metrics. In my opinion, there’s one metric in particular that doesn’t entirely sync with today’s modern work streams, and that’s Deployment Frequency.
What is it missing? How can we go further as teams?
Before I dive too deep, let’s first review what Deployment Frequency means. If you need a quick refresher of all four DORA metrics, be sure to check out my introduction to DORA metrics here.
What Is Deployment Frequency?
Deployment Frequency is one of four DORA metrics. It measures the frequency of new code being deployed into production and serves as a common measuring stick for engineering velocity. Many teams use the term “delivery” to mean code changes that are released into a pre-production staging environment, while “deployment” is reserved only for production environments.
You can see how Deployment Frequency doesn’t have the same weight it did before feature flags were a big part of the engineering work flow.
From Deployment Frequency to Value Delivery
With feature flags, Deployment Frequency sky rockets. Of course it does: We can fire off more into production without turning features ON. But, just because we’re shipping more often, it doesn’t mean we’re maximizing customer value. In fact, 70% of the features we put into production end up having zero or a negative impact. So what are we truly delivering?
Let’s look at just one feature for example: You could turn it ON and OFF multiple times throughout its lifecycle. Should you account for every little change? What about measuring your change frequency at every stage of a progressive rollout, from 1% to 100% of the customer base? Tracking these ongoing changes in production might get tedious (and meaningless).
Deploying often isn’t enough. Sure it’s a necessary condition for engineering greatness, but we need to go further. We should also be focused on how quickly it takes to turn ON a worthwhile feature to our entire customer base. From no value to full value–this should be the next metric that DORA captures. After all, the speed at which meaningful changes reach customers means more than those that don’t.
Accelerated Learning Is the Future
Feature flags help us ship faster, but how can we improve the success rate of features going into production? Sure we can increase the volume of changes to bring in more wins, but whether that’s 100 features a day or 1,000, we’re still faced with the odds that 70% will inevitably fall flat.
To further improve our hit rates, we need to go beyond boosting the speed and volume of features; we need to accelerate the pace of learning at every stage of the feature lifecycle. Capturing this accelerated pace of learning is the future of feature delivery.
So, then how do we gain the knowledge to build the right things quicker? How can we increase our hit rate from 30% to 50%? This is where feature management with measurement & learning capabilities come into play.
Solutions like the Split Feature Data Platform™ accelerate learning and provide two very important benefits. First, they can quickly mitigate features with negative impact. By monitoring every rollout with causal analysis, we can watch our dashboards for granular degradations that appear to just a small percentage of populations. We can catch even the slightest issues that an APM tool won’t even notice, allowing us to speed up the decision making process. In this scenario, we won’t find out a feature is problematic after it’s been ramped up to 50% of users, because we’d kill it after failing the first 1% percent. This greatly reduces the destruction of value.
But it doesn’t end there. There are two sides to this valuable token. The second benefit of feature management platforms with measurement & learning is the ability to accelerate features with positive impact, too. Let’s say we’re seeing significant improvements to a new feature’s engagement at just 1% of users: Why wait to slowly ramp it up to your entire customer base? Get out it sooner, and realize that positive impact faster.
The feature-level observability you gain from platforms like Split is a game changer for accelerated learning. If only DORA could capture the pace at which we learn and improve.
In Conclusion
Eliminate the time you waste slowly ramping things up that aren’t working, and speed up the features that show value early on. By adopting modern feature management with measurement and learning capabilities like the Split Feature Data Platform, your engineering team will be at the center of every valuable product decision, focal point of outcome driven development, and pretty soon DORA will be rewriting the rules around you.
More on Rethinking the DORA Metrics
I speak more in depth about reimagining the DORA metrics and leveraging feature flags in a recent podcast interview on Dev Interrupted. Be sure to listen here.
Switch It On With Split
The Split Feature Data Platform™ gives you the confidence to move fast without breaking things. Set up feature flags and safely deploy to production, controlling who sees which features and when. Connect every flag to contextual data, so you can know if your features are making things better or worse and act without hesitation. Effortlessly conduct feature experiments like A/B tests without slowing down. Whether you’re looking to increase your releases, to decrease your MTTR, or to ignite your dev team without burning them out–Split is both a feature management platform and partnership to revolutionize the way the work gets done. Schedule a demo to learn more.
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.