“But it worked in staging” …..
We have all said this at some point in our careers. A critical release is on the line, we are behind our deadlines, we push to production and something breaks. Under stress, the words tumble out: “But it worked in staging”.
The myth of staging
But, even if it had worked in staging, we couldn’t have been confident of it working in production.
Because production is its own beast.
Because production infrastructure is more powerful than staging.
Because production traffic can’t be reproduced in staging.
Because copying production data to staging is a security nightmare.
So, why have a staging environment? In fact, why have 2-4 pre-production environments each designed to take the risk out of software delivery when we will inevitably say: “But it worked in staging”.
Getting to the final act: production
Instead, the answer is to:
- invest in CI/CD pipelines to get to production as soon as possible
- give developers the ability to gradually rollout features in production behind a feature flag
- invest in automated QA that detects problems in production.
If you do these steps, you will be able to get rid of practically all your pre-production environments. And if you think this is costly, imagine the savings in infrastructure and human costs if you could remove all those environments.
And the faster your team can get from ‘it worked in staging’ to ‘it works in production’.
Stay up to date
Don’t miss out! Subscribe to our digest to get the latest about feature flags, continuous delivery, experimentation, and more.
It’s known that modern companies, especially the ones that have adopted agile principles, have achieved a faster degree of delivery cadence, deploy many times per day, diminished the risk of each release, and lowered the time to fix defects in production environments. Feature flags are one of the few key…
Great teams don’t run experiments to prove they are right; they run them to answer questions. Guard against wishful thinking and hidden biases with this shortlist of core principles for productive online controlled experiments. (Video, transcript and screenshots from talk given at Pinterest HQ on September 13, 2019)
Last week we walked The Path To Progressive Delivery. This week, we go deeper. What goals can we meet with the Four Shades of Progressive Delivery? After you read or watch this week’s episode, you will be better equipped to choose a Progressive Delivery approach that works for you. My…