Come join our happy hour in Seattle! – Click here.

Look Away! Developer Horror Stories

Contents

Split - Blog-Horror-2160x1080-1-1

Be afraid developers, be very afraid.

Of zombies, knife-wielding slayers, or bloodthirsty sharks?

No! Of brutal bugs, demons from merge hell, and the dreaded Count Rollbackula!

Well maybe not afraid, just very, very tired.

What really haunts a developer’s or software engineer’s dreams? Just ask any developer and there’s a good chance they have a horror story to tell. A rollback while they’re on vacation. A nightmare hiding in the code, and staying up all night to solve it. Slack blowing up when something broke, and the big meeting with the CEO to explain what happened. Boo!

The following haunted tales are from current Splitters’ past lives, before they escaped into the open arms of Split. And they are all too real.

Do you have a developer horror story of your own? Share it with us on LinkedIn or twitter at #StopTheDevHorror.

The Homegrown Feature Flag of Horrors

When I worked at a cloud-based content management company, we used to store feature flags in our main database. These feature flags were from a homegrown system we created internally. It was pretty flaky. Once, a developer added a feature flag in the code in a really high-traffic area.

The feature flag was for a piece of code that was also using the database. When the homegrown flag was turned on—all at once—the new code hit the DB so hard that it brought the DB down. Because the DB was down, we couldn’t turn off the feature. The monster couldn’t be killed.

We had to keep the app down, go manually in the DB to turn the feature off, and then restart our application. Lesson one: Homegrown feature flags are easy to mess up. Lesson two: FF systems should be disconnected fully from your application and its direct dependencies. Lesson three: Percentage rollouts are a good idea, even more so if you can have metrics on your DB in association with your rollout.

Pierre-Alexandre Masse, VP Engineering

Night of the Living Dread

My last product used text files on disk to control features. These files were embedded into the EC2 images. When a new instance was launched, we needed to sync them, but some were region-specific or tests that were only running on certain servers.

As a result, we regularly found that three random machines had the wrong property set out. With a fleet of hundreds of machines, we had no idea which ones had the issues. That’s about as scary as it gets.

Ron Pierce, Engineering Leader, Experimentation

Something Wicked This Way Comes

When I worked at a customer relationship management corporation, we decided to take the whole company to a theme park. It was supposed to be a day of mandatory fun. Upon landing, and as part of a release, we turned a blog page on. However, the homegrown feature flag solution didn’t have a good way of testing all cases or variations of a flag. So the application and world went down after switching the flag on. It was an integration issue that surfaced when turning the flag on.

As a result, unfortunately this team of developers had to spend most of their fun day in a coffee shop solving the problem.

Split localhost mode is a good way to enable earlier testing to avoid a similar fate.

Patricio Echague, Founder and CTO

Nightmare on Dev Street

Do you have a developer horror story of your own? Share it with us on LinkedIn and Twitter using #StopTheDevHorror.

Get Split Certified

Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.

Switch It On With Split

Split gives product development teams the confidence to release features that matter faster. It’s the only feature management and experimentation solution that automatically attributes data-driven insight to every feature that’s released—all while enabling astoundingly easy deployment, profound risk reduction, and better visibility across teams. Split offers more than a platform: It offers partnership. By sticking with customers every step of the way, Split illuminates the path toward continuous improvement and timely innovation. Switch on a trial account, schedule a demo, or contact us for further questions.

Want to Dive Deeper?

We have a lot to explore that can help you understand feature flags. Learn more about benefits, use cases, and real world applications that you can try.

Create Impact With Everything You Build

We’re excited to accompany you on your journey as you build faster, release safer, and launch impactful products.