Software testing is usually a point of friction for software development teams. Feature flags create an added layer of complexity that you need to account for. So how can we make the process of testing in a feature-flagged codebase easier?
Isolate Tests to Reduce the Scope of the System Under Test
Testing your code should be independent of all external factors, including databases, remote systems, and feature flagging frameworks. Software tests should be isolated and run fast without having to connect to any network.
For example, let’s say you use the traditional user-based testing approach. Using this method, you can target your internal teammates for their review, test your code with a small number of known users in production, and then release it to the world once you know its working. Although this approach works well for end-to-end tests in production, it is not recommended for all tests because of its fragility. If your tests need to have a full-on feature flagging system running just to run the tests, it violates the ritual of having independent tests. What if someone goes into the feature flagging framework configuration and changes the target test users in the UI? All of your tests will break. Your tests should be isolated to avoid this unnecessary delicacy and reduce the scope of the system under test.
Override Feature Flags in Automated Tests
Let’s look at an example of how feature flags should be used in an automated test. Let’s say you are a developer who is experimenting with giving people free shipping on your e-commerce site, and you are now testing the shipping calculator. If the feature flag is on for you, you get free shipping, and the feature flag is off for you, you get the existing calculated shipping cost.
One recommended approach is to make a custom feature flag abstraction, making it easy to mock out. In our example, you could have three tests. In the first test, you simulate the feature flag on, and the shipping cost is zero. For this test’s duration, if you get any requests asking if the feature flag is on (i.e., from the test), you say yes. In the second test, you simulate the feature flag off, and the shipping cost is not zero, and if any requests come in from the test asking if the feature flag is on, you say no. Finally, in the last test, you want to validate that you can go through the entire purchasing flow regardless of if the flag is on or off. With this approach, you are explicit in the test, and the test becomes much more self-documenting and descriptive.
Whenever any test runs in a system under test using feature flags, the system under test would fake out all variants of the experiment. Because it’s fake, you reduce the complexity of the different scenarios, which means faster tests. In this approach, you declaratively set the state of a flag for the duration of the test.
Test Framework Setup with Feature Flags
There are two types of tests that we care about in a feature-flagged codebase. In the first type, you explicitly say you want the flag to be on or off (i.e., the test to calculate free shipping). This type of test requires the state of the feature flag to be on or off. The second type of test that we care about is the ability to follow the entire flow regardless of if the flag is on or off. In this test type, you specifically want to verify that the state of the feature flag does not impact other functionality, such as the tax that you calculate. Although you can write two different tests here, you will lose the spirit of the test by doing so. You should create a helper function, like runTwice, once with flagOn and once with flagOff. Then, you pass in the name of the flag, and it executes twice, once with it faked on and once with it faked off. This way, you are avoiding duplication, and the test is more expressive.
Testing For the Win!
By isolating your tests from all other system dependencies, you make them run faster, thereby saving time and money. You can maintain the added complexity that a feature flagging system brings by using a custom feature flagging abstraction. This makes mocking your test much more manageable. For the test’s duration, you will either run the test with the feature flag on or off, without ever talking to the feature flag management system. This reduces the complexity of the test, while also making it faster and more efficient.
Learn More About Testing and Feature Flags
Software testing can be overwhelming when you don’t have a plan. With this implementation guide and the following resources, you’ll be on your way to testing like a pro!
- Watch my video on testing in production
- If you’re having trouble convincing your team to test in production, link them to my Breakup Letter to Staging
- Learn more about The Benefits of Feature Flags in Software Development
To stay up to date on all things testing in prod and feature flagging, follow us on Twitter @splitsoftware, and subscribe to our YouTube channel!
Get Split Certified
Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.
Deliver Features That Matter, Faster. And Exhale.
Split is a feature management platform that attributes insightful data to everything you release. Whether your team is looking to test in production, perform gradual rollouts, or experiment with new features–Split ensures your efforts are safe, visible, and highly impactful. What a Release. Get going with a free account, schedule a demo to learn more, or contact us for further questions and support.