With a database of 50 million electronic components, Octopart is leading the way as the top data platform and search engine for electronic parts. As Octopart has grown, they needed to scale the way they approached making smart, data-driven decisions for their platform that increased revenue, while still providing a great user experience. Their solution? A/B testing via feature flags, with Split.

Guiding engineering decisions with an experimentation platform

Sam Bobb, Sr. Director of Data Science, chose Split after considering both self-hosted A/B testing solutions and building a tool in-house.

Our research showed there were too many details in an experimentation platform that, if implemented incorrectly, would have cost the team time, cost the business money, and led to inaccuracies.

Sam Bobb
Sr. Director of Data Science

As Sam has seen, too many times engineers develop their own in-house experimentation platform and think they are getting valid results. When they find out their implementation is incorrect to achieve their goals, it’s too late, and the company loses faith in testing-driven decision making.

Sam and his team decided it was best to use a hosted tool for A/B testing, and decided Split was the most trustworthy for experiment results. “There’s a big incentive for the developers of A/B testing tools to make UX and statistics choices that show users positive results all the time. Any tool that promises to ‘run until the experiment reaches significance,’ should send your data scientists running for the hills.”

I wanted a tool I could trust to do statistics correctly, in an honest and unbiased way, so our team didn’t have to second-guess the tool.

Sam Bobb
Sr. Director of Data Science

Octopart wanted a platform that was transparent about the data, and found that in Split. In addition, Split was “the nerdier option” with strong documentation, tutorials, and an API to support implementation by the Octopart engineering team.

Iterating and growing website conversion rate via experimentation

Octopart was interested in measuring their conversion rate from search and, in particular, how design changes impacted that conversion rate. They realized early on that little differences matter because they process a high number of conversions every month – their revenue comes from a large number of tiny transactions. When an electrical engineer finds a part on their website, they click through to the seller and the seller pays a cost per click.

When Octopart was an earlier-stage company with less traffic, the Product and Engineering teams often made changes to production without conducting a controlled experiment. However, as the company has grown this became high-risk because of the traffic volume and revenue at stake. Not unexpectedly, as the application became more complex, it also became harder to predict the impact to user behavior, user flow through the site, and ultimately conversions. This slowed down innovation.

One of the first experiments that Octopart ran was to change the conversion button CTAs. The copy had been the same for 10 years, and there was a lot of justified concern around the risks of making a shift. Within two months of starting with Split, they conducted a series of A/B tests with different copy and found new wording that significantly increased conversions and revenue.

Using experiments to ‘do no harm’

With Split, the Octopart team now regularly runs ‘do no harm experiments’ for new feature launches. They launch a feature that’s expected to have a long-term, strategically important impact, and run an experiment to (hopefully) show that conversions and KPIs are not negatively impacted.

Soon after implementing Split, Octopart conducted an experiment that changed their UX for search results. The feature had been developed based on user observation and competitive research. “We felt really confident that this feature was good for users,” Sam explained. However, much to their surprise, an experiment using Split to allocate a portion of traffic to the feature and measure impact showed that the feature caused a dramatic and significant reduction in conversions.

We worked with the Split support folks to gain confidence in our results – that we’d set up and analyzed the experiment correctly. When we were dealing with surprising results, it was so useful to lean on the experimentation experts at Split.

Sam Bobb
Sr. Director of Data Science

With confidence in the experiment confirmed, the Octopart team reviewed session recordings showing users interacting with the feature and learned that it made an important user-flow inconsistent, which caused frustration and reduced conversions.

That experiment was an excellent learning experience for the Octopart team. Sam said, “we became very humble about our ability to predict how users will interact with any change – the only way to know is to do a high-quality, controlled experiment. Now we test almost every change – including the ones we’re confident will be positive or should have no impact on users.”

It’s powerful to think about all features launched before Split and how many of those must have cost us money that we don’t know about.

Sam Bobb
Sr. Director of Data Science

Analyzing user behavior with Split

Through Split’s integration with Segment, all raw experiment data is exported into Octopart’s data warehouse. This enables the Octopart data science team to do a deep dive into any experiment – which is particularly useful when something surprising happens. One common use-case is to retroactively segment users within an experiment to understand variation in behavior. In one case, things looked odd in an experimental result and they segmented users by country and found a strong regional effect: the feature in question impacted European users differently from non-European users.

Over the past year the team has scaled up to running several experiments concurrently and will begin to implement feature flags for operational efficiency as the team continues to grow.

Split and Octopart

Here you’ve seen how Octopart uses experimentation to not only increase site conversion but also increase their revenue. They use Split’s powerful metrics and monitoring tools to measure features’ success before rolling out to their entire user base. As Sam says, many in-house experimentation tools are not set up to properly handle the vast amount of data and analysis that needs to be done. Split helps Octopart manage their data, without having to worry about the intricacies of its implementation.

If you’d like to learn more about Split, we’d love to hear from you! Reach out to our sales team, or sign up to try out feature flags for your organization.