Overcoming Experimentation Obstacles In B2B

Implementing a B2B experimentation program can be more complex than its B2C counterpart. Two of the most prominent considerations are smaller sample sizes and unbalanced user allocation. Here’s a guide for how to overcome those hurdles to roll out the best experiments you possible.

Determine the Optimal Traffic Type

Consider the advantages and disadvantages of employing a tenant (e.g., account-based) traffic type versus a conventional user traffic type for each experiment. Unless it is crucial to provide a consistent experience for all users within a specific account, opt for a user traffic type to facilitate experimentation and measurement. By following these steps, you can significantly increase your sample size, unlocking greater potential for insights and analysis.

Important to note: In Split, the traffic type for an experiment can be decided on a case-by-case basis, depending on the feature change, the test’s success metrics, and the sample size needed.

Make a Plan for Lower Sample Sizes

Utilize the 10 Tips for Running Experiments With Low Traffic guide. You can thank us later!

Normalize Data for Tenant Traffic Types

Split’s application doesn’t consider that some tenants may have more users than others, resulting in unbalanced user allocation across treatments. Utilize the following tips for normalizing data:

Experimenting with tenant traffic

This often results in unbalanced user allocation

Split ensures that a 50/50 experiment divides tenants according to that ratio, but doesn’t consider that some tenants may have more users than others.

This can result in an unbalanced user allocation, as shown here:

Utilize a “percent of” metric type

This can be helpful when you want an accurate measurement, but don’t need balanced users. 

Use this metric type when the event you want to measure only needs to occur once per tenant to be considered a success (e.g., percent of accounts that upgraded plans). Instead of leveling your users across treatments, it equalizes the data.

Utilize a “ratio of two events” metric type:

Helpful when you want accurate measurement, but don’t need balanced users

Use this metric type when measuring total engagement volume with a feature (e.g., number of clicks to “add to favorites”). This lets you control the numerator and denominator to equalize the metric data. 

A reminder: The numerator is set to the event you want to count. The denominator is set to an event that occurs leading up to the numerator event. This can also be a generic event that tracks the number of users who saw the treatment.

If you follow these steps, you should be able to overcome any obstacle when running a B2B experiment. And remember: Split offers the unique flexibility to run experiments based on the traffic type that suits your needs. Learn more here.

Want to Dive Deeper?

A/B testing can tip the scale by reducing risk while exploring opportunities during peak traffic.

What happens when you have low traffic but want to run experiments? Follow these tips.

Experiment with software features that trigger the right chemicals in the brain. Follow this guide and increase customer conversion.

Get Split Certified

Split Arcade includes product explainer videos, clickable product tutorials, manipulatable code examples, and interactive challenges.

Switch It On With Split

Split gives product development teams the confidence to release features that matter faster. It’s the only feature management and experimentation platform that automatically attributes data-driven insight to every feature that’s released— all while enabling astoundingly easy deployment, profound risk reduction, and better visibility across teams. Split offers more than a platform: It offers partnership. By sticking with customers every step of the way, Split illuminates the path toward continuous improvement and timely innovation. Switch on a free account today, Schedule a demo to learn more, or contact us for further questions and support.