A Data-Driven and Agile Approach to the Budding Cannabis Industry
Leafly is the largest global cannabis website, enabling more than 15 million visitors a month to research, find, and buy cannabis from legal, licensed retailers. As the most trusted brand in cannabis, Leafly leads the charge in bringing cannabis into the mainstream with cutting edge journalism and innovative technology to provide education and accessibility for all.
The company’s mission of helping everyone experience the benefits of cannabis across their website and apps means every product initiative must start and end with the customer. In the past, customer interviews served as the primary source of product insights. However, to remain a leader in a fast-growing industry, Leafly’s product and engineering teams knew they needed to be more agile in developing new features and more data-driven in understanding its impact on the user experience and business goals.
Quantifying Feature Impact on Millions in their Cannabis Journey
With deep roots in big tech including Amazon and Apple, Ian Metz, Senior Product Manager at Leafly, was very experienced with A/B testing and canary releases and knew they needed a robust platform for feature flagging and experimentation. He championed Split within his organization to manage the entire lifecycle for new features using Agile best practices and steer the product roadmap with data-driven insights. Using Split to test and iterate on new features rigorously, the team launched a complete rebrand of their homepage alongside new pages detailing different cannabis strains.
The goal of the frontend redesign was to increase orders for delivery or pickup at customers’ local dispensaries. Ian spearheaded this effort by experimenting with different carousel placements on the homepage and strain detail pages. Using Split, they ran A/B tests with various positioning of deals, product recommendations, and dispensaries. From there, Split passed feature flag data to Google Analytics to drill deeper into engagement data by feature variation. Prior to Split, it had been impossible to distinguish between variation and control in their Google Analytics events. Split allowed Ian and his team to attribute features to events in understanding every feature’s causality and impact on key metrics.
“Split has changed the way we release product features and how Leafly operates as a business by enabling us to collectively understand the work we’re doing and the impact it has on our customers.”
Ian Metz, Sr. Product Manager
On strain detail pages, when dispensaries that carried the strain were shown in the first few viewports at the top of the page, the results in Split showed CTR increased 107%. However, to Ian’s surprise, conversions stayed flat. The metrics showed dropoff occurred on the product detail page showing items carried at a local dispensary. Customers weren’t adding items to their cart and the team was unsure why.
Relating to the Customer Experience with Qualitative Data
By integrating Split with FullStory, Leafly validated their experiment results with qualitative data and determined why customers were abandoning their shopping experience after exhibiting high intent to purchase. With the community integration set up, Split sends feature data as custom events to FullStory to mark each user session recording with the feature flag treatment the user received. FullStory sessions are then segmented by feature flag treatment for more in-depth analysis on the conversion funnel and to uncover frustration signals such as rage clicks, dead clicks, and thrashed cursor.
The team also watched individual sessions and witnessed firsthand why customers weren’t converting. The purchase paths for online and in-store-only had identical user experiences. Customers trying to buy in-store-only items struggled to look for an “Add to cart” button not knowing the items were unavailable online. The recordings revealed that customers were eager to purchase, but simply didn’t know how. Seeing the customer’s cursor thrash back and forth across the page was evidence enough for the team to prioritize building a better purchase process for in-store-only. This time, they were able to combine the power of experimentation and digital intelligence to ensure the new experience was a success.
“Using Split with FullStory allowed us to humanize our experiments and empathize with our customers by validating the quantitative results with real user sessions that revealed how customers felt when interacting with our features.”
Ian Metz, Sr. Product Manager
Cultivating a Culture of Experimentation and Customer Empathy
Today, Leafly combines feature releases and experiments with Split with biweekly viewings of FullStory sessions. These viewings have become a collaborative venue for product managers and engineers to study key metrics alongside visual evidence of the user journey, identify new events to track, brainstorm new iterations to test, and guide feature rollout plans. Leveraging Split and FullStory together has allowed Leafly’s product and engineering teams to align on their features and experiments’ statistically significant impact while also cultivating empathy for their customers.
Interested in seeing how Split and FullStory can help your team build impactful products that resonate with your customers? Schedule a demo with us to learn more.
Leafly is the largest global cannabis website, enabling more than 15 million visitors a month to research, find, and buy cannabis from legal, licensed retailers.
- Use Cases: Feature Experimentation