Experimentation in Product Management: “The Product Experience”

13 Nov

Produced By Think & Do

Estimated Reading Time: 8 minutes - time saved 41 minutes 🔥

Dive into the "Experimentation" episode, where hosts chat with Faith Forster of Dext and Iqbal Ali, an experimentation guru. They unpack the critical role of experimentation in product development, from defining it to embedding it into your company culture.

🔍 Theme 1: Defining Experimentation

[00:06:16]

Faith Forster kicks things off by describing experimentation as a decision-making tool in data-scarce situations. Iqbal Ali adds that it's all about a systematic, data-driven approach to mitigate risks and optimize experiences. They agree: it's more than just A/B testing; it's a continuous process. 🔄🔬 

🔬 Theme 2: Common Mistakes in Experimentation

[00:08:54]

The hosts explore frequent blunders, like treating experimentation as an afterthought or lacking clarity in learning objectives. The key? Focus on insights, not just data points. 🎯

🧠 Theme 3: Setting Up a Culture of Experimentation

[00:11:15 - Setting up a culture of experimentation]

The hosts and guests delve into how to establish a culture of experimentation within a business. Faith Forster shares her experience at Dext, where they used experimentation to demonstrate progress and secure funding. They set up a system to prioritize experiments, focusing on the most impactful ideas and running quick tests to gather data. Iqbal Ali emphasizes the need to train teams in the experimentation process and align their roles with the overall goal. They also discuss the importance of reframing conversations and metrics to foster a culture of curiosity and learning. 🚀

🚧 Theme 4: Overcoming Resistance to Experimentation

[00:16:37]

The hosts and guests address the challenge of getting people to embrace experimentation, especially when experiments yield negative results or challenge existing beliefs. They suggest reframing the conversation from failure to learning and focusing on insights gained rather than conversion rates. Faith Forster shares her experience of using experimentation to de-risk a major pivot at Dext, emphasizing the need to communicate the purpose and metrics effectively. They also discuss the importance of involving data experts or statisticians to guide decision-making.💡

🧭 Theme 5: Experimentation Throughout the Product Lifecycle

[00:19:41]

Experimentation evolves throughout the product lifecycle. Faith Forster introduces a two-by-two matrix that maps the clarity of opportunity against the risk of getting it wrong. This matrix helps teams determine the right approach for each initiative, whether it's design-led, tech-led, or requires experimentation. Iqbal Ali adds that experimentation can be used to de-risk major changes during the discovery phase or to validate assumptions during the mature stage. They emphasize the importance of breaking down problems and using experiments to mine for insights.🌱

⏰ Theme 6: Duration of Experiments

[00:35:24]

How long should an experiment run? It depends on many factors, but balancing time investment with desired results is key. ⌛Iqbal Ali explains that the duration depends on variables such as traffic volume, conversion rates, seasonality, and the desired level of certainty. He mentions the concept of minimum detectable effect and the need to balance the time invested with the results obtained. Faith Forster suggests that the decision to continue or end an experiment should be based on the results obtained and the potential for further learnings.

⛗ Theme 7: Experimentation with Limited Traffic

[00:43:14]

Running experiments with low traffic? Iqbal Ali advises focusing on secondary metrics and using A/B testing to gather insights. He suggests looking beyond conversion rates and analysing user behaviour to understand the impact of changes. Faith Forster adds that user testing can also be valuable in understanding usability and perception issues. They emphasize the importance of de-risking solutions and adapting the experimentation approach based on the available traffic.📊 

Listen to the full episode to dive deeper into practical tips and guidance on setting up a culture of experimentation, overcoming resistance, and leveraging experimentation throughout the product lifecycle.

Podcast Link: [Apple Podcasts]

Timestamp Details:

  • [00:00:44] Experimentation at Mind the Product conference.

  • [00:03:34] Product experimentation and insights.

  • [00:08:54] Common experimentation mistakes.

  • [00:11:15] Experimentation for business success.

  • [00:15:09] Going to markets through small businesses.

  • [00:19:41] Fixing the language around experiments.

  • [00:24:04] Assessment and prioritization process.

  • [00:27:13] When to stop experimenting.

  • [00:33:14] Other forms of experimentation.

  • [00:35:11] How long should experiments run?

  • [00:41:05] Minimum customers for experimentation.

  • [00:44:24] A-B testing and metrics.

  • [00:46:59] Navigating change as a product leader.