🤖 A Conversation with OpenAI's CPO Kevin Weil, Anthropic's CPO Mike Krieger, and Sarah Guo

Given the topic of the year (and the trends for next) is firmly AI, Today’s Pod Shot covers the recent Lenny and Friends interview with two leaders shaping the future of AI products—Kevin Weil from OpenAI and Mike Krieger from Anthropic. They were joined on stage by investor Sarah Guo, and the trio shared lessons on leadership, product management, and AI's extraordinary potential. Here's the story of what they revealed, packed with practical insights for founders and product managers.

🎥Watch the full episode here

📆 Published: November 12th, 2024

🕒 Estimated Reading Time: 3 mins. Time saved: 37 mins🔥

🚀 The Leap into AI Leadership

When Kevin and Mike took on their roles, reactions ranged from excitement to surprise. Kevin described his role as "challenging, interesting, and sleepless," grappling with AI's rapid development. Unlike many other product roles, AI product management involves navigating ever-evolving capabilities, creating an ongoing cycle of discovery and adaptation.

Mike's decision to join Anthropic sparked curiosity, given his high-profile background at Instagram. His reasoning? A vow to ensure each year of his life is different—and AI offered that dynamic change. His biggest adjustment? Adapting to enterprise timelines and the delightfully direct feedback loops that come with customer interactions.

Key Takeaways for Founders & PMs:

• Be ready to embrace new challenges; dynamic fields like AI push boundaries.

• Enterprise work requires patience—longer feedback loops can yield deeper insights.

• Constantly seek fresh perspectives and roles that challenge your comfort zone.

🔄 Redefining Product Instincts in AI

Both Kevin and Mike discussed how product instincts evolved in their roles. Kevin highlighted the dual nature of AI product management: on one hand, it feels like traditional product work when fine-tuning a near-ready feature; on the other, it's a stochastic, emergent process where possibilities evolve unpredictably. Mike likened AI's impact to Apple's WWDC announcements—game-changing but disruptive to roadmaps.

Key Takeaways for Founders & PMs:

• Traditional instincts may guide some decisions, but flexibility is critical.

• Expect disruption; an AI capability can emerge unexpectedly and alter plans.

• Collaborative alignment between product and research teams is essential.

📈 60% Success: Designing for Imperfection

The conversation tackled a pivotal question: how to design products when AI capabilities might only be 60% accurate. Kevin cited GitHub Copilot as an example—it launched with imperfect models but delivered immense value by saving developers time. Mike added that success often depends on lumpy outcomes—where one client finds immense value, another may experience challenges.

Key Takeaways for Founders & PMs:

• Embrace imperfection; a 60% accurate AI can still deliver value with human-in-the-loop design.

• Evaluate feedback iteratively to adapt and improve models in real-world settings.

• Pilot programs with diverse customers can uncover nuanced use cases and limitations.

🛠️ The Art of Evals: A Core PM Skill

Evaluations—or "evals"—emerged as a cornerstone skill for AI product managers. Both leaders stressed that building successful AI features depends on creating effective evaluation frameworks. These frameworks determine how well AI models perform and guide iterative improvement.

Kevin explained how OpenAI conducts PM bootcamps to teach eval-writing, while Mike shared Anthropic's strategy of embedding PMs early in the design and fine-tuning process.

Key Takeaways for Founders & PMs:

• Learn to write and refine evaluation metrics—this skill will shape AI product quality.

• Integrate PMs deeply into research and design workflows for maximum impact.

• Use models themselves to iterate on eval creation and grading.

🧠 Prototyping and Intuition with AI

One standout insight: prototyping with AI isn't just useful—it’s transformative. Mike praised Anthropic’s best PMs for rapidly using AI to compare UI concepts, enabling faster iterations. Kevin added that gaining technical depth in AI accelerates innovation and enhances product decision-making.

Key Takeaways for Founders & PMs:

• Leverage AI to prototype solutions quickly and explore possibilities.

• Build technical intuition about AI capabilities to strengthen product design.

• Experimentation isn’t about perfection—it’s about learning fast.

🗣️ AI as a Universal Translator—and Beyond

Kevin shared an inspiring story about using ChatGPT’s advanced voice mode as a translator during his travels in Korea and Japan. The experience demonstrated AI’s potential to break down language barriers and expand opportunities globally.

Mike highlighted the future of AI-powered assistants that proactively generate solutions—like creating project drafts or adapting product roadmaps to market conditions. These capabilities could revolutionise workflows by automating repetitive tasks and allowing humans to focus on creativity.

Key Takeaways for Founders & PMs:

• Explore AI as a tool for accessibility and inclusion in global markets.

• Look for opportunities to automate repetitive tasks, freeing up time for strategic work.

• Encourage proactive AI usage in generating ideas and adapting processes.

🤖 Personality, Empathy, and the Next Evolution of AI

AI models are more than tools—they’re evolving into entities with distinct "personalities." Kevin discussed user attachment to model behaviour, such as users preferring specific interactions. Mike described a growing empathy toward models, likening their evolution to relationships with people.

Key Takeaways for Founders & PMs:

• Consider how model "personalities" impact user experience and brand loyalty.

• Align model behaviour with user expectations to build trust and engagement.

• Design AI systems with empathy, focusing on how users interpret their interactions.

✨ The Magic and Speed of AI Innovation

Closing the discussion, the trio reflected on the rapid pace of AI advancements. Kevin observed how features that felt magical one year ago now seem primitive. Mike predicted a future where AI becomes proactive, asynchronous, and deeply integrated into our daily workflows.

Key Takeaways for Founders & PMs:

• Stay agile and adaptive; the pace of AI innovation is accelerating.

• Anticipate user needs as they evolve alongside advancing technologies.

• Look for opportunities to pioneer new behaviours and experiences in AI.

Wrapping Up

This conversation painted a pretty comprehensive picture of AI’s present and future and how how PMs operating in the space need to adapt. From embracing imperfection to mastering evals and prototyping, the advice shared give plenty of food for thought for help PMs and founders trying to harness AI’s potential while staying in line with its breakneck speed of innovation.

 🎥Watch the full episode here

📅Timestamps:

• 00:00:00 - Introduction and Roles

• 00:01:19 - Excitement and Challenges

• 00:03:03 - Learning About Enterprise

• 00:04:56 - Instincts and Product Development

• 00:08:03 - Planning in Uncertainty

• 00:10:14 - Evaluating Model Performance

• 00:12:09 - Human-Machine Collaboration

• 00:14:42 - The Role of Evaluations

• 00:16:05 - Prototyping with AI

• 00:19:03 - Understanding the Tech Stack

• 00:21:05 - Adapting to Non-Deterministic Systems

• 00:22:57 - Educating End Users

• 00:26:00 - Change Management in Enterprises

• 00:27:28 - Internal Use Cases of AI

• 00:30:07 - Reasoning in AI Models

• 00:33:14 - Future Experiences with AI

• 00:36:57 - Surprising User Interactions