We Need to Raise the Bar for AI Product Managers

🌈 Abstract

The article discusses the importance of product managers (PMs) taking a more hands-on approach to the development of AI-powered products, rather than treating AI models as "black boxes" and deferring responsibility to model developers. It highlights the consequences of the hands-off approach and the benefits of a more rigorous, hands-on approach.

🙋 Q&A

[01] The Problem: PMs are Incentivized to Keep Their Distance

1. What are the key issues with the hands-off approach that PMs often take towards AI model development?

• PMs tend to treat AI models like "black boxes" and deflect responsibility for poor outcomes onto model developers
• This behavior is similar to blaming a designer for bad signup numbers after a site redesign, rather than taking ownership of the results
• The hands-off approach is the norm, even though tech companies expect PMs working on consumer products to make informed decisions about design changes and take ownership of the results

2. What are the consequences of the hands-off approach?

• When a model launch falls flat, it is often due to a hands-off approach being employed
• This is less common at large companies with a history of deploying AI, but is still an issue for many organizations
• Overcoming the inertia of the hands-off approach can be challenging, especially when company leadership doesn't expect anything more and a PM might face pushback for "slowing down" the development cycle

[02] The Hands-On Approach

1. What are the key elements of the hands-on approach that PMs should take towards AI model development?

• Clearly define the goal and metrics for the AI-powered product, beyond just growth or A/B testing
• Actively participate in the model evaluation process, including reviewing sample recommendations for different user segments and income levels
• Provide input on the specific model architecture and design, not just leaving it to the model developer
• Engage in a more rigorous process of product reviews and quality control to ensure the model is delivering the desired outcomes

2. What are the challenges and risks of the hands-on approach?

• The hands-on approach requires significantly more work upfront for the PM
• PMs may face pushback from leadership for "slowing down" the development cycle by adopting these practices
• The hands-off approach may still "kind-of work" in some cases, providing a temptation to take the path of least resistance

[03] The Role of Product Leadership

1. What is the critical role that product leadership plays in elevating the standards for AI products?

• Product leadership must hold PMs accountable for understanding what they are shipping and the impact of AI models on user experience and long-term product outcomes
• They should ask probing questions about the evaluation process, model architecture, and quality control measures
• They should be thoughtful about where different types of evaluations are used and ensure that resources are available for follow-up work

2. What is the key shift that needs to happen in the way PMs approach AI product development?

• PMs must move beyond the hands-off approach that has too often led to suboptimal outcomes
• They need to take a deeper, more hands-on approach to understanding and shaping the development of AI models
• This requires upskilling for many teams, but the necessary resources are readily available