Lenny's Podcast: Product | Career | Growth · Lenny Rachitsky

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

·1 hr 47 min·5 clips

To build great AI products, you need to be really good at building evals—it's the highest ROI activity you can engage in.

Lenny Rachitsky opens by noting that evals, short for evaluations, have emerged over the past year as the most discussed skill in AI product development, cited by the chief product officers of both Anthropic and OpenAI as the most important new capability for builders. His guests are Hamel Hussain and Shreya Shankar, who teach the definitive online course on evals at Maven, which has become the platform's most enrolled course, and who have trained engineers across 500 companies including large portions of the OpenAI and Anthropic teams. The conversation is organized around show-versus-tell: rather than just describing what evals are, Hussain and Shankar walk through the process of developing them for a real AI application. Hussain opens with the core claim that evals are the highest ROI activity an AI product team can engage in, and that most teams dramatically underinvest in them relative to the value they provide. The episode covers the most common misconceptions about evals, starting with the belief that since we live in the age of AI, AI can just eval itself, which Hussain and Shankar argue does not work as a primary evaluation strategy. The concept of the benevolent dictator is introduced: for open coding tasks where teams need to develop evaluation criteria, committees tend to create bloated processes that make evals too expensive to run frequently, whereas appointing a single trusted person with domain expertise produces faster and more consistent results. Shankar addresses why many practitioners are anti-evals, explaining that most opposition comes from having done them badly in the past and then incorrectly concluding that evals do not work rather than that the process was flawed. The episode covers how to build an eval pipeline from scratch, when to use human judges versus automated evaluation, and how to track improvement over time. The tone is practical and pedagogical, aimed at people who want to do this work rather than just understand it conceptually. The audience is product managers, engineers, and AI builders who want systematic ways to improve their products.

As heard by us

A practical case for treating evals as core product work.

This discussion makes a solid case that AI evals belong inside product work, not off to the side as cleanup. It stays grounded in operator choices: open coding, bringing in a trusted person with domain expertise, and shaping the findings into something a team can actually use.

Read the full review in PlayNext →

Why you'd press play

If your AI product feels fuzzy, this is the evals playbook you need.

Read the full recommendation in PlayNext →

Listen to the show on

Apple Podcasts Spotify

Play clips from this episode More from Lenny's Podcast: Product | Career | Growth