Episode

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

This podcast episode features Hamel Husain and Shreya Shankar discussing the importance and methodology of evaluations (evals) for AI products. They provide a comprehensive walkthrough of their systematic approach to building evals, starting with error analysis of real application traces, then categorizing problems using open and axial coding techniques, and finally implementing automated LLM-as-judge evaluators. The conversation addresses common misconceptions about evals, clarifies the ongoing debate in the AI community about their necessity, and emphasizes that evals are essentially data science applied to AI applications.