Practice
Reading full failure transcripts before writing an eval
Digging into the exact trajectory and detail of a user complaint (what was asked, what the model did, why it failed) rather than acting on a vague complaint, to identify a reproducible, actionable failure pattern.
Reading full failure transcripts before writing an eval
1 visible of 1 connections
Context
Related Concepts10
Transcript mentions
… activity we have on the team is reading the transcripts and understanding uh what was the trajectories that failed very deeply to then say was this like a hallucination? was this claw being overconfident.
Practice
Reading full failure transcripts before writing an eval
Digging into the exact trajectory and detail of a user complaint (what was asked, what the model did, why it failed) rather than acting on a vague complaint, to identify a reproducible, actionable failure pattern.
Context
Related Concepts10
Transcript mentions
… activity we have on the team is reading the transcripts and understanding uh what was the trajectories that failed very deeply to then say was this like a hallucination? was this claw being overconfident.