Concept
Evals
A small, concrete set of test cases used to measure whether an AI system meets a defined bar of quality, useful for clarifying what success looks like and tracking progress toward it.
Evals
9 visible of 11 connections
Related Concepts7
Related Frameworks2
Related Practices37
Ask about this concept
Transcript mentions
… I think the third thing that is useful, but not everyone loves doing, is building evals. You don't need to build hundreds of evals for them to be useful.
We have done a lot of work to figure out how we do evals, and we're also continuing to evolve our process. Yeah, it's something that you have to be really focused on, and I think that it's easy to go on vibes for too long.
I think first of all it's like evals are hard. We talked about that earlier.
… Evals is a way to systematically measure and improve an AI application, and it really doesn't have to be scary or unapproachable at all. It really is, at i...
… Well, it ties to what I said earlier about how if the model is the product, evals are the PRD, but also subsequently the sales collateral, right? Because evals are what you give to researchers to show them what they should be build...
Concept
Evals
A small, concrete set of test cases used to measure whether an AI system meets a defined bar of quality, useful for clarifying what success looks like and tracking progress toward it.
Related Concepts7
Related Frameworks2
Related Practices37
Ask about this concept
Transcript mentions
… I think the third thing that is useful, but not everyone loves doing, is building evals. You don't need to build hundreds of evals for them to be useful.
We have done a lot of work to figure out how we do evals, and we're also continuing to evolve our process. Yeah, it's something that you have to be really focused on, and I think that it's easy to go on vibes for too long.
I think first of all it's like evals are hard. We talked about that earlier.
… Evals is a way to systematically measure and improve an AI application, and it really doesn't have to be scary or unapproachable at all. It really is, at i...
… Well, it ties to what I said earlier about how if the model is the product, evals are the PRD, but also subsequently the sales collateral, right? Because evals are what you give to researchers to show them what they should be build...