Concept
Scaling Laws
The empirical relationship showing that increasing training compute and data predictably and smoothly reduces a model's loss and increases its overall capability.
Scaling Laws
4 visible of 5 connections
Related Concepts2
Related Frameworks2
Related Practices9
Ask about this concept
Transcript mentions
… some really interesting graphs in the original scaling law papers and I think folks are very familiar with the scaling loss in in the lens of um as you add in more compute and data what's called loss aka the loss from next token…
… Those are the three ingredients in the scaling laws. And just to make that concrete, before we had transformers, we had LSTMs and we've done scaling laws on what the exponent is on those two things.
Concept
Scaling Laws
The empirical relationship showing that increasing training compute and data predictably and smoothly reduces a model's loss and increases its overall capability.
Related Concepts2
Related Frameworks2
Related Practices9
Ask about this concept
Transcript mentions
… some really interesting graphs in the original scaling law papers and I think folks are very familiar with the scaling loss in in the lens of um as you add in more compute and data what's called loss aka the loss from next token…
… Those are the three ingredients in the scaling laws. And just to make that concrete, before we had transformers, we had LSTMs and we've done scaling laws on what the exponent is on those two things.