Opening Intuitions on Language Models and the History of Transformer Architectures
Intuitions on Language Models and the History of Transformer Architectures
- Length
- 1 h 17 min
- Level
- Undergraduate
- Recorded
- Stanford University2024
Chapter 1 of 11: Before the lecture
Frames next-word prediction as massive multitask learning, then reduces an encoder-decoder transformer to decoder-only.
Language model behaviour under scaling is the subject of this two-part Stanford CS25 session by Jason Wei and Hyung Won Chung of OpenAI. Wei treats next-word prediction as implicit multitask learning across grammar, world knowledge and arithmetic. He reconciles smooth compute scaling laws with sudden emergent abilities on BIG-bench tasks, and decomposes prompts to explain inverse and U-shaped scaling curves. Chung argues that cheaper compute drives the field, citing Rich Sutton's bitter lesson. He converts an encoder-decoder transformer into a decoder-only model step by step, showing how separate parameters and bidirectional attention lose their value at scale.
Language model behaviour under scaling is the subject of this two-part Stanford CS25 session by Jason Wei and Hyung Won Chung of OpenAI. Wei treats next-word prediction as implicit multitask learning across grammar, world knowledge and arithmetic. He reconciles smooth compute scaling laws with sudden emergent abilities on BIG-bench tasks, and decomposes prompts to explain inverse and U-shaped scaling curves. Chung argues that cheaper compute drives the field, citing Rich Sutton's bitter lesson. He converts an encoder-decoder transformer into a decoder-only model step by step, showing how separate parameters and bidirectional attention lose their value at scale.
Scaling Laws for Neural Language Models
Are emergent abilities of large language models a mirage?
Scaling Instruction-Finetuned Language Models
This lecture
Talk 13 of 21 of Transformers United.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in