/
lectern.tv
Play15s30s1x0:00/--:--
DoneWatched
Expand
lectern.tv
1:36/1:56:20
7:52
1:36
4:59
7:44
12:42
3:07
4:13
4:58
4:43
2:48
2:18
1:26
1:52
56:02
Let's Build GPT: From Scratch, in Code, Spelled Out
Perhaps the single most-watched modern AI engineering lecture: constructs a full decoder-only Transformer live from an empty file.
Constructs a full decoder-only Transformer from an empty file in PyTorch, walking through tokenization, scaled dot-product causal self-attention, multi-head blocks, residuals, and LayerNorm. Trained live on the Tiny Shakespeare character-level dataset to demonstrate autoregressive text generation.
Lecture Guide
- 1Language models are trained to predict the next token in a sequence, modeled here using a character-level Tiny Shakespeare dataset.
- 2Self-attention acts as a communication mechanism where query and key dot products determine data-dependent token affinities.
- 3A lower triangular mask enforces causality in decoder-only transformers so future tokens cannot influence past predictions.
- 4Residual connections and pre-norm LayerNorm layers form a gradient superhighway that enables the optimization of deep neural networks.
Proofs, Examples & Demos