Let's Build GPT: From Scratch, in Code, Spelled Out
Founder of Eureka Labs, Former Director of AI at Tesla
Perhaps the single most-watched modern AI engineering lecture: constructs a full decoder-only Transformer (scaled dot-product causal self-attention, multi-head blocks, residuals, LayerNorm) live from an empty file.
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.
Links:
- Google colab for the video: https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing
Building makemore Part 5: Hierarchical Dilated Causal Convolutions (WaveNet)
Next · Lecture 8State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
Seminar notes & discussion
No public seminar notes on this lecture yet.