Andrej Karpathy
Founder · Eureka Labs · Distinguished
Andrej Karpathy is the founder of Eureka Labs, a founding member of OpenAI and former Senior Director of AI at Tesla, who created Stanford's CS231n course.
- lectures
- 13
- series
- 1
- hours
- 24
Start here
- Spelled-Out Intro to Neural Networks & Backpropagation: Building microgradfrom Raschka, Build a Large Language Model (From Scratch)

- Let's Build GPT: From Scratch, in Code, Spelled Outfrom Raschka, Build a Large Language Model (From Scratch)

- Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNormfrom Raschka, Build a Large Language Model (From Scratch)

Series and courses
Neural Networks: Zero to Hero
Open the course
- 1Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
- 2Spelled-Out Intro to Language Modeling: Building makemore (Bigram & Cross-Entropy)
- 3Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
- 4Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
Show 7 moreShow fewer
- 5Building makemore Part 4: Becoming a Backprop Ninja (Manual Tensor Gradients)
- 6Building makemore Part 5: Hierarchical Dilated Causal Convolutions (WaveNet)
- 7Let's Build GPT: From Scratch, in Code, Spelled Out
- 8State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- 9Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- 10Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- 11Introduction to Large Language Models
In these lectures
- Automatic differentiation evaluates gradients by recursively applying the calculus chain rule backward through a dynamic computation graph.Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
- Language models are trained to predict the next token in a sequence, modeled here using a character-level Tiny Shakespeare dataset.Let's Build GPT: From Scratch, in Code, Spelled Out
- Overconfident initial logits cause artificially high loss values at step zero.Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
Draws on
People cited in the lectures, at the moment of the citation.
- ChatGPT / GPT-4Cited as the best language model currently available
- Greg BrockmanFounder of OpenAI, cited for his vision demo.
- Sam AltmanAnnounced the GPTs app store at OpenAI DevDay in Nov 2023.
About
Andrej Karpathy is the founder of Eureka Labs, a founding member of OpenAI and former Senior Director of AI at Tesla, who created Stanford's CS231n course. His lectures here build neural networks in code, from the micrograd autograd engine and makemore models to a GPT Transformer and a GPT-2 (124M) reproduction.
Same subject, AI & Machine Learning
Connections
23Their places, courses, books and lectures.
Scholar1
Institutions5
Lectures13
- Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
- Building makemore Part 4: Becoming a Backprop Ninja (Manual Tensor Gradients)
- Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
Show 7 moreShow fewer
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Building makemore Part 5: Hierarchical Dilated Causal Convolutions (WaveNet)
- Introduction to Large Language Models
- Spelled-Out Intro to Language Modeling: Building makemore (Bigram & Cross-Entropy)
- Deep Dive into Large Language Models: Pretraining to RLHF
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- Scalability in Autonomous Driving (CVPR 2020 Keynote)