ConceptAI & ML
Pre-training
The initial stage of training where a model learns a general understanding of the world from large amounts of internet text or private data.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Neural networks3Deep neural networks1Large language models1Machine learning1Text corpora1Text datasets1Tokenization1Transformer1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
4Each in their own terms, from the easiest level.
Geoffrey Hinton
A technique to initialize the weights of a deep neural network before fine-tuning it on a specific task.Andrej Karpathy
The first stage of training where a network is trained on a massive quantity of internet text to generate internet documents.Percy Liang
The initial phase of training a model on a large dataset before fine-tuning it for specific tasks.
Lectures that assume it
13The first 12 are shown.
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Transfer Learning: Models
- Demystifying Mixtral of Experts
- Behind the Scenes of LLM Pre-training: StarCoder Use Case
- Reinforcement Learning from Human Feedback: Algorithms & Applications
- Differentially Private Table-Image Multimodal Data Generation
- Beyond Brute Force: The Era of Adaptive Intelligence
- The Future of Artificial Intelligence
- Transformers in Vision: Tackling problems in Computer Vision
- A 'Deep Blue-Kasparov Moment for Mathematics': Euler Equations and AI
- Overview of Transformers (Stanford CS25: Transformers United V6)
Connections
28The lectures that define it or assume it, and the concepts next to it.
Concepts9
- Neural networks
- Deep neural networks
- Large language models
- Machine learning
- Supervised fine-tuning
- Post-training
Show 3 moreShow fewer
Lectures19
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- Introduction to Large Language Models
- Deep Dive into Large Language Models: Pretraining to RLHF
- Why We Need Machine Learning: An Introduction to Neural Networks
- Marin: An Open Lab For Frontier AI
- How Always-On Agents Work
Show 13 moreShow fewer
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Transfer Learning: Models
- Demystifying Mixtral of Experts
- Behind the Scenes of LLM Pre-training: StarCoder Use Case
- Reinforcement Learning from Human Feedback: Algorithms & Applications
- Differentially Private Table-Image Multimodal Data Generation
- Beyond Brute Force: The Era of Adaptive Intelligence
- The Future of Artificial Intelligence
- Transformers in Vision: Tackling problems in Computer Vision
- A 'Deep Blue-Kasparov Moment for Mathematics': Euler Equations and AI
- Overview of Transformers (Stanford CS25: Transformers United V6)
- Beyond LLMs: Agents, Emergent Abilities, Intermediate-Guided Reasoning, BabyLM