ConceptAI & ML
Scaling laws
Empirical power-law relationships predicting model performance as a function of compute, dataset size, and parameter count.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Language model4Model parameters3Compute2Compute budget2Loss function2Model capacity2Model pretraining2Algorithm scaling1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
9Each in their own terms, from the easiest level.
Andrej Karpathy
The mathematical relationships showing that as model size and compute increase, downstream task performance improves predictably.Loubna Ben Allal
Scaling laws study the allocation of a compute budget between data size and model size.Jeremy Bernstein
Empirical relationships describing how model performance improves predictably with compute, dataset size, and number of parameters.Phillip Isola
Predictive relationships between the amount of compute or model size and the resulting model performance.Lucas Beyer
Empirical laws relating model performance to compute, dataset size, and parameter count.Lorenzo Zanisi
Empirical relations predicting how quickly plasma leaks out of magnetic confinement shapes based on experimental data.Percy Liang
A technique that relates model performance to compute and data scale to ensure small-scale findings translate to large-scale models.Irina Rish
Mathematical relationships describing how model performance depends on parameters like model size, dataset size, and compute.
Lectures that assume it
9- Speculations on Test-Time Scaling for Large Language Models (o1)
- Training Sand to Think: Artificial General Intelligence & Future of Physics
- Related Rates
- Intuitions on Language Models and the History of Transformer Architectures
- Demystifying Mixtral of Experts
- Aerothermodynamics
- Interactions with the Physical Environment
- Beauty and Elegance in Physics
- Overview of Transformers
Connections
26The lectures that define it or assume it, and the concepts next to it.
Concepts7
- Language model
- Loss function
- Emergent abilities
- Architecture selection for MoE
- Binary scaling
- Chinchilla scaling laws
Show 1 moreShow fewer
Lectures19
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Introduction to Large Language Models
- Marin: An Open Lab For Frontier AI
- Behind the Scenes of LLM Pre-training: StarCoder Use Case
- Beyond Brute Force: The Era of Adaptive Intelligence
- Approximation Theory
Show 13 moreShow fewer
- Scaling Laws
- Transformers in Vision: Tackling problems in Computer Vision
- AI for Fusion Energy: Accelerating STEP Tokamak Development
- Emergent Behaviors in Foundational Models
- Speculations on Test-Time Scaling for Large Language Models (o1)
- Training Sand to Think: Artificial General Intelligence & Future of Physics
- Related Rates
- Intuitions on Language Models and the History of Transformer Architectures
- Demystifying Mixtral of Experts
- Aerothermodynamics
- Interactions with the Physical Environment
- Beauty and Elegance in Physics
- Overview of Transformers