Sign inSign in
ConceptAI & ML

Scaling laws

Empirical power-law relationships predicting model performance as a function of compute, dataset size, and parameter count.
as Sara Hooker defines itAdaption LabsThe wording closest to all 10 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Language model4Model parameters3Compute2Compute budget2Loss function2Model capacity2Model pretraining2Algorithm scaling1

Opens up

Concepts that name this one as a prerequisite.

Lecturers

9

Each in their own terms, from the easiest level.

  • Andrej Karpathy

    Undergraduate

    The mathematical relationships showing that as model size and compute increase, downstream task performance improves predictably.
    • 0:47Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
    • 25:43Introduction to Large Language Models
  • Loubna Ben Allal

    Stanford University · Undergraduate

    Scaling laws study the allocation of a compute budget between data size and model size.
    • 5:51Behind the Scenes of LLM Pre-training: StarCoder Use Case
  • Jeremy Bernstein

    MIT · Undergraduate

    Empirical relationships describing how model performance improves predictably with compute, dataset size, and number of parameters.
  • Phillip Isola

    MIT · Graduate

    Predictive relationships between the amount of compute or model size and the resulting model performance.
  • Lucas Beyer

    Stanford University · Graduate

    Empirical laws relating model performance to compute, dataset size, and parameter count.
    • 1:02:44Transformers in Vision: Tackling problems in Computer Vision
  • Lorenzo Zanisi

    UK Atomic Energy Authority · Graduate

    Empirical relations predicting how quickly plasma leaks out of magnetic confinement shapes based on experimental data.
    • 24:39AI for Fusion Energy: Accelerating STEP Tokamak Development
  • Percy Liang

    Stanford University · Research

    A technique that relates model performance to compute and data scale to ensure small-scale findings translate to large-scale models.
    • 14:02Marin: An Open Lab For Frontier AI
  • Irina Rish

    Université de Montréal, MILA · Research

    Mathematical relationships describing how model performance depends on parameters like model size, dataset size, and compute.
    • 41:47Emergent Behaviors in Foundational Models

Lectures that assume it

9

Connections

26

The lectures that define it or assume it, and the concepts next to it.

9 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.