Sign inSign in
ConceptAI & ML

Pre-training

The initial stage of training where a model learns a general understanding of the world from large amounts of internet text or private data.
as Lee Robinson defines itStanford UniversityThe wording closest to all 6 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Opens up

Concepts that name this one as a prerequisite.

Lecturers

4

Each in their own terms, from the easiest level.

  • Geoffrey Hinton

    University of Toronto · Introductory

    A technique to initialize the weights of a deep neural network before fine-tuning it on a specific task.
    • 11:20Why We Need Machine Learning: An Introduction to Neural Networks
  • Andrej Karpathy

    Undergraduate

    The first stage of training where a network is trained on a massive quantity of internet text to generate internet documents.
    • 1:27State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
    • 14:26Introduction to Large Language Models
    • 2:24:50Deep Dive into Large Language Models: Pretraining to RLHF
  • Percy Liang

    Stanford University · Research

    The initial phase of training a model on a large dataset before fine-tuning it for specific tasks.
    • 1:01:26Marin: An Open Lab For Frontier AI

Lectures that assume it

13

The first 12 are shown.

Connections

28

The lectures that define it or assume it, and the concepts next to it.

Concepts9

Show 3 more

Lectures19

Show 13 more

4 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.