Sign inSign in
ConceptAI & ML

Learning rate decay

A training strategy where the learning rate is systematically decreased over time, typically by a factor of 10, during the later stages of training to allow the model to fine-tune its parameters.
as Andrej Karpathy defines itThe wording closest to all 3 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Lecturers

2

Each in their own terms, from the easiest level.

  • Andrej Karpathy

    Undergraduate

    A technique where the learning rate starts high and is scaled down as a function of the number of iterations to help the network settle into fine details near the end of training.
    • 2:20:42Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
  • Fei-Fei Li, Justin Johnson & Serena Yeung

    Stanford University · Undergraduate

    A strategy of reducing the learning rate over the course of training according to a schedule such as step decay, exponential decay, or 1/t decay.
    • 45:19Training Neural Networks II

Connections

7

The lectures that define it or assume it, and the concepts next to it.

2 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.