ConceptAI & ML
Learning rate decay
A training strategy where the learning rate is systematically decreased over time, typically by a factor of 10, during the later stages of training to allow the model to fine-tune its parameters.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Lecturers
2Each in their own terms, from the easiest level.
Andrej Karpathy
A technique where the learning rate starts high and is scaled down as a function of the number of iterations to help the network settle into fine details near the end of training.Fei-Fei Li, Justin Johnson & Serena Yeung
A strategy of reducing the learning rate over the course of training according to a schedule such as step decay, exponential decay, or 1/t decay.
Connections
7The lectures that define it or assume it, and the concepts next to it.