ConceptAI & ML
Learning rate
A constant multiplier that determines the step size taken during gradient descent optimization.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
6Each in their own terms, from the easiest level.
Andrej Karpathy
A hyperparameter that controls the step size taken in the direction of the gradient during optimization.Brian Yu
A parameter that determines how quickly the weight values are updated during training.Pieter Abbeel
A tuning parameter that determines the step size at each iteration while moving toward a minimum or maximum.Gilbert Strang
The step size parameter multiplied by the gradient in gradient-based optimization methods.Jeremy Bernstein
A hyperparameter that determines the size of the steps taken during optimization.
Lectures that assume it
6- Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
- Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Training Neural Networks II
- Loss Functions and Optimization
- Transformers in Vision: Tackling problems in Computer Vision
Connections
22The lectures that define it or assume it, and the concepts next to it.
Concepts10
Show 4 moreShow fewer
Lectures12
- Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
- Harvard CS50’s Artificial Intelligence with Python – Full University Course
- The Linear Model II
- Machine Learning: Optimization and Neural Networks
- Minimizing a Function Step by Step
- Scaling Rules for Optimization
Show 6 moreShow fewer
- Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
- Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Training Neural Networks II
- Loss Functions and Optimization
- Transformers in Vision: Tackling problems in Computer Vision