ConceptAI & ML
Stochastic gradient descent
An optimization method that updates parameters using the gradient computed from a single randomly selected data point per iteration.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Gradient descent12Expectation2Expected value2Loss function2Backpropagation1Batch size1Convexity1Finite sums1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
9Each in their own terms, from the easiest level.
Gilbert Strang
A gradient descent variant that uses a mini-batch of training samples at each step instead of all the data at once.Fei-Fei Li, Justin Johnson & Serena Yeung
An approximation of gradient descent where the loss and gradient are computed on a small random subset of training examples called a minibatch.Brian Yu
A variation of gradient descent that calculates the gradient based on one data point at a time.Geoffrey Hinton
Computing how to change connection strengths based on a small batch of examples rather than the whole training set.Yaser Abu-Mostafa
A randomized version of gradient descent where parameter updates are computed using one randomly chosen training example at a time.Sara Beery
An approximation of gradient descent where the gradient is computed on a small subset or batch of data rather than the entire dataset.David Silver
An incremental gradient descent method that samples individual states to update parameters rather than computing the full expectation over all states.Chris Ré
An iterative optimization method that estimates the true gradient by computing the gradient on a single randomly sampled example.
Lectures that assume it
17The first 12 are shown.
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Training Neural Networks I
- Exploration of fundamental concepts in deep reinforcement learning
- Mechanism Design & Game-Theoretic Incentives for Collaborative Learning
- Privacy Amplification for Correlated-Noise Mechanisms via b-Min-Sep Subsampling
- Generative Models: Representation Learning Meets Generative Modeling
- Transfer Learning: Models
- Differentially Private Table-Image Multimodal Data Generation
- Leveraging Per-Instance Privacy for Machine Unlearning
- Generalization: Out-of-Distribution (OOD)
- Dual Averaging Method for Regularized Stochastic Learning and Online Optimization
Connections
39The lectures that define it or assume it, and the concepts next to it.
Concepts10
Show 4 moreShow fewer
Lectures29
- Harvard CS50’s Artificial Intelligence with Python – Full University Course
- Value Function Approximation & Deep Q-Networks (DQN)
- Training Neural Networks II
- The Deep Learning Revolution (Turing Award Lecture)
- Neural Networks
- How to Train a Neural Net
Show 23 moreShow fewer
- Neural Networks I: Feedforward Architectures & Backpropagation
- Stochastic Gradient Descent
- Loss Functions and Optimization
- Accelerating Gradient Descent: Momentum and Nesterov's Method
- Structure of Neural Networks for Deep Learning
- Backpropagation: Finding Partial Derivatives
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Training Neural Networks I
- Exploration of fundamental concepts in deep reinforcement learning
- Mechanism Design & Game-Theoretic Incentives for Collaborative Learning
- Privacy Amplification for Correlated-Noise Mechanisms via b-Min-Sep Subsampling
- Generative Models: Representation Learning Meets Generative Modeling
- Transfer Learning: Models
- Differentially Private Table-Image Multimodal Data Generation
- Leveraging Per-Instance Privacy for Machine Unlearning
- Generalization: Out-of-Distribution (OOD)
- Dual Averaging Method for Regularized Stochastic Learning and Online Optimization
- Scaling Rules for Optimization
- Transfer Learning: Data
- Foundational concepts in deep reinforcement learning, likely
- PyTorch Tutorial
- Emergent Behaviors in Foundational Models