Opening Stochastic Gradient Descent
Stochastic Gradient Descent
- Length
- 53 min
- Level
- Undergraduate
- Recorded
- MIT2019
Chapter 1 of 8: Before the lecture
Sra uses one-dimensional least squares to show why SGD races ahead early, then fluctuates near the optimum.
Stochastic gradient descent trains large machine learning models by replacing the full gradient of a finite sum with the gradient of one randomly chosen term. Sra gives this guest lecture in MIT 18.065, casting machine learning problems as finite sum optimizations over massive datasets. He contrasts Cauchy's full gradient descent, too slow at that scale, with the stochastic method Robbins and Monro proposed around 1951. A one-dimensional least squares example from Bertsekas explains the rapid early progress and the fluctuation near the optimum. Later sections show the stochastic gradient is unbiased, tie convergence speed to its variance, and cover sampling without replacement, minibatches, step sizes and neural network generalization.
Stochastic gradient descent trains large machine learning models by replacing the full gradient of a finite sum with the gradient of one randomly chosen term. Sra gives this guest lecture in MIT 18.065, casting machine learning problems as finite sum optimizations over massive datasets. He contrasts Cauchy's full gradient descent, too slow at that scale, with the stochastic method Robbins and Monro proposed around 1951. A one-dimensional least squares example from Bertsekas explains the rapid early progress and the fluctuation near the optimum. Later sections show the stochastic gradient is unbiased, tie convergence speed to its variance, and cover sampling without replacement, minibatches, step sizes and neural network generalization.

This lecture
Lecture 25 of 34 of Matrix Methods in Data Analysis, Signal Processing, and Machine Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in