Opening Accelerating Gradient Descent: Momentum and Nesterov's Method
Accelerating Gradient Descent: Momentum and Nesterov's Method
- Length
- 49 min
- Level
- Undergraduate
- Recorded
- MIT2019
Chapter 1 of 7: Before the lecture
Strang tunes step size and momentum so the decay factor's becomes , then sketches Nesterov's method.
Momentum and Nesterov's method accelerate gradient descent by giving each step a memory of the previous one. Strang begins with the zigzag of steepest descent across the long, thin elliptical level sets of a badly conditioned quadratic. A momentum term turns the update into a three-level recurrence, rewritten as two first-order equations and followed one eigenvector at a time. Choosing step size and momentum to minimize the largest eigenvalue of the iteration matrix changes the decay factor from to . Session 23 of MIT 18.065 closes with Nesterov's variant, which evaluates the gradient at a shifted point, and assumes eigenvectors and positive definite matrices.
Momentum and Nesterov's method accelerate gradient descent by giving each step a memory of the previous one. Strang begins with the zigzag of steepest descent across the long, thin elliptical level sets of a badly conditioned quadratic. A momentum term turns the update into a three-level recurrence, rewritten as two first-order equations and followed one eigenvector at a time. Choosing step size and momentum to minimize the largest eigenvalue of the iteration matrix changes the decay factor from to . Session 23 of MIT 18.065 closes with Nesterov's variant, which evaluates the gradient at a shifted point, and assumes eigenvectors and positive definite matrices.

This lecture
Lecture 23 of 34 of Matrix Methods in Data Analysis, Signal Processing, and Machine Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Convex Optimization