Opening Metrized Deep Learning
Metrized Deep Learning
- Length
- 1 h 8 min
- Level
- Graduate
- Recorded
- MIT2026
Chapter 1 of 10: Before the lecture
Bernstein composes layer norms into a modular norm and derives modular dualization, behind nanoGPT speed records.
Metrized deep learning treats a neural network as a composition of modules, each carrying a norm on its weights. In session 22 of MIT 6.7960, Bernstein starts from the Gauss-Newton decomposition, which splits the second-order loss term into model curvature and error curvature. Induced operator norms then bound these terms through Lipschitz smoothness and continuity conditions. A formal theory of modules checks linear and embedding layers as well-normed, with series and parallel rules that preserve the property and track sensitivity. The final part derives modular dualization, combining layer-wise duality maps into a steepest-descent step; the material assumes multivariable calculus, operator norms and basic optimization theory.
Metrized deep learning treats a neural network as a composition of modules, each carrying a norm on its weights. In session 22 of MIT 6.7960, Bernstein starts from the Gauss-Newton decomposition, which splits the second-order loss term into model curvature and error curvature. Induced operator norms then bound these terms through Lipschitz smoothness and continuity conditions. A formal theory of modules checks linear and embedding layers as well-normed, with series and parallel rules that preserve the property and track sensitivity. The final part derives modular dualization, combining layer-wise duality maps into a steepest-descent step; the material assumes multivariable calculus, operator norms and basic optimization theory.
Visualizing the loss landscape of neural nets
On the distance between two neural networks and the stability of learning
A spectral condition for feature learning

This lecture
Lecture 22 of 24 of Deep Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Scalable Optimization in the Modular Norm
Shampoo: Preconditioned Stochastic Gradient Descent