/
lectern.tv
Play15s30s1x0:00/--:--
DoneWatched
Expand
lectern.tv
14:49/1:19:34
1:26
3:08
3:21
2:00
3:15
3:44
2:05
1:37
5:32
2:58
1:01
0:41
2:10
3:13
1:48
0:51
1:15
0:45
2:03
0:36
4:31
0:54
4:41
1:55
0:45
2:42
3:36
0:37
1:35
1:40
2:25
1:43
4:07
3:44
How to Train a Neural Net
Learn how neural network training actually works under the hood through backpropagation, computational graphs, and differentiable programming.
Lecture guide · Written by AI from the lecture. For the lecture’s own words, open the Transcript tab.
Explains the mechanics of training neural networks, covering gradient descent, stochastic gradient descent (SGD), momentum, and loss landscapes. It then introduces computational graphs and details how backpropagation operates through chains, multi-layer perceptrons, and directed acyclic graphs (DAGs) using matrix calculus. Finally, it explores differentiable programming, parameter sharing, and optimizing inputs versus parameters.
This lecture assumes
- Gradienttaught in Optimization: A Bootcamp for Machine Learning, Inverse Problems, and Control at 10:28 and 1 more
- Derivativetaught in Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd at 8:11
- Loss Functiontaught in Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd at 1:52:20
- Optimizationtaught in Optimization: A Bootcamp for Machine Learning, Inverse Problems, and Control at 1:04
- Activation Functiontaught in Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd at 54:17
- Matrix multiplicationtaught in States, Logic & Complex Vector Spaces at 1:24:14
Key Takeaways
- Gradient descent finds parameters by moving iteratively in the direction of maximal slope on a loss landscape.
- Stochastic gradient descent approximates full dataset gradients using small random subsets or batches, trading stability for speed.
- Backpropagation uses the chain rule and dynamic programming to efficiently compute and propagate gradients across shared terms in a computation graph.
- Training a linear layer involves multiplying forward by weights and backward by transposed weights.
Concepts Explained28
Gradient Descent—An iterative optimization algorithm that moves parameters in the direction of the maximal gradient of the cost function to find a minimum.
Stochastic Gradient Descent (SGD)—An approximation of gradient descent where the gradient is computed on a small subset or batch of data rather than the entire dataset.
Momentum—An addition to gradient descent that biases gradient steps to continue in the direction of the previous update to gain speed and navigate loss landscapes.
Differentiable Loss Landscapes—Functions mapping parameters to loss values that possess valid derivatives everywhere or on one-sided limits.
Examples, Proofs & Derivations
Works & Sources Cited4
Paper
Gaussian Error Linear Units (GELUs)—arXiv:1606.08415, cited for GELU activation function
Paper
Neural Module Networks—Andreas et al., 2017, cited for modular neural network architectures.
Paper
CLIP—Radford et al., cited for contrastive pre-training slide
Paper
Neural Model Networks—Andreas et al., cited in source permissions