Building makemore Part 4: Becoming a Backprop Ninja (Manual Tensor Gradients)
Deletes loss.backward() and manually derives analytical tensor gradients through BatchNorm, linear layers, and fused cross-entropy softmax—checking every tensor to floating-point precision.
We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(): through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get a strong intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks.
!!!!!!!!!!!!
I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here:
Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
Next · Lecture 6Building makemore Part 5: Hierarchical Dilated Causal Convolutions (WaveNet)