Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
Essential deep-learning diagnostics: visualizing saturated tanh tails, dead ReLUs, Kaiming He variance preservation, Batch Normalization internals, and gradient-to-data update-ratio histograms.
We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.
Links:
- makemore on github: https://github.com/karpathy/makemore
Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
Next · Lecture 5Building makemore Part 4: Becoming a Backprop Ninja (Manual Tensor Gradients)