The Geometry of Linear Equations
Gilbert Strang · 39:49
Transposes, Permutations & Vector Spaces R^n
Gilbert Strang · 47:41
Convex Functions — First/Second-Order Conditions & Epigraphs
Stephen Boyd · 1:17:14
Saddle-Point Interpretation, KKT Conditions & Perturbation
Stephen Boyd · 1:09:19
The Learning Problem
Yaser Abu-Mostafa · 1:21:27
Is Learning Feasible?
Yaser Abu-Mostafa · 1:16:48
Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
Andrej Karpathy · 1:55:57
Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
Founder of Eureka Labs
Essential deep-learning diagnostics: visualizing saturated tanh tails, dead ReLUs, Kaiming He variance preservation, Batch Normalization internals, and gradient-to-data update-ratio histograms.
We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.