/
lectern.tv
Play15s30s1x0:00/--:--
DoneWatched
Expand
lectern.tv
41:44/2:34:45
14:20
17:40
20:00
35:00
1:07:45
Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
Demystifies PyTorch's autograd in 150 lines of clean Python by manually computing numerical derivatives, applying the multivariable chain rule, and verifying gradients against torch.Tensor.
Lecture guide · Written by AI from the lecture. For the lecture’s own words, open the Transcript tab.
Provides a step-by-step live derivation and implementation of reverse-mode automatic differentiation over a dynamic expression DAG, culminating in building a scalar-valued autograd engine called micrograd and training a multi-layer perceptron from scratch in pure Python.
This lecture assumes
- Neural Networkstaught in Artificial Intelligence at 43:38
- Gradienttaught in Optimization: A Bootcamp for Machine Learning, Inverse Problems, and Control at 10:28 and 1 more
- Computation Graphtaught in How to Train a Neural Net at 27:28
- Object-oriented programmingtaught in Overview and Introduction to Lisp: Procedural & Symbolic Abstraction at 24:12
- Directed Acyclic Graph (DAG)taught in How to Train a Neural Net at 29:09
- Object-Oriented Programming in Pythontaught in SQL & Relational Databases at 24:26
Key Takeaways
- Automatic differentiation evaluates gradients by recursively applying the calculus chain rule backward through a dynamic computation graph.
- Mathematical expressions, neurons, layers, and multi-layer perceptrons can be represented using simple recursive data structures in Python.
- During backpropagation, gradient contributions from multiple paths must be accumulated via addition rather than overwritten.
- Gradient descent iteratively updates parameters by taking small steps in the direction opposite to the loss gradient to minimize error.
Concepts Explained36
Autograd engine—A tool that implements backpropagation to efficiently evaluate the gradients of a loss function with respect to a network's weights.
Backpropagation—An algorithm that allows you to efficiently evaluate the gradient of a loss function with respect to the weights of a neural network.
Derivative—A derivative gives the rate of change or slope of a function at any given input point.
Derivative (Numerical Approximation)—The derivative measures the sensitivity of a function's output to a small change in its input, approximated by taking a very small step h.
Examples, Proofs & Derivations
Part of course·Lecture 1 of 11
NN-Z2H: Neural Networks: Zero to Hero
Start of sequence
Next · Lecture 2Opening lecture of 11-session syllabus
Spelled-Out Intro to Language Modeling: Building makemore (Bigram & Cross-Entropy)
About the lecturer
Founder, Eureka Labs · Former Senior Director of AI at Tesla & Founding Member of OpenAI
Deep learning researcher and educator who created Stanford CS231n and Neural Networks: Zero to Hero.
1 course · 13 lectures in archiveView profile
Discussion
Share a seminar note, question, or timestamped reference (e.g. 14:20)...Sign in