Neural Networks: Zero to Hero

Demystifies PyTorch's autograd in 150 lines of clean Python by manually computing numerical derivatives, applying the multivariable chain rule, and verifying gradients against torch.Tensor.

Builds character-level autoregressive language models two ways—explicit transition count tables and gradient-based one-hot neural networks—proving they converge to identical negative log-likelihood loss.

Implements Bengio et al.'s landmark 2003 neural probabilistic language model from scratch: learned token embeddings, mini-batch SGD, learning-rate finder sweeps, and train/dev/test splits.

Essential deep-learning diagnostics: visualizing saturated tanh tails, dead ReLUs, Kaiming He variance preservation, Batch Normalization internals, and gradient-to-data update-ratio histograms.

Deletes loss.backward() and manually derives analytical tensor gradients through BatchNorm, linear layers, and fused cross-entropy softmax—checking every tensor to floating-point precision.

Transforms a flat MLP into a DeepMind WaveNet tree-structured architecture while building clean torch.nn-style Sequential and FlattenConsecutive abstractions.

Perhaps the single most-watched modern AI engineering lecture: constructs a full decoder-only Transformer (scaled dot-product causal self-attention, multi-head blocks, residuals, LayerNorm) live from an empty file.

Crisp architectural map of the four-stage LLM training pipeline (base pretraining, supervised finetuning, reward modeling, RLHF) plus practical inference psychology on token-by-token compute.

Reveals why tokenization is the root cause of weirdly brittle LLM behaviors (spelling, arithmetic, trailing whitespace, non-English token bloat) by implementing UTF-8 Byte Pair Encoding from scratch.

Four-hour systems-and-ML tour de force: reproduces OpenAI's GPT-2 (124M) from an empty file, optimizing throughput 10x via BF16 tensor cores, torch.compile, FlashAttention, cosine decay, and multi-GPU DDP.

A remarkably dense, hype-free 60-minute mental model of what an LLM actually is—two files on disk (weights + inference code) shaped by pretraining compression and post-training alignment.