Spelled-Out Intro to Language Modeling: Building makemore (Bigram & Cross-Entropy)
Founder of Eureka Labs, Former Director of AI at Tesla
Builds character-level autoregressive language models two ways—explicit transition count tables and gradient-based one-hot neural networks—proving they converge to identical negative log-likelihood loss.
We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).
Links:
- makemore on github: https://github.com/karpathy/makemore
Spelled-Out Intro to Neural Networks & Backpropagation: Building micrograd
Next · Lecture 3Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)