Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
Reveals why tokenization is the root cause of weirdly brittle LLM behaviors (spelling, arithmetic, trailing whitespace, non-English token bloat) by implementing UTF-8 Byte Pair Encoding from scratch.
The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
intro: Tokenization, GPT-2 paper, tokenization-related issues
State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
Next · Lecture 10Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb