ConceptAI & ML
Tokenizer
A tokenizer converts raw text as a string to some sequence of integers according to some vocabulary of possible elements.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
3Each in their own terms, from the easiest level.
Jim Harris
A tokenizer takes all the words and symbols of language and assigns a number to them.Niles Salter
A component or process that breaks a stream of text into smaller elements like tokens for parsing.
Lectures that assume it
3Connections
15The lectures that define it or assume it, and the concepts next to it.
Concepts9
Show 3 moreShow fewer
Lectures6
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Accelerating the Zig Tokenizer with AVX-512
- Mythbusting AI: Understanding Large Language Models and Their Limitations
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Deep Learning in Biology