ConceptAI & ML
Tokenization
The process for translating strings or text into sequences of tokens and vice versa.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
String manipulation2Token2Byte pair encoding1Discrete characters1Embedding1Language model1Linear projection1Neural networks1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
6Each in their own terms, from the easiest level.
Andrej Karpathy
A lossless translation of raw text into sequences of integer tokens.Jeremy Howard
To create tokens from a string.Phillip Isola
The process of chopping input data into chunks and projecting each chunk into a vector.Brian Yu
The process of splitting a sequence of characters into pieces, or tokens, such as splitting text into individual words.Niles Salter
Chunking up source code into irreducible semantic pieces called tokens.Edo Dotan
Converting discrete characters into a list of tokens where each token has a specific embedding vector.
Lectures that assume it
13The first 12 are shown.
- DeepMind's Perceiver and Perceiver IO: New data family architecture
- Artificial Intelligence
- Architectures: Memory
- Shaping the Future of AI from the History of Transformer Architectures
- Transformer Circuits, Induction Heads, In-Context Learning
- Private Adaptations of Large Language Models
- What Humans Want: Models of Preference and Reward in Minds and Machines
- Intuitions on Language Models and the History of Transformer Architectures
- Demystifying Mixtral of Experts
- Large Language Models and Concepts in Natural Language
- From Large Language Models to Large Multimodal Models
- Ship Reliable RAG: Evals That Grow with Your Retrieval Complexity
Connections
29The lectures that define it or assume it, and the concepts next to it.
Concepts7
Show 1 moreShow fewer
Lectures22
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- Harvard CS50’s Artificial Intelligence with Python – Full University Course
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Deep Dive into Large Language Models: Pretraining to RLHF
- A Hacker's Guide to Language Models
Show 16 moreShow fewer
- Architectures: Transformers
- Accelerating the Zig Tokenizer with AVX-512
- Deep Learning in Biology
- DeepMind's Perceiver and Perceiver IO: New data family architecture
- Artificial Intelligence
- Architectures: Memory
- Shaping the Future of AI from the History of Transformer Architectures
- Transformer Circuits, Induction Heads, In-Context Learning
- Private Adaptations of Large Language Models
- What Humans Want: Models of Preference and Reward in Minds and Machines
- Intuitions on Language Models and the History of Transformer Architectures
- Demystifying Mixtral of Experts
- Large Language Models and Concepts in Natural Language
- From Large Language Models to Large Multimodal Models
- Ship Reliable RAG: Evals That Grow with Your Retrieval Complexity
- Overview of Transformers (Stanford CS25: Transformers United V6)