Sign inSign in
ConceptAI & ML

Tokenization

The process for translating strings or text into sequences of tokens and vice versa.
as Andrej Karpathy defines itThe wording closest to all 9 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

String manipulation2Token2Byte pair encoding1Discrete characters1Embedding1Language model1Linear projection1Neural networks1

Opens up

Concepts that name this one as a prerequisite.

Lecturers

6

Each in their own terms, from the easiest level.

  • Andrej Karpathy

    Independent · Undergraduate

    A lossless translation of raw text into sequences of integer tokens.
    • 2:46State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
    • 3:15:35Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
    • 7:56Deep Dive into Large Language Models: Pretraining to RLHF
  • Jeremy Howard

    Undergraduate

    To create tokens from a string.
    • 5:06A Hacker's Guide to Language Models
  • Phillip Isola

    MIT · Undergraduate

    The process of chopping input data into chunks and projecting each chunk into a vector.
    • 12:29Architectures: Transformers
  • Brian Yu

    Harvard University · Undergraduate

    The process of splitting a sequence of characters into pieces, or tokens, such as splitting text into individual words.
    • 11:02:36Harvard CS50’s Artificial Intelligence with Python – Full University Course
  • Niles Salter

    Software You Can Love · Undergraduate

    Chunking up source code into irreducible semantic pieces called tokens.
    • 10:07Accelerating the Zig Tokenizer with AVX-512
  • Edo Dotan

    Tufts University · Graduate

    Converting discrete characters into a list of tokens where each token has a specific embedding vector.
    • 15:20Deep Learning in Biology

Lectures that assume it

13

The first 12 are shown.

Connections

29

The lectures that define it or assume it, and the concepts next to it.

Concepts7

Show 1 more

Lectures22

Show 16 more

6 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.