Sign inSign in
ConceptAI & ML

Positional embedding

A learned vector representation added to each token embedding to give the transformer information about the token's absolute position in the sequence.
as Andrej Karpathy defines itThe wording closest to all 4 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Lecturers

3

Each in their own terms, from the easiest level.

  • Andrej Karpathy

    Undergraduate

    An embedding table that assigns a vector to each token's position from zero to block size minus one, allowing the model to incorporate token order.
    • 1:00:38Let's Build GPT: From Scratch, in Code, Spelled Out
  • Ashish Vaswani, Lucas Beyer & William Fedus

    Stanford University · Undergraduate

    A technique to impart the notion of ordering to the input, as self-attention is inherently an unordered function.
    • 11:46Transformers United: DL Models that have revolutionized NLP, CV, RL
  • Drew Jaegle

    Stanford University · Graduate

    A feature that denotes where a point is located in an input sequence.
    • 30:00DeepMind's Perceiver and Perceiver IO: New data family architecture

Connections

8

The lectures that define it or assume it, and the concepts next to it.

3 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.