ConceptAI & ML
Positional embedding
A learned vector representation added to each token embedding to give the transformer information about the token's absolute position in the sequence.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Lecturers
3Each in their own terms, from the easiest level.
Andrej Karpathy
An embedding table that assigns a vector to each token's position from zero to block size minus one, allowing the model to incorporate token order.Ashish Vaswani, Lucas Beyer & William Fedus
A technique to impart the notion of ordering to the input, as self-attention is inherently an unordered function.Drew Jaegle
A feature that denotes where a point is located in an input sequence.
Connections
8The lectures that define it or assume it, and the concepts next to it.