Sign inSign in
ConceptAI & ML

Multi-head attention

An attention mechanism where queries, keys, and values are projected into multiple subspaces to allow the model to jointly attend to information from different representation subspaces in parallel.
as Andrej Karpathy defines itThe wording closest to all 5 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Opens up

Concepts that name this one as a prerequisite.

Lecturers

3

Each in their own terms, from the easiest level.

  • Andrej Karpathy

    Undergraduate

    Multi-head attention applies multiple scaled dot-product attentions in parallel across projected queries, keys, and values, then concatenates their results.
    • 1:22:09Let's Build GPT: From Scratch, in Code, Spelled Out
  • Steven Feng

    Stanford University · Undergraduate

    Using multiple learned attention matrices or models in parallel on the same sequence to capture diverse relationships.
    • 14:04Overview of Transformers (Stanford CS25: Transformers United V6)
    • 12:38Overview of Transformers
  • Ashish Vaswani

    Stanford University · Graduate

    An attention mechanism that splits the hidden dimension into independent subspaces to compute attention across multiple heads in parallel.
    • 27:07How I Learned to Stop Worrying and Love the Transformer

Lectures that assume it

3

Connections

16

The lectures that define it or assume it, and the concepts next to it.

3 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.