Sign inSign in
ConceptAI & ML

Cross-attention

An attention mechanism where queries come from the decoder output while keys and values come from the encoder output.
as Steven Feng defines itStanford UniversityThe wording closest to all 6 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Opens up

Concepts that name this one as a prerequisite.

Lecturers

5

Each in their own terms, from the easiest level.

  • Phillip Isola

    MIT · Undergraduate

    A mechanism where queries come from one set of tokens and keys and values come from another set, allowing modalities to interact.
    • 54:16Generative Models: Conditional Models
    • 1:13:32Architectures: Transformers
  • Andrej Karpathy

    Undergraduate

    An attention mechanism where queries come from one source while keys and values come from a separate external source.
    • 1:15:46Let's Build GPT: From Scratch, in Code, Spelled Out
  • Jason Wei & Hyung Won Chung

    Stanford University · Undergraduate

    An attention mechanism in encoder-decoder architectures that connects the decoder to the encoder by allowing output vectors to attend to encoder outputs.
    • 50:19Intuitions on Language Models and the History of Transformer Architectures
  • Drew Jaegle

    Stanford University · Graduate

    An attention mechanism replacing the input query with a learned query vector to achieve linear scaling complexity with respect to the input size.
    • 10:43DeepMind's Perceiver and Perceiver IO: New data family architecture

Lectures that assume it

2

Connections

16

The lectures that define it or assume it, and the concepts next to it.

5 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.