Sign inSign in
ConceptAI & ML

Self-attention

A mechanism that allows a model to weigh the importance of different words in an input sequence relative to a target word.
as Steven Feng defines itStanford UniversityThe wording closest to all 10 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Opens up

Concepts that name this one as a prerequisite.

Lecturers

8

Each in their own terms, from the easiest level.

  • Ashish Vaswani, Lucas Beyer & William Fedus

    Stanford University · Undergraduate

    When your source and target sequences are the same, relating each element within a sequence to another element in the same sequence.
    • 1:44Self Attention and Non-parametric transformers (NPTs)
    • 7:55Transformers United: DL Models that have revolutionized NLP, CV, RL
  • Steven Feng

    Stanford University · Undergraduate

    A mechanism that learns what to focus on for each token by computing relationships between different words.
    • 12:28Overview of Transformers (Stanford CS25: Transformers United V6)
  • Ava Amini

    MIT · Undergraduate

    The mechanism of identifying and attending to the most important parts of an input stream of information.
    • 48:11Recurrent Neural Networks, Transformers, and Attention
  • Phillip Isola

    MIT · Undergraduate

    An attention mechanism where every single token in the input sequence emits queries to the other tokens, allowing tokens to attend to similar tokens in the signal itself.
    • 41:04Architectures: Transformers
  • Brian Yu

    Harvard University · Undergraduate

    An attention mechanism that allows words in an input sequence to pay attention to other words in the input to better determine their contextual meaning.
    • 11:43:38Harvard CS50’s Artificial Intelligence with Python – Full University Course
  • Drew Jaegle

    Stanford University · Graduate

    An attention mechanism that relates different positions of a single sequence to compute a representation of the sequence.
    • 3:28DeepMind's Perceiver and Perceiver IO: New data family architecture
  • Ashish Vaswani

    Stanford University · Graduate

    A content-based pooling mechanism where tokens compute dot products with all other tokens to pool information across positions.
    • 21:36How I Learned to Stop Worrying and Love the Transformer
  • Jake Williams

    Stanford University · Graduate

    A pivotal component in transformer architectures that requires a quadratic comparison of vectors to produce feature weights needed to model long-range dependencies in text.
    • 3:17Transformers that Transform Well Enough to Support Near-Shallow Architectures

Lectures that assume it

9

Connections

29

The lectures that define it or assume it, and the concepts next to it.

Concepts10

Show 4 more

Lectures19

Show 13 more

8 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.