Sign inSign in
ConceptAI & ML

Attention mechanism

A mechanism that allows models to focus on particular aspects of the input text by computing weights via queries, keys, and values.
as Steven Feng defines itStanford UniversityThe wording closest to all 6 definitions
Watch it explained

Know first

Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.

Opens up

Concepts that name this one as a prerequisite.

Lecturers

6

Each in their own terms, from the easiest level.

  • Brian Yu

    Harvard University · Undergraduate

    A mechanism that allows the model to decide which input values are important to pay attention to when generating each part of the output sequence.
    • 11:34:13Harvard CS50’s Artificial Intelligence with Python – Full University Course
  • Ellie Pavlick

    Brown University · Graduate

    A read-write mechanism that computes token representations as a weighted combination of tokens from previous layers.
    • 20:17Symbols and Grounding in Large Language Models
  • Elahe Kamel Targhi

    Delft University of Technology · Graduate

    A mechanism that gives the parts of the pattern or signal with more important information a higher weight in the prediction.
    • 31:52From Outcrop Observations to Dynamic Simulations: Generating Geologically Plausible Fracture Networks
  • Trenton Bricken

    Stanford University · Graduate

    A computational process that uses keys and values to weight information, which can be interpreted as an associative memory operation.
    • 35:23Neuroscience-Inspired Artificial Intelligence
  • Yoshua Bengio

    MILA, Université de Montréal · Research

    Algorithmic components in deep learning that allow models to focus selectively on parts of the input data.
    • 7:49From System 1 to System 2 Deep Learning

Lectures that assume it

16

The first 12 are shown.

Connections

32

The lectures that define it or assume it, and the concepts next to it.

Concepts10

Show 4 more

Lectures22

Show 16 more

6 lecturers. Definitions are written by AI from each lecture’s transcript. A time with a play mark starts the lecture at the moment the lecturer says it.