ConceptAI & ML
Self-attention
A mechanism that allows a model to weigh the importance of different words in an input sequence relative to a target word.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
8Each in their own terms, from the easiest level.
Ashish Vaswani, Lucas Beyer & William Fedus
When your source and target sequences are the same, relating each element within a sequence to another element in the same sequence.Steven Feng
A mechanism that learns what to focus on for each token by computing relationships between different words.Ava Amini
The mechanism of identifying and attending to the most important parts of an input stream of information.Phillip Isola
An attention mechanism where every single token in the input sequence emits queries to the other tokens, allowing tokens to attend to similar tokens in the signal itself.Brian Yu
An attention mechanism that allows words in an input sequence to pay attention to other words in the input to better determine their contextual meaning.Drew Jaegle
An attention mechanism that relates different positions of a single sequence to compute a representation of the sequence.Ashish Vaswani
A content-based pooling mechanism where tokens compute dot products with all other tokens to pool information across positions.Jake Williams
A pivotal component in transformer architectures that requires a quadratic comparison of vectors to produce feature weights needed to model long-range dependencies in text.
Lectures that assume it
9- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- RL with Sequence Models & Language Models, Part 2
- Shaping the Future of AI from the History of Transformer Architectures
- Generative Models: Conditional Models
- Intuitions on Language Models and the History of Transformer Architectures
- Symbols and Grounding in Large Language Models
- Transformers in Diffusion Models for Image Generation and Beyond
- Transformers in Vision: Tackling problems in Computer Vision
Connections
29The lectures that define it or assume it, and the concepts next to it.
Concepts10
Lectures19
- Harvard CS50’s Artificial Intelligence with Python – Full University Course
- DeepMind's Perceiver and Perceiver IO: New data family architecture
- Self Attention and Non-parametric transformers (NPTs)
- Recurrent Neural Networks, Transformers, and Attention
- Transformers United: DL Models that have revolutionized NLP, CV, RL
- How I Learned to Stop Worrying and Love the Transformer
Show 13 moreShow fewer
- Architectures: Transformers
- Overview of Transformers (Stanford CS25: Transformers United V6)
- Transformers that Transform Well Enough to Support Near-Shallow Architectures
- Overview of Transformers
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- RL with Sequence Models & Language Models, Part 2
- Shaping the Future of AI from the History of Transformer Architectures
- Generative Models: Conditional Models
- Intuitions on Language Models and the History of Transformer Architectures
- Symbols and Grounding in Large Language Models
- Transformers in Diffusion Models for Image Generation and Beyond
- Transformers in Vision: Tackling problems in Computer Vision