ConceptAI & ML
Attention mechanism
A mechanism that allows models to focus on particular aspects of the input text by computing weights via queries, keys, and values.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Self-attention3Sliding window attention2Attention for attribute extraction1Attention head tensor representation1Attention scores1Bidirectionality in attention1Dependency parse1Eigenvalue analysis of attention circuits1FlashAttention1Identity+successor operation1Image Captioning with attention1Influence function1
Lecturers
6Each in their own terms, from the easiest level.
Brian Yu
A mechanism that allows the model to decide which input values are important to pay attention to when generating each part of the output sequence.Ellie Pavlick
A read-write mechanism that computes token representations as a weighted combination of tokens from previous layers.Elahe Kamel Targhi
A mechanism that gives the parts of the pattern or signal with more important information a higher weight in the prediction.Trenton Bricken
A computational process that uses keys and values to weight information, which can be interpreted as an associative memory operation.Yoshua Bengio
Algorithmic components in deep learning that allow models to focus selectively on parts of the input data.
Lectures that assume it
16The first 12 are shown.
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Self Attention and Non-parametric transformers (NPTs)
- Recurrent Neural Networks
- Shaping the Future of AI from the History of Transformer Architectures
- Transformer Circuits, Induction Heads, In-Context Learning
- AI Seminar: Actionable Interpretability for Language Model Reasoning
- Marin: An Open Lab For Frontier AI
- Demystifying Mixtral of Experts
- Transformers United: DL Models that have revolutionized NLP, CV, RL
- How I Learned to Stop Worrying and Love the Transformer
- Architectures: Transformers
- Retrieval Augmented Language Models
Connections
32The lectures that define it or assume it, and the concepts next to it.
Concepts10
Lectures22
- Harvard CS50’s Artificial Intelligence with Python – Full University Course
- From System 1 to System 2 Deep Learning
- Symbols and Grounding in Large Language Models
- From Outcrop Observations to Dynamic Simulations: Generating Geologically Plausible Fracture Networks
- Neuroscience-Inspired Artificial Intelligence
- Overview of Transformers
Show 16 moreShow fewer
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Self Attention and Non-parametric transformers (NPTs)
- Recurrent Neural Networks
- Shaping the Future of AI from the History of Transformer Architectures
- Transformer Circuits, Induction Heads, In-Context Learning
- AI Seminar: Actionable Interpretability for Language Model Reasoning
- Marin: An Open Lab For Frontier AI
- Demystifying Mixtral of Experts
- Transformers United: DL Models that have revolutionized NLP, CV, RL
- How I Learned to Stop Worrying and Love the Transformer
- Architectures: Transformers
- Retrieval Augmented Language Models
- Transformers in Diffusion Models for Image Generation and Beyond
- Theoretical Foundations of Modern Machine Learning Paradigms
- Reasonable AI
- Emergent Behaviors in Foundational Models