ConceptAI & ML
Multi-head attention
An attention mechanism where queries, keys, and values are projected into multiple subspaces to allow the model to jointly attend to information from different representation subspaces in parallel.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Self-attention4Concatenation1Linear layer1Linear transformation1Matrix multiplication1Scaled dot-product attention1Tensor transpose and Reshaping1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
3Each in their own terms, from the easiest level.
Andrej Karpathy
Multi-head attention applies multiple scaled dot-product attentions in parallel across projected queries, keys, and values, then concatenates their results.Steven Feng
Using multiple learned attention matrices or models in parallel on the same sequence to capture diverse relationships.Ashish Vaswani
An attention mechanism that splits the hidden dimension into independent subspaces to compute attention across multiple heads in parallel.
Lectures that assume it
3Connections
16The lectures that define it or assume it, and the concepts next to it.
Concepts8
Show 2 moreShow fewer
Lectures8
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- How I Learned to Stop Worrying and Love the Transformer
- Overview of Transformers (Stanford CS25: Transformers United V6)
- Overview of Transformers
- Demystifying Mixtral of Experts