ConceptAI & ML
Cross-attention
An attention mechanism where queries come from the decoder output while keys and values come from the encoder output.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Self-attention6Queries, keys, and values3Encoder-decoder models1Keys, queries, and values1Matrix multiplication1Transformer1Transformer architecture1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
5Each in their own terms, from the easiest level.
Phillip Isola
A mechanism where queries come from one set of tokens and keys and values come from another set, allowing modalities to interact.Andrej Karpathy
An attention mechanism where queries come from one source while keys and values come from a separate external source.Jason Wei & Hyung Won Chung
An attention mechanism in encoder-decoder architectures that connects the decoder to the encoder by allowing output vectors to attend to encoder outputs.Drew Jaegle
An attention mechanism replacing the input query with a learned query vector to achieve linear scaling complexity with respect to the input size.
Lectures that assume it
2Connections
16The lectures that define it or assume it, and the concepts next to it.
Concepts8
- Self-attention
- Queries, keys, and values
- Matrix multiplication
- CogAgent
- DEtection TRansformer
- Encoder-decoder transformer