ConceptAI & ML
Transformer
A neural network architecture that uses self-attention mechanisms to process sequences in parallel, often replacing recurrent networks.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
8Each in their own terms, from the easiest level.
Andrej Karpathy
The neural network architecture that does all the heavy lifting under the hood for modern language models.Geoffrey Hinton
A kind of neural network architecture introduced by Google researchers in 2017 capable of advanced natural language tasks.Ashish Vaswani, Lucas Beyer & William Fedus
A neural network architecture that uses self-attention to model dependencies between tokens in a sequence, allowing for parallel computation.Phillip Isola
The standard deep learning architecture that relies on attention mechanisms and operates over tokens to capture both local and global relationships.Gillian Tett
A type of deep learning architecture based on self-attention mechanisms used for probabilistic prediction systems in modern AI.Sergey Levine
A sequence model that processes tokens through position-wise encoders, masked self-attention layers, and nonlinear transformations to output token distributions.Drew Jaegle
A general-purpose neural network architecture that uses self-attention mechanisms without relying on task-specific spatial convolutions.
Lectures that assume it
34The first 12 are shown.
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Deep Dive into Large Language Models: Pretraining to RLHF
- Architectures: Memory
- The Relational Model & Relational Algebra
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- Self Attention and Non-parametric transformers (NPTs)
- Transformer Circuits, Induction Heads, In-Context Learning
- Generative Models: Conditional Models
- Intuitions on Language Models and the History of Transformer Architectures
- Demystifying Mixtral of Experts
Connections
52The lectures that define it or assume it, and the concepts next to it.
Concepts10
Lectures42
- Let's Build GPT: From Scratch, in Code, Spelled Out
- RL with Sequence Models & Language Models, Part 2
- DeepMind's Perceiver and Perceiver IO: New data family architecture
- Nonlinear Dynamics
- Digital vs. Biological Intelligence (Vienna Gödel Lecture)
- Transformers United: DL Models that have revolutionized NLP, CV, RL
Show 36 moreShow fewer
- Whom Do We Trust? How AI Is (Re)Shaping Our Interactions Today
- Architectures: Transformers
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Deep Dive into Large Language Models: Pretraining to RLHF
- Architectures: Memory
- The Relational Model & Relational Algebra
- Decision Transformer: Reinforcement Learning via Sequence Modeling
- Self Attention and Non-parametric transformers (NPTs)
- Transformer Circuits, Induction Heads, In-Context Learning
- Generative Models: Conditional Models
- Intuitions on Language Models and the History of Transformer Architectures
- Demystifying Mixtral of Experts
- Inference Methods for Deep Learning
- Objective-Driven AI: World Models that Learn, Remember, Reason, and Plan
- An Observation on Generalization and Unsupervised Compression
- Beyond Brute Force: The Era of Adaptive Intelligence
- Mixture of Experts (MoE) paradigm and the Switch Transformer
- Representation Learning: Reconstruction-Based
- Symbols and Grounding in Large Language Models
- From Large Language Models to Large Multimodal Models
- Transformers in Diffusion Models for Image Generation and Beyond
- Offline Reinforcement Learning II
- Language Models: Prompting, Chain-of-Thought, and Instruction-Tuning
- Transformers in Vision: Tackling problems in Computer Vision
- Imitation Learning III: Inverse RL & Generative Adversarial Imitation Learning (GAIL)
- From Large Language Models to General Artificial Intelligence?
- Computational Thinking on Learning Models
- Overview of Transformers (Stanford CS25: Transformers United V6)
- Guest Lecture: Dorsa Sadigh (UC Berkeley CS 285)
- Neuroscience-Inspired Artificial Intelligence
- Emergent Behaviors in Foundational Models
- An Occam's Razor Principle for Transformers?
- Beyond LLMs: Agents, Emergent Abilities, Intermediate-Guided Reasoning, BabyLM
- Mojo: Modular's Unified Device Accelerator Language