Transformers United

This introductory lecture for Stanford's CS25 series details the foundational Transformer architecture, including its attention mechanisms, encoder-decoder structure, and essential components. It contextualizes the model's impact across NLP, computer vision, and reinforcement learning applications.

In this talk, Lucas discusses some of the ways transformers have been applied to problems in Computer Vision.

This lecture introduces the Decision Transformer architecture, reframing reinforcement learning as a conditional sequence modeling problem. It details how this approach, by conditioning an autoregressive model on desired returns, states, and actions, generates optimal future actions, matching or exceeding state-of-the-art offline RL baselines.

This lecture introduces the Mixture of Experts (MoE) paradigm, contrasting it with traditional dense models by selectively activating parameters for each input. It details the Switch Transformer, an architecture addressing MoE's complexity, communication costs, and training instability, demonstrating its application to large-scale language models with significant pre-training speedups.

Drew Jaegle presents DeepMind's Perceiver and Perceiver IO, exploring an asymmetric attention mechanism that allows these architectures to scale to high-dimensional and multimodal data without domain-specific assumptions, outperforming standard Transformers.

This session from Stanford's CS25 course introduces the foundational intuitions behind Transformer architectures, then delves into the theoretical and practical aspects of Non-Parametric Transformers (NPTs), a model recently accepted at NeurIPS.

This lecture explores mechanistic interpretability in transformer language models, focusing on "induction head circuits." It elucidates how these attention head pairs enable in-context learning by allowing models to repeat or translate text, and mimic functions from prior examples.

This lecture explores how attention mechanisms within Transformers can approximate sparse distributed memory, drawing parallels to neuroscience for advanced AI architectures. It analyzes the computational mechanisms enabling such approximations.

Ashish Vaswani, a co-author of "Attention Is All You Need," discusses the foundational motivations and historical evolution of the Transformer model. The session concludes with an overview of promising future research directions in Transformer architectures.