ConceptAI & ML
Softmax
A function that converts a vector of logits into a probability distribution by exponentiating each value and normalizing by the sum.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
8Each in their own terms, from the easiest level.
Sara Beery
A function that takes a vector of real numbers and normalizes it into a probability distribution.Andrej Karpathy
An activation function that exponentiates and normalizes logits to output a valid probability distribution.Christopher Manning
A probability formula that exponentiates and normalizes vector scores.Patrick Winston
A function that takes an output vector and normalizes the values into probabilities across multiple classes.Pieter Abbeel
A mathematical function that converts a vector of raw scores (activations) into a probability distribution that sums to one.Phillip Isola
A function that maps an arbitrary real-valued vector to a probability distribution over classes, residing on a simplex.Trenton Bricken
A function that normalizes a vector of values into a probability distribution, making larger values more prominent.
Lectures that assume it
15The first 12 are shown.
- Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
- Building makemore Part 4: Becoming a Backprop Ninja (Manual Tensor Gradients)
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Machine Learning: Optimization and Neural Networks
- Deep Dive into Large Language Models: Pretraining to RLHF
- Recurrent Neural Networks, Transformers, and Attention
- From System 1 to System 2 Deep Learning
- Transfer Learning: Models
- How I Learned to Stop Worrying and Love the Transformer
- GSQ-RCO: Better Non-Uniform GGUF Quantization
Connections
33The lectures that define it or assume it, and the concepts next to it.
Concepts10
Show 4 moreShow fewer
Lectures23
- Architectures: Memory
- Spelled-Out Intro to Language Modeling: Building makemore (Bigram & Cross-Entropy)
- Natural Language Processing with Deep Learning: Introduction & Word Vectors
- Deep Neural Networks: MIT Artificial Intelligence
- Neural Networks I: Feedforward Architectures & Backpropagation
- Machine Learning: Perceptrons and Logistic Regression
Show 17 moreShow fewer
- Representation Learning: Reconstruction-Based
- Neuroscience-Inspired Artificial Intelligence
- Building makemore Part 3: Activations, Gradients, Kaiming Init & BatchNorm
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Building makemore Part 2: Multi-Layer Perceptron Language Models (Bengio et al. 2003)
- Building makemore Part 4: Becoming a Backprop Ninja (Manual Tensor Gradients)
- Let's Reproduce GPT-2 (124M): Distributed PyTorch, FlashAttention & FineWeb
- Machine Learning: Optimization and Neural Networks
- Deep Dive into Large Language Models: Pretraining to RLHF
- Recurrent Neural Networks, Transformers, and Attention
- From System 1 to System 2 Deep Learning
- Transfer Learning: Models
- How I Learned to Stop Worrying and Love the Transformer
- GSQ-RCO: Better Non-Uniform GGUF Quantization
- Mixture of Experts (MoE) paradigm and the Switch Transformer
- Introduction to Deep Learning
- Transformers that Transform Well Enough to Support Near-Shallow Architectures