ConceptAI & ML
Language modeling
A task where a network reads a sequence of text and learns to predict the next character or word.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Opens up
Concepts that name this one as a prerequisite.
Lecturers
2Each in their own terms, from the easiest level.
Jake Williams
Strict language modeling is a setup where a conversation between a human and a lamp is modeled by extending context windows similar to decoder-only architectures.
Lectures that assume it
14The first 12 are shown.
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Speculations on Test-Time Scaling for Large Language Models (o1)
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
- A Hacker's Guide to Language Models
- Computing, Life, and Intelligence
- Transformers United: DL Models that have revolutionized NLP, CV, RL
- Machine Learning: Naive Bayes
- How I Learned to Stop Worrying and Love the Transformer
- Mixture of Experts (MoE) paradigm and the Switch Transformer
- Hacker's Guide to Deep Learning
- Symbols and Grounding in Large Language Models
Connections
25The lectures that define it or assume it, and the concepts next to it.
Concepts9
- Context window
- Decoder-only architecture
- Recurrent neural network
- Sequence-to-sequence models
- Masked language modeling
- AI complete
Show 3 moreShow fewer
Lectures16
- Recurrent Neural Networks
- Transformers that Transform Well Enough to Support Near-Shallow Architectures
- Let's Build GPT: From Scratch, in Code, Spelled Out
- Speculations on Test-Time Scaling for Large Language Models (o1)
- Let's Build the GPT Tokenizer: Byte Pair Encoding & Unicode Internals
- State of GPT: Pretraining, SFT, Reward Modeling & RLHF Pipeline
Show 10 moreShow fewer
- A Hacker's Guide to Language Models
- Computing, Life, and Intelligence
- Transformers United: DL Models that have revolutionized NLP, CV, RL
- Machine Learning: Naive Bayes
- How I Learned to Stop Worrying and Love the Transformer
- Mixture of Experts (MoE) paradigm and the Switch Transformer
- Hacker's Guide to Deep Learning
- Symbols and Grounding in Large Language Models
- From Large Language Models to Large Multimodal Models
- Language Models: Prompting, Chain-of-Thought, and Instruction-Tuning