ConceptAI & ML
Direct preference optimization
A method to optimize language models directly on preference data without training a separate reward model.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Language model3Bradley-Terry model2Reinforcement learning from human feedback2Supervised fine-tuning2Gradient ascent1Language model policies1Policy optimization1Preference data1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
5Each in their own terms, from the easiest level.
Steven Feng
DPO is an RL-free approach that directly trains the model to prefer human outputs to rank higher without requiring a separate reward model.Ming Ding
An alignment method using preference data pairs to update the model without requiring a separate reward model.Eric Mitchell
An algorithm that simplifies RLHF by reparameterizing the reward model directly into the policy, eliminating the need for an explicit reward model training and RL phase.Nathan Lambert
An algorithm that optimizes language models directly from preferences using gradient ascent without fitting a separate reward model.
Lectures that assume it
1Connections
17The lectures that define it or assume it, and the concepts next to it.
Concepts10
- Language model
- Bradley-Terry model
- Reinforcement learning from human feedback
- Supervised fine-tuning
- Gradient ascent
- DPO failure cases (first token cancellation)
Show 4 moreShow fewer
Lectures7
- POPRi: Private Federated Learning using Preference-Optimized Synthetic Data
- Reinforcement Learning from Human Feedback: Algorithms & Applications
- From Large Language Models to Large Multimodal Models
- Aligning Open Language Models
- Overview of Transformers (Stanford CS25: Transformers United V6)
- Overview of Transformers