Opening Reinforcement Learning from Human Feedback: Algorithms & Applications
Reinforcement Learning from Human Feedback: Algorithms & Applications
- Length
- 54 min
- Level
- Graduate
- Recorded
- UC Berkeley2023
Chapter 1 of 7: Before the lecture
Mitchell derives the DPO loss by writing the reward through the closed-form optimal policy, then tests it against PPO.
Reinforcement learning from human feedback (RLHF) turns a pre-trained language model such as GPT-3 into an instruction follower like ChatGPT. Eric Mitchell of Stanford, a co-author of direct preference optimization (DPO), gives this guest lecture in Berkeley's CS 285. He sets out the standard pipeline of pre-training, supervised fine-tuning, a Bradley-Terry reward model fit to pairwise preferences, and PPO. Mitchell then shows the KL-constrained optimal policy has a closed form, so the reward can be expressed through the policy and trained without reinforcement learning. Sentiment-generation experiments compare reward–KL frontiers, and a final section uses automatically generated preferences to reduce hallucination.
Reinforcement learning from human feedback (RLHF) turns a pre-trained language model such as GPT-3 into an instruction follower like ChatGPT. Eric Mitchell of Stanford, a co-author of direct preference optimization (DPO), gives this guest lecture in Berkeley's CS 285. He sets out the standard pipeline of pre-training, supervised fine-tuning, a Bradley-Terry reward model fit to pairwise preferences, and PPO. Mitchell then shows the KL-constrained optimal policy has a closed form, so the reward can be expressed through the policy and trained without reinforcement learning. Sentiment-generation experiments compare reward–KL frontiers, and a final section uses automatically generated preferences to reduce hallucination.
Contrastive Preference Learning: Learning from Human Feedback without RL
Rank Analysis of Incomplete Block Designs

This lecture
Lecture 63 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Proximal Policy Optimization Algorithms
Training language models to follow instructions with human feedback
Quality Diversity through Human Feedback
Constitutional AI: Harmlessness from AI Feedback
Direct Preference Optimization
Towards understanding sycophancy in language models
Fine-tuning language models for factuality