Opening RL with Sequence Models & Language Models, Part 2
RL with Sequence Models & Language Models, Part 2
- Length
- 24 min
- Level
- Graduate
- Recorded
- UC Berkeley2023
Chapter 1 of 4: Before the lecture
Reinforcement learning for language models starts from a single-step bandit in which the prompt is the state and the whole completion is the action. Levine reviews masked self-attention in transformer language models, then derives the policy gradient estimator for this bandit. Because sampling completions is expensive, importance-weighted estimators such as PPO take several optimization steps per batch. Reward models are fit to pairwise human preferences, with preference probability an exponential ratio of rewards, as in maximum entropy inverse RL. A KL penalty against the supervised policy, used by Ziegler et al. and Ouyang et al., limits overoptimization of the learned reward.
Your comment is kept while you sign up.Sign in
Levine casts language generation as a one-step bandit and curbs reward overoptimization with a KL penalty.
Reinforcement learning for language models starts from a single-step bandit in which the prompt is the state and the whole completion is the action. Levine reviews masked self-attention in transformer language models, then derives the policy gradient estimator for this bandit. Because sampling completions is expensive, importance-weighted estimators such as PPO take several optimization steps per batch. Reward models are fit to pairwise human preferences, with preference probability an exponential ratio of rewards, as in maximum entropy inverse RL. A KL penalty against the supervised policy, used by Ziegler et al. and Ouyang et al., limits overoptimization of the learned reward.
Fine-Tuning Language Models from Human Preferences
Training language models to follow instructions with human feedback
This lecture
Lecture 66 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.