Opening RL with Sequence Models & Language Models, Part 3
RL with Sequence Models & Language Models, Part 3
- Length
- 17 min
- Level
- Graduate
- Recorded
- UC Berkeley2023
Chapter 1 of 3: Multi-turn reinforcement learning with language models
Multi-turn reinforcement learning with language models treats dialogue as a partially observable Markov decision process whose state is the conversation history and whose reward arrives at the end. Levine weighs two ways to define a time step. A whole utterance per step keeps the horizon short but makes the action space exponentially large. A single token per step gives a simple discrete action space but a much longer horizon. Because online policy gradient would need live human interaction for every rollout, the session turns to offline value-based methods, comparing actor-critic designs with Q-learning objectives that handle distribution shift. Three papers illustrate the choices: Jaques et al. with a policy constraint and sentiment reward, Verma et al.'s CHAI with CQL, and Snell et al.'s Implicit Language Q-Learning.
Your comment is kept while you sign up.Sign in
Levine sets per-utterance against per-token steps for offline value-based RL in dialogue, via Jaques, Verma and Snell.
Multi-turn reinforcement learning with language models treats dialogue as a partially observable Markov decision process whose state is the conversation history and whose reward arrives at the end. Levine weighs two ways to define a time step. A whole utterance per step keeps the horizon short but makes the action space exponentially large. A single token per step gives a simple discrete action space but a much longer horizon. Because online policy gradient would need live human interaction for every rollout, the session turns to offline value-based methods, comparing actor-critic designs with Q-learning objectives that handle distribution shift. Three papers illustrate the choices: Jaques et al. with a policy constraint and sentiment reward, Verma et al.'s CHAI with CQL, and Snell et al.'s Implicit Language Q-Learning.
Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Human-Centric Dialog Training via Offline Reinforcement Learning
CHAI: A Chatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
Offline RL for Natural Language Generation with Implicit Language Q-Learning
This lecture
Lecture 67 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.