Opening Offline Reinforcement Learning II
Offline Reinforcement Learning II
- Length
- 18 min
- Level
- Graduate
- Recorded
- UC Berkeley2021
Chapter 1 of 3: Model-based offline reinforcement learning
Model-based offline reinforcement learning trains a dynamics model on a static dataset, then uses it to optimize a policy or to plan directly. Levine, in session 49 of UC Berkeley's CS 285, shows how policies exploit model errors by steering into out-of-distribution states and actions. MOPO counters this by subtracting an uncertainty penalty from the reward, and MOReL is noted as concurrent work. COMBO instead carries conservative Q-learning into the model setting, pushing down Q-values on model-generated state-action tuples. The Trajectory Transformer then treats reinforcement learning as sequence modeling over discretized trajectories, planning with behavior probabilities to stay in distribution.
Your comment is kept while you sign up.Sign in
Levine contrasts MOPO's uncertainty-penalized reward, COMBO's conservative Q-values and the Trajectory Transformer.
Model-based offline reinforcement learning trains a dynamics model on a static dataset, then uses it to optimize a policy or to plan directly. Levine, in session 49 of UC Berkeley's CS 285, shows how policies exploit model errors by steering into out-of-distribution states and actions. MOPO counters this by subtracting an uncertainty penalty from the reward, and MOReL is noted as concurrent work. COMBO instead carries conservative Q-learning into the model setting, pushing down Q-values on model-generated state-action tuples. The Trajectory Transformer then treats reinforcement learning as sequence modeling over discretized trajectories, planning with behavior probabilities to stay in distribution.
MOPO: Model-Based Offline Policy Optimization
MOReL : Model-Based Offline Reinforcement Learning
COMBO: Conservative Offline Model-Based Policy Optimization
Reinforcement Learning as One Big Sequence Modeling Problem
This lecture
Lecture 49 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.