Opening Offline Reinforcement Learning II
Offline Reinforcement Learning II
- Length
- 32 min
- Level
- Graduate
- Recorded
- UC Berkeley2021
Chapter 1 of 5: Offline reinforcement learning and distributional shift
Levine derives advantage-weighted actor-critic from a Lagrangian, then implicit Q-learning from expectile regression.
Offline reinforcement learning trains policies from a fixed dataset, and standard Q-learning breaks there because policy updates exploit Q-function errors on out-of-distribution actions. Levine begins from this distributional shift and surveys policy constraint methods that keep the learned policy near the behavior policy. Explicit penalties built on KL divergence or MMD, added to the actor objective or the reward, often prove too restrictive in practice. Solving the constrained problem through its Lagrangian instead gives advantage-weighted actor-critic, a weighted behavioral cloning objective with no explicit behavior model. The CS 285 session at UC Berkeley closes by deriving implicit Q-learning, which fits values only on dataset state-action pairs using expectile regression.
Offline reinforcement learning trains policies from a fixed dataset, and standard Q-learning breaks there because policy updates exploit Q-function errors on out-of-distribution actions. Levine begins from this distributional shift and surveys policy constraint methods that keep the learned policy near the behavior policy. Explicit penalties built on KL divergence or MMD, added to the actor objective or the reward, often prove too restrictive in practice. Solving the constrained problem through its Lagrangian instead gives advantage-weighted actor-critic, a weighted behavioral cloning objective with no explicit behavior model. The CS 285 session at UC Berkeley closes by deriving implicit Q-learning, which fits values only on dataset state-action pairs using expectile regression.
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
This lecture
Lecture 48 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
Taming the Noise in Reinforcement Learning via Flow Regularization
Off-Policy Deep Reinforcement Learning without Exploration
Way Off-Policy Batch Deep Reinforcement Learning of Human Preferences in Dialog
Behavior Regularized Offline Reinforcement Learning
Advantage-Weighted Regression
Accelerating Online Reinforcement Learning with Offline Datasets
REPS
psi-learning
Offline Reinforcement Learning with Implicit Q-Learning