Opening Offline Reinforcement Learning
Offline Reinforcement Learning
- Length
- 38 min
- Level
- Graduate
- Recorded
- UC Berkeley2021
Chapter 1 of 5: Motivation for offline reinforcement learning
Levine traces Q-value overestimation on static data to counterfactual queries on actions the dataset never covers.
Offline reinforcement learning trains policies from previously collected datasets, with no further interaction with the environment. Levine starts from the cost of active data collection and formalizes the problem, separating off-policy evaluation from learning the best policy. He then argues that offline methods can stitch together good behaviours in the data to reach behaviours better than any single trajectory. The second half explains why naive off-policy Q-learning and actor-critic methods fail on static data. Robotic grasping results from the QT-Opt work and the Half-Cheetah benchmark show severe Q-value overestimation, which Levine traces to counterfactual queries on actions outside the behaviour policy's support. The material follows his 2020 tutorial with Kumar, Tucker and Fu, and assumes Bellman equations, Q-learning and actor-critic algorithms.
Offline reinforcement learning trains policies from previously collected datasets, with no further interaction with the environment. Levine starts from the cost of active data collection and formalizes the problem, separating off-policy evaluation from learning the best policy. He then argues that offline methods can stitch together good behaviours in the data to reach behaviours better than any single trajectory. The second half explains why naive off-policy Q-learning and actor-critic methods fail on static data. Robotic grasping results from the QT-Opt work and the Half-Cheetah benchmark show severe Q-value overestimation, which Levine traces to counterfactual queries on actions outside the behaviour policy's support. The material follows his 2020 tutorial with Kumar, Tucker and Fu, and assumes Bellman equations, Q-learning and actor-critic algorithms.
Playing Atari with Deep Reinforcement Learning
Trust Region Policy Optimization / Proximal Policy Optimization
This lecture
Lecture 45 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in
End-to-End Training of Deep Visuomotor Policies
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Should I Run Offline Reinforcement Learning or Behavioral Cloning?
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning
Connecting New Skills to Past Experience with Offline Reinforcement Learning
QT-Opt: Scalable Deep Reinforcement Learning of Vision-Based Robotic Manipulation Skills
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction