Opening Exploration of advanced topics in deep reinforcement learning
Exploration of advanced topics in deep reinforcement learning
- Length
- 10 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2020
Chapter 1 of 2: Uncertainty in model-based reinforcement learning
Model-based reinforcement learning with deep neural network dynamics models often trails model-free methods early in training, when data is scarce. Levine traces the cause to overfitting and distributional shift: the planner exploits spurious peaks in the learned model and produces flawed trajectories. Evidence comes from the ICRA 2018 work by Nagabandi, Kahn and Fearing on the half cheetah simulation task. The remedy is uncertainty estimation, which predicts a distribution over next states so the planner evaluates expected reward across plausible dynamics. A cliff-edge example shows uncertain regions penalized automatically, with confidence growing as more data arrives. The ten-minute CS 285 segment assumes MDPs, neural network overfitting and the version 1.5 model-based architecture.

Your comment is kept while you sign up.Sign in
Shows why planners exploit overfitted dynamics models, and how predicting a distribution of next states curbs it.
Model-based reinforcement learning with deep neural network dynamics models often trails model-free methods early in training, when data is scarce. Levine traces the cause to overfitting and distributional shift: the planner exploits spurious peaks in the learned model and produces flawed trajectories. Evidence comes from the ICRA 2018 work by Nagabandi, Kahn and Fearing on the half cheetah simulation task. The remedy is uncertainty estimation, which predicts a distribution over next states so the planner evaluates expected reward across plausible dynamics. A cliff-edge example shows uncertain regions penalized automatically, with confidence growing as more data arrives. The ten-minute CS 285 segment assumes MDPs, neural network overfitting and the version 1.5 model-based architecture.
ICRA 2018 paper by Nagabandi, Kahn, and Fearing
This lecture
Lecture 33 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.