Opening Model-Based Reinforcement Learning with Policies
Model-Based Reinforcement Learning with Policies
- Length
- 15 min
- Level
- Graduate
- Recorded
- UC Berkeley2022
Chapter 1 of 3: Limitations of open-loop control
Model-based reinforcement learning with closed-loop policies replaces the open-loop planning of model predictive control with a neural network that maps states to actions. Levine starts from the math test problem, where committing to a fixed action sequence ignores what the agent will observe later. He then builds the computation graph that chains learned dynamics and policy across time and backpropagates rewards through it. The gradients turn out ill-conditioned, with high curvature and vanishing or exploding products like those in recurrent neural networks. Because the model must match the true dynamics, it cannot be reshaped for easier differentiation, so learned models instead generate synthetic data for model-free algorithms.

Your comment is kept while you sign up.Sign in
Levine shows why backpropagating through learned dynamics into a policy fails, as in recurrent networks.
Model-based reinforcement learning with closed-loop policies replaces the open-loop planning of model predictive control with a neural network that maps states to actions. Levine starts from the math test problem, where committing to a fixed action sequence ignores what the agent will observe later. He then builds the computation graph that chains learned dynamics and policy across time and backpropagates rewards through it. The gradients turn out ill-conditioned, with high curvature and vanishing or exploding products like those in recurrent neural networks. Because the model must match the true dynamics, it cannot be reshaped for easier differentiation, so learned models instead generate synthetic data for model-free algorithms.
This lecture
Lecture 36 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.