Opening Model-Based Reinforcement Learning with Policies
Model-Based Reinforcement Learning with Policies
- Length
- 15 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2022
Chapter 1 of 3: Likelihood ratio and path-wise gradients
Model-based reinforcement learning with policies contrasts policy gradient estimators against path-wise backpropagation gradients through learned dynamics models. The derivation shows how path-wise gradients multiply Jacobian matrices over time horizons, causing exploding or vanishing eigenvalues that destabilize training. To mitigate these numerical instabilities and the compounding errors of long model rollouts, the lecture introduces algorithms using policy gradients with learned models. It then builds toward short-rollout model-based reinforcement learning strategies that mix real-world trajectory states with short synthetic rollouts.

Your comment is kept while you sign up.Sign in
Levine shows backpropagating through learned dynamics multiplies Jacobians, and switches to policy gradients.
Model-based reinforcement learning with policies contrasts policy gradient estimators against path-wise backpropagation gradients through learned dynamics models. The derivation shows how path-wise gradients multiply Jacobian matrices over time horizons, causing exploding or vanishing eigenvalues that destabilize training. To mitigate these numerical instabilities and the compounding errors of long model rollouts, the lecture introduces algorithms using policy gradients with learned models. It then builds toward short-rollout model-based reinforcement learning strategies that mix real-world trajectory states with short synthetic rollouts.
PIPP: Flexible Model-Based Policy Search Robust to the Curse of Chaos
This lecture
Lecture 37 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.