Opening Exploration of advanced topics in deep reinforcement learning
Exploration of advanced topics in deep reinforcement learning
- Length
- 17 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Challenges of model-based reinforcement learning with images
Model-based reinforcement learning from images must cope with high-dimensional, redundant observations and with partial observability, which Levine frames as a POMDP. His answer is a latent state-space model that factors into an observation model, a dynamics model over unobserved states, and a reward model. These are trained jointly by maximizing an expected log-likelihood under an approximate posterior. A deterministic encoder that maps each observation to a single state removes the variational inference and reduces training to joint backpropagation. The session then places these models inside model-based MPC loops, citing Embed to Control, SOLAR, and visual foresight work on robotic manipulation and LQR planning.

Your comment is kept while you sign up.Sign in
Levine factors image-based world models into observation, dynamics and reward models trained in a learned latent space.
Model-based reinforcement learning from images must cope with high-dimensional, redundant observations and with partial observability, which Levine frames as a POMDP. His answer is a latent state-space model that factors into an observation model, a dynamics model over unobserved states, and a reward model. These are trained jointly by maximizing an expected log-likelihood under an approximate posterior. A deterministic encoder that maps each observation to a single state removes the variational inference and reduces training to joint backpropagation. The session then places these models inside model-based MPC loops, citing Embed to Control, SOLAR, and visual foresight work on robotic manipulation and LQR planning.
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning
Deep Visual Foresight for Planning Robot Motion
Self-Supervised Visual Planning with Temporal Skip Connections
This lecture
Lecture 35 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.