Opening Fundamental concepts in deep reinforcement learning
Fundamental concepts in deep reinforcement learning
- Length
- 18 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Model-based reinforcement learning basics
Model-based reinforcement learning fits a dynamics model from data and plans actions through it. Levine, in session 32 of UC Berkeley's CS 285, starts from a prototype that fits the model to an initial dataset and then plans with it. He shows why this fails with high-capacity neural networks: planning drives the system into states the training data never covered, a distributional shift. Iterative data collection answers this by appending newly visited states to the dataset, in the manner of DAgger. Model predictive control goes further, executing only the first planned action, observing the result and replanning. Basic reinforcement learning definitions, supervised loss functions and planning concepts are assumed.

Your comment is kept while you sign up.Sign in
Levine builds model-based RL from a naive fit-then-plan prototype to model predictive control, which replans each step.
Model-based reinforcement learning fits a dynamics model f(sₜ, aₜ) = sₜ₊₁ from data and plans actions through it. Levine, in session 32 of UC Berkeley's CS 285, starts from a prototype that fits the model to an initial dataset and then plans with it. He shows why this fails with high-capacity neural networks: planning drives the system into states the training data never covered, a distributional shift. Iterative data collection answers this by appending newly visited states to the dataset, in the manner of DAgger. Model predictive control goes further, executing only the first planned action, observing the result and replanning. Basic reinforcement learning definitions, supervised loss functions and planning concepts are assumed.
This lecture
Lecture 32 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.