Opening Exploration of deep reinforcement learning, likely
Exploration of deep reinforcement learning, likely
- Length
- 19 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: State distribution mismatch
Policy optimization with an importance-weighted advantage objective relies on a bound linking the new policy's expected return to the old policy's. In this session of UC Berkeley CS 285, Levine starts from the state distribution mismatch that arises when old-policy samples stand in for the new policy. A deterministic case, where the new policy deviates with bounded probability, gives a linear bound on state-marginal divergence in the time step. A joint distribution lemma extends the argument to stochastic policies through total variation divergence. Bounding expectations of functions under the shifted distribution yields a worst-case error term scaling with horizon and maximum reward, the basis for Trust Region Policy Optimization.

Your comment is kept while you sign up.Sign in
Bounds the state-marginal mismatch between two policies, deterministic first, then via total variation.
Policy optimization with an importance-weighted advantage objective relies on a bound linking the new policy's expected return to the old policy's. In this session of UC Berkeley CS 285, Levine starts from the state distribution mismatch that arises when old-policy samples stand in for the new policy. A deterministic case, where the new policy deviates with bounded probability, gives a linear bound on state-marginal divergence in the time step. A joint distribution lemma extends the argument to stochastic policies through total variation divergence. Bounding expectations of functions under the shifted distribution yields a worst-case error term scaling with horizon and maximum reward, the basis for Trust Region Policy Optimization.
Trust Region Policy Optimization
This lecture
Lecture 26 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.