Opening Deep Reinforcement Learning
Deep Reinforcement Learning
- Length
- 21 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 4: Recap of policy gradient methods
Advanced policy gradient methods rest on a link between policy gradients and policy iteration, which Levine derives in session 25 of UC Berkeley's CS 285. He starts from REINFORCE and its reward-to-go estimator, then recasts the policy gradient as a softened policy iteration that shifts the policy gradually along advantage estimates. A telescoping sum over the value function writes the difference between two policies' objectives as the old policy's expected advantage under the new policy's trajectories. Importance sampling and an approximation of the new policy's state distribution then recover the standard policy gradient expression. The derivation assumes REINFORCE, reward-to-go estimators and policy iteration from earlier sessions.

Your comment is kept while you sign up.Sign in
Levine telescopes the value function to recast policy gradients as a softened form of policy iteration.
Advanced policy gradient methods rest on a link between policy gradients and policy iteration, which Levine derives in session 25 of UC Berkeley's CS 285. He starts from REINFORCE and its reward-to-go estimator, then recasts the policy gradient as a softened policy iteration that shifts the policy gradually along advantage estimates. A telescoping sum over the value function writes the difference between two policies' objectives as the old policy's expected advantage under the new policy's trajectories. Importance sampling and an approximation of the new policy's state distribution then recover the standard policy gradient expression. The derivation assumes REINFORCE, reward-to-go estimators and policy iteration from earlier sessions.
This lecture
Lecture 25 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.