Opening Exploration of advanced topics in deep reinforcement learning
Exploration of advanced topics in deep reinforcement learning
- Length
- 21 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Natural gradient
The natural policy gradient replaces the Euclidean distance constraint of standard gradient ascent with a second-order approximation of KL divergence between policies. In session 27 of UC Berkeley's CS 285, Levine shows why plain gradient ascent gives ill-conditioned updates, such as over-adjusting the variance parameters of a Gaussian policy. A Taylor expansion of the KL constraint yields the Fisher Information Matrix and an ellipse-shaped trust region. Multiplying the policy gradient by the inverse Fisher matrix gives the natural gradient update. The last part covers estimating the Fisher matrix from samples, conjugate gradient methods, and the links to TRPO and PPO, assuming familiarity with REINFORCE.

Your comment is kept while you sign up.Sign in
Levine replaces the Euclidean step constraint with a Fisher-matrix ellipse to derive the natural policy gradient.
The natural policy gradient replaces the Euclidean distance constraint of standard gradient ascent with a second-order approximation of KL divergence between policies. In session 27 of UC Berkeley's CS 285, Levine shows why plain gradient ascent gives ill-conditioned updates, such as over-adjusting the variance parameters of a Gaussian policy. A Taylor expansion of the KL constraint yields the Fisher Information Matrix and an ellipse-shaped trust region. Multiplying the policy gradient by the inverse Fisher matrix gives the natural gradient update. The last part covers estimating the Fisher matrix from samples, conjugate gradient methods, and the links to TRPO and PPO, assuming familiarity with REINFORCE.
Reinforcement learning of motor skills with policy gradients
Trust region policy optimization
This lecture
Lecture 27 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.