Opening Exploration of deep reinforcement learning architectures and algorithms
Exploration of deep reinforcement learning architectures and algorithms
- Length
- 14 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Numerical issues in policy gradients
Policy gradients in continuous action spaces suffer from poor conditioning, because different parameters change the policy at very different rates. Levine shows the effect in a Gaussian policy, where the variance shrinks faster than the mean and optimization stalls. He then replaces the step in parameter space with a constraint on the Kullback–Leibler divergence between old and new policies. A second-order Taylor expansion turns that divergence into the Fisher information matrix, and multiplying the gradient by its inverse yields the natural gradient. The session, from CS 285 at UC Berkeley, assumes REINFORCE and matrix calculus, and closes with trust region and proximal policy optimization papers.

Your comment is kept while you sign up.Sign in
Levine derives the natural policy gradient by bounding KL divergence, with the Fisher matrix as preconditioner.
Policy gradients in continuous action spaces suffer from poor conditioning, because different parameters change the policy at very different rates. Levine shows the effect in a Gaussian policy, where the variance shrinks faster than the mean and optimization stalls. He then replaces the step in parameter space with a constraint on the Kullback–Leibler divergence between old and new policies. A second-order Taylor expansion turns that divergence into the Fisher information matrix, and multiplying the gradient by its inverse yields the natural gradient. The session, from CS 285 at UC Berkeley, assumes REINFORCE and matrix calculus, and closes with trust region and proximal policy optimization papers.
Natural Actor-Critic
Reinforcement learning of motor skills with policy gradients
Trust Region Policy Optimization
Guided policy search
Benchmarking Deep Reinforcement Learning for Continuous Control
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Infinite-horizon policy-gradient estimation
Proximal Policy Optimization Algorithms
This lecture
Lecture 12 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.