Opening Exploration of fundamental concepts in deep reinforcement learning
Exploration of fundamental concepts in deep reinforcement learning
- Length
- 13 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Intuition for policy gradients
Policy gradients in deep reinforcement learning can be read as a weighted maximum likelihood objective that raises the log probability of high-reward trajectories. Levine builds this intuition in session 9 of UC Berkeley CS 285, after the derivation of the policy gradient. He extends the estimator from discrete actions to continuous ones, using a multivariate normal policy whose mean a neural network outputs. Because the derivation never invokes the Markov property, the same estimator applies unchanged to partially observed systems. The last part shows how finite sample sizes give high variance, which makes the vanilla method hard to train. Familiarity with maximum likelihood, basic probability and backpropagation is assumed.

Your comment is kept while you sign up.Sign in
Reads the policy gradient as weighted maximum likelihood, then shows why finite samples make it noisy.
Policy gradients in deep reinforcement learning can be read as a weighted maximum likelihood objective that raises the log probability of high-reward trajectories. Levine builds this intuition in session 9 of UC Berkeley CS 285, after the derivation of the policy gradient. He extends the estimator from discrete actions to continuous ones, using a multivariate normal policy whose mean a neural network outputs. Because the derivation never invokes the Markov property, the same estimator applies unchanged to partially observed systems. The last part shows how finite sample sizes give high variance, which makes the vanilla method hard to train. Familiarity with maximum likelihood, basic probability and backpropagation is assumed.
This lecture
Lecture 9 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.