Opening Exploration of advanced topics in deep reinforcement learning
Exploration of advanced topics in deep reinforcement learning
- Length
- 14 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: State coverage and intrinsic motivation
State marginal matching in deep reinforcement learning minimizes the KL divergence between a policy's state marginal distribution and a target distribution over states. Levine starts from intrinsic motivation with density estimators and shows why novelty bonuses alone make policies chase their tails across the state space. Treating the density estimator and the policy as two players turns the problem into a game, and self-play recovers its Nash equilibrium. Returning a historical mixture of policy iterates, not the final one, yields uniform coverage, drawing on work by Lee, Eysenbach and colleagues and by Hazan and colleagues. The session, part of UC Berkeley's CS 285, assumes policy gradients, KL divergence and basic game theory.
Your comment is kept while you sign up.Sign in
Levine casts state marginal matching as a two-player game and averages past policies to reach uniform state coverage.
State marginal matching in deep reinforcement learning minimizes the KL divergence between a policy's state marginal distribution and a target distribution over states. Levine starts from intrinsic motivation with density estimators and shows why novelty bonuses alone make policies chase their tails across the state space. Treating the density estimator and the policy as two players turns the problem into a game, and self-play recovers its Nash equilibrium. Returning a historical mixture of policy iterates, not the final one, yields uniform coverage, drawing on work by Lee, Eysenbach and colleagues and by Hazan and colleagues. The session, part of UC Berkeley's CS 285, assumes policy gradients, KL divergence and basic game theory.
Efficient Exploration via State Marginal Matching
Provably Efficient Maximum Entropy Exploration
Unsupervised Meta-Learning for Reinforcement Learning
This lecture
Lecture 44 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.