Opening Exploration of deep reinforcement learning architectures and algorithms
Exploration of deep reinforcement learning architectures and algorithms
- Length
- 17 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Bellman operator and value iteration
Convergence of value-based deep reinforcement learning fails because the Bellman backup and neural network regression contract in different norms. Levine, in session 19 of UC Berkeley's CS 285, starts from the Bellman operator and proves that tabular value iteration converges as a contraction in the infinity norm. Fitted value iteration then appears as that backup followed by an L2 projection onto the hypothesis set. Each step contracts alone, but in mismatched norms, so their composition need not contract. The same argument carries over to fitted Q-iteration, Q-learning and actor-critic methods, whose targets depend on the parameters and so follow no true gradient.

Your comment is kept while you sign up.Sign in
Shows that an infinity-norm Bellman contraction composed with an L2 regression projection is no contraction at all.
Convergence of value-based deep reinforcement learning fails because the Bellman backup and neural network regression contract in different norms. Levine, in session 19 of UC Berkeley's CS 285, starts from the Bellman operator and proves that tabular value iteration converges as a contraction in the infinity norm. Fitted value iteration then appears as that backup followed by an L2 projection onto the hypothesis set. Each step contracts alone, but in mismatched norms, so their composition need not contract. The same argument carries over to fitted Q-iteration, Q-learning and actor-critic methods, whose targets depend on the parameters and so follow no true gradient.
This lecture
Lecture 19 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.