Opening Foundational concepts in deep reinforcement learning
Foundational concepts in deep reinforcement learning
- Length
- 16 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Transition from actor-critic to value-based methods
Value function methods in reinforcement learning begin by extending actor-critic algorithms to estimate advantages and update policies without explicit policy networks. The lecture demonstrates how to construct implicit policies using the argmax of the advantage function, which guarantees monotonic improvement. For small discrete state spaces, tabular reinforcement learning applies dynamic programming to compute exact value functions through iterative recursion and Bellman backups. Finally, the derivation of value iteration shows how taking the maximum over actions short-circuits policy iteration to update Q-values directly.

Your comment is kept while you sign up.Sign in
Derives policy and value iteration from actor-critic advantages in tabular reinforcement learning settings.
Value function methods in reinforcement learning begin by extending actor-critic algorithms to estimate advantages and update policies without explicit policy networks. The lecture demonstrates how to construct implicit policies using the argmax of the advantage function, which guarantees monotonic improvement. For small discrete state spaces, tabular reinforcement learning applies dynamic programming to compute exact value functions through iterative recursion and Bellman backups. Finally, the derivation of value iteration shows how taking the maximum over actions short-circuits policy iteration to update Q-values directly.
This lecture
Lecture 17 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.