Opening Deep Reinforcement Learning
Deep Reinforcement Learning
- Length
- 9 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Recursive formulation of the reinforcement learning objective
Value functions and Q-functions start from the recursive expansion of the reinforcement learning objective into nested expectations over states and actions. The derivation shows how a partial trajectory return can be abstracted into a single function that evaluates the expected future reward from a given state and action. Levine connects these functions to policy improvement by demonstrating how the argmax of a Q-function yields a better policy, which forms the basis for policy iteration. Finally, the lecture relates Q-functions to state value functions to construct gradient-based update rules for policy optimization.

Your comment is kept while you sign up.Sign in
Derives the recursive expansion of expected rewards to define state-action value and policy value functions.
Value functions and Q-functions start from the recursive expansion of the reinforcement learning objective into nested expectations over states and actions. The derivation shows how a partial trajectory return can be abstracted into a single function that evaluates the expected future reward from a given state and action. Levine connects these functions to policy improvement by demonstrating how the argmax of a Q-function yields a better policy, which forms the basis for policy iteration. Finally, the lecture relates Q-functions to state value functions to construct gradient-based update rules for policy optimization.
This lecture
Lecture 7 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.