Opening Offline Reinforcement Learning (CS 285, Lecture 15, Part 3)
Offline Reinforcement Learning (CS 285, Lecture 15, Part 3)
- Length
- 21 min
- Level
- Graduate
- Recorded
- UC Berkeley2021
Chapter 1 of 3: Linear fitted value functions
Offline reinforcement learning with linear function approximation rests on least squares temporal difference and least squares policy iteration. Levine, in CS 285 at UC Berkeley, starts from model-based estimators of rewards and transitions in a feature space. Solving the resulting Bellman equation with normal equations yields value-function weights in closed form, and the explicit model then drops out. An empirical Markov decision process built from batch transition tuples replaces the state-space matrices. Extending the same derivation to Q-functions gives least squares policy iteration, and the session closes by showing why greedy improvement still suffers from distributional shift.

Your comment is kept while you sign up.Sign in
Levine derives least squares temporal difference in closed form, then extends it to policy iteration.
Offline reinforcement learning with linear function approximation rests on least squares temporal difference and least squares policy iteration. Levine, in CS 285 at UC Berkeley, starts from model-based estimators of rewards and transitions in a feature space. Solving the resulting Bellman equation with normal equations yields value-function weights in closed form, and the explicit model then drops out. An empirical Markov decision process built from batch transition tuples replaces the state-space matrices. Extending the same derivation to Q-functions gives least squares policy iteration, and the session closes by showing why greedy improvement still suffers from distributional shift.
This lecture
Lecture 47 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.