Opening Offline Reinforcement Learning
Offline Reinforcement Learning
- Length
- 26 min
- Level
- Graduate
- Recorded
- UC Berkeley2021
Chapter 1 of 4: Before the lecture
Offline reinforcement learning via importance sampling derives policy gradient estimators and value evaluations from static datasets without environment interaction. The lecture evaluates naive importance sampling weights and exposes their exponential variance growth over long time horizons. It then constructs a recursive formulation to introduce the doubly robust estimator, which combines importance-weighted rewards with value function approximations to reduce variance. Finally, it outlines marginalized importance sampling, using state-action stationary distribution ratios and consistency conditions like GenDICE to estimate policies without trajectory-product weights.

Your comment is kept while you sign up.Sign in
Shows why trajectory importance weights explode with horizon, then builds the doubly robust estimator to curb them.
Offline reinforcement learning via importance sampling derives policy gradient estimators and value evaluations from static datasets without environment interaction. The lecture evaluates naive importance sampling weights and exposes their exponential variance growth over long time horizons. It then constructs a recursive formulation to introduce the doubly robust estimator, which combines importance-weighted rewards with value function approximations to reduce variance. Finally, it outlines marginalized importance sampling, using state-action stationary distribution ratios and consistency conditions like GenDICE to estimate policies without trajectory-product weights.
Doubly robust off-policy value evaluation for reinforcement learning
GenDICE
Eligibility traces for off-policy policy evaluation
Learning from scarce experience
Data-efficient off-policy policy evaluation for reinforcement learning
High-confidence off-policy evaluation
Consistent on-line off-policy evaluation
Off-policy policy gradient with state distribution correction
This lecture
Lecture 46 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.