Opening Exploration of fundamental concepts in deep reinforcement learning
Exploration of fundamental concepts in deep reinforcement learning
- Length
- 15 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2020
Chapter 1 of 2: Tabular representation and function approximation
Fitted value iteration and fitted Q-iteration carry value-based reinforcement learning from lookup tables to neural networks. Levine, in session 18 of UC Berkeley's CS 285, starts from the curse of dimensionality, which makes tabular representations fail in large or continuous state spaces. He then derives fitted value iteration as least squares regression of a network onto Bellman backup targets. Because that method needs the transition dynamics to take expectations and try several actions from one state, he moves to the Q-function recurrence. The result is fitted Q-iteration, built from sampled tuples , with data collection, target computation and regression steps and no separate policy network.

Your comment is kept while you sign up.Sign in
Levine derives fitted Q-iteration from fitted value iteration, trading known dynamics for sampled transitions.
Fitted value iteration and fitted Q-iteration carry value-based reinforcement learning from lookup tables to neural networks. Levine, in session 18 of UC Berkeley's CS 285, starts from the curse of dimensionality, which makes tabular representations fail in large or continuous state spaces. He then derives fitted value iteration as least squares regression of a network onto Bellman backup targets. Because that method needs the transition dynamics to take expectations and try several actions from one state, he moves to the Q-function recurrence. The result is fitted Q-iteration, built from sampled tuples (s, a, s', r), with data collection, target computation and regression steps and no separate policy network.
This lecture
Lecture 18 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.