Opening RL with Sequence Models & Language Models, Part 1 (CS 285, Lecture 21)
RL with Sequence Models & Language Models, Part 1 (CS 285, Lecture 21)
- Length
- 30 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2023
Chapter 1 of 4: Partial observability and MDPs
Levine shows why Q-learning and model-based RL break under non-Markovian observations, and how history states fix them.
Partially observed Markov decision processes arise when an agent receives observations that lack the Markov property of true states. Levine opens Part 1 of the CS 285 sequence-models unit with behaviours absent from fully observed MDPs, including information-gathering actions and stochastic optimal policies. Policy gradients remain valid under partial observability, though advantage estimators need care because value functions depend on past observations. Q-learning and model-based methods fail, since values and transitions depend on history. Treating the observation history as the state restores the Markov property, so RNNs, LSTMs and transformers serve as function approximators. Levine cites Recurrent Experience Replay in Distributed RL for storing hidden states in the replay buffer.
Partially observed Markov decision processes arise when an agent receives observations that lack the Markov property of true states. Levine opens Part 1 of the CS 285 sequence-models unit with behaviours absent from fully observed MDPs, including information-gathering actions and stochastic optimal policies. Policy gradients remain valid under partial observability, though advantage estimators need care because value functions depend on past observations. Q-learning and model-based methods fail, since values and transitions depend on history. Treating the observation history as the state restores the Markov property, so RNNs, LSTMs and transformers serve as function approximators. Levine cites Recurrent Experience Replay in Distributed RL for storing hidden states in the replay buffer.
Recurrent Experience Replay in Distributed Reinforcement Learning

This lecture
Lecture 65 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.
Your comment is kept while you sign up.Sign in