Opening Control as Inference, Part 2
Control as Inference, Part 2
- Length
- 29 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 4: Derivation of backward messages
Control as inference treats optimality as a binary observed variable in a graphical model of states and actions, and reframes optimal control as probabilistic inference. In this session of UC Berkeley's CS 285, Levine derives the backward messages recursively from the end of the trajectory. In log space these give a soft value function, , which generalizes value iteration. He shows that action priors fold into the reward, and recovers the policy as , the exponential of a soft advantage. Forward messages then give state marginals proportional to the product of forward and backward messages, compared with Li and Todorov's 2006 data on human reaching. Prior work with Markov decision processes, dynamic programming and message passing is assumed.

Your comment is kept while you sign up.Sign in
Derives a soft Bellman backup from backward messages and recovers the policy as an exponentiated soft advantage.
Control as inference treats optimality as a binary observed variable in a graphical model of states and actions, and reframes optimal control as probabilistic inference. In this session of UC Berkeley's CS 285, Levine derives the backward messages recursively from the end of the trajectory. In log space these give a soft value function, V(sₜ) = log ∫ exp(Q(sₜ, aₜ)) daₜ, which generalizes value iteration. He shows that action priors fold into the reward, and recovers the policy as βₜ(sₜ, aₜ)/βₜ(sₜ), the exponential of a soft advantage. Forward messages then give state marginals proportional to the product of forward and backward messages, compared with Li and Todorov's 2006 data on human reaching. Prior work with Markov decision processes, dynamic programming and message passing is assumed.
Human motor control study
This lecture
Lecture 57 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.