Opening Control as Inference, Part 3
Control as Inference, Part 3
- Length
- 21 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Approximate inference and the optimism problem
Control as probabilistic inference, treated exactly, backs up a log of expected exponentiated value, which overweights rare high-reward outcomes and makes the resulting policy optimistic. Levine resolves this optimism problem with a structured variational distribution that keeps the original transition dynamics fixed and learns only the action probabilities. The variational lower bound then decomposes into expected reward plus an entropy term on actions. Dynamic programming on that objective yields soft value iteration, where a log-sum-exp soft maximum replaces the hard maximum in the Bellman backup. The session closes with discounting, a temperature parameter that recovers hard-max value iteration in the zero limit, and infinite-horizon extensions.
Your comment is kept while you sign up.Sign in
Levine fixes the dynamics in a variational posterior to cure optimism bias, then derives soft value iteration.
Control as probabilistic inference, treated exactly, backs up a log of expected exponentiated value, which overweights rare high-reward outcomes and makes the resulting policy optimistic. Levine resolves this optimism problem with a structured variational distribution that keeps the original transition dynamics fixed and learns only the action probabilities. The variational lower bound then decomposes into expected reward plus an entropy term on actions. Dynamic programming on that objective yields soft value iteration, where a log-sum-exp soft maximum replaces the hard maximum in the Bellman backup. The session closes with discounting, a temperature parameter that recovers hard-max value iteration in the zero limit, and infinite-horizon extensions.
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
This lecture
Lecture 58 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.