Opening Inverse Reinforcement Learning, Part 2
Inverse Reinforcement Learning, Part 2
- Length
- 12 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Maximum likelihood inverse reinforcement learning
Inverse reinforcement learning builds reward functions from observed expert trajectories by maximizing their likelihood under a softmax policy. The primary computational challenge involves the partition function, whose gradient resolves into the difference between expert feature expectations and state-action visitation frequencies. Computing these frequencies requires forward and backward messages over a graphical model with known transition dynamics. Brian Ziebart's maximum entropy algorithm implements this by iteratively computing messages, forming state-action marginals, and taking gradient ascent steps on the reward parameters. This approach avoids arbitrary assumptions about sub-optimal behavior by maximizing policy entropy subject to empirical feature matching constraints.

Your comment is kept while you sign up.Sign in
Levine reduces the partition-function gradient to expert minus model feature expectations for Ziebart's algorithm.
Inverse reinforcement learning builds reward functions from observed expert trajectories by maximizing their likelihood under a softmax policy. The primary computational challenge involves the partition function, whose gradient resolves into the difference between expert feature expectations and state-action visitation frequencies. Computing these frequencies requires forward and backward messages over a graphical model with known transition dynamics. Brian Ziebart's maximum entropy algorithm implements this by iteratively computing messages, forming state-action marginals, and taking gradient ascent steps on the reward parameters. This approach avoids arbitrary assumptions about sub-optimal behavior by maximizing policy entropy subject to empirical feature matching constraints.
Maximum Entropy Inverse Reinforcement Learning
This lecture
Lecture 61 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.