Opening Inverse Reinforcement Learning, Part 1
Inverse Reinforcement Learning, Part 1
- Length
- 24 min
- Level
- Graduate
- Recorded
- UC Berkeley2020
Chapter 1 of 3: Introduction to inverse reinforcement learning
Inverse reinforcement learning recovers an unknown reward function from demonstrations of expert behavior, rather than assuming the reward is given. Levine motivates the problem by contrasting imitation learning, which copies actions, with human imitation, which copies intent. He frames optimal behavior as maximizing a utility function, then shows that a single pattern of behavior is explained by infinitely many valid rewards. The classical answers, feature matching from Abbeel and Ng and maximum margin planning from Ratliff et al., recover linear rewards but falter on suboptimal experts and neural network rewards. The session closes by setting up a probabilistic graphical model that learns reward parameters by maximizing trajectory likelihood. Familiarity with Markov decision processes, optimal control and basic inference is assumed.
Your comment is kept while you sign up.Sign in
Levine shows why demonstrations fit infinitely many rewards, then tests feature matching and maximum margin fixes.
Inverse reinforcement learning recovers an unknown reward function from demonstrations of expert behavior, rather than assuming the reward is given. Levine motivates the problem by contrasting imitation learning, which copies actions, with human imitation, which copies intent. He frames optimal behavior as maximizing a utility function, then shows that a single pattern of behavior is explained by infinitely many valid rewards. The classical answers, feature matching from Abbeel and Ng and maximum margin planning from Ratliff et al., recover linear rewards but falter on suboptimal experts and neural network rewards. The session closes by setting up a probabilistic graphical model that learns reward parameters by maximizing trajectory likelihood. Familiarity with Markov decision processes, optimal control and basic inference is assumed.
Apprenticeship Learning via Inverse Reinforcement Learning
Maximum Margin Planning
This lecture
Lecture 60 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.