ConceptAI & ML
Bellman equation
An equation expressing the value of a state or state-action pair in terms of the immediate reward and the expected value of subsequent states.
Know first
Named as a prerequisite by the lectures that define this. The number is how many say so. A grey one has no page yet.
Dynamic programming2Expected value2Expectimax2Markov decision processes2Value function2Cost-to-go function1Markov reward processes1MDP1
Opens up
Concepts that name this one as a prerequisite.
Lecturers
6Each in their own terms, from the easiest level.
Pieter Abbeel
Equations that express optimal values or Q-values in terms of other optimal values or Q-values one time step ahead.David Silver
The Bellman equation decomposes the value function recursively into an immediate reward plus the discounted value of the successor state.Russ Tedrake
A recursive decomposition of the cost-to-go function expressing the cost from a state as the sum of a one-step cost and the optimal cost-to-go from the next state.Esther Duflo
An equation that expresses the value function of a problem as the maximum over possible savings choices of instantaneous utility plus expected future value.Fei-Fei Li, Justin Johnson & Serena Yeung
An identity that relates the optimal Q-value at a state-action pair to the immediate reward and the discounted maximum Q-value of the next state.
Lectures that assume it
18The first 12 are shown.
- Model-Free Control — ε-Greedy SARSA & Off-Policy Q-Learning
- Planning by Dynamic Programming — Policy & Value Iteration
- Deep Reinforcement Learning
- Reinforcement Learning Part 1/2
- Exploration of fundamental concepts in deep reinforcement learning
- Deep Reinforcement Learning
- Model-Free Prediction — Monte Carlo, TD(0) & TD(λ)
- Dynamic Programming II
- Acrobots, Cart-poles, and Quadrotors I
- Convex Optimization II: Model Predictive Control
- Offline Reinforcement Learning (CS 285, Lecture 15, Part 3)
- Control as Inference, Part 2
Connections
35The lectures that define it or assume it, and the concepts next to it.
Concepts10
Show 4 moreShow fewer
Lectures25
- Markov Processes, Reward Processes & Bellman Equations
- Markov Decision Processes (MDPs) Part 1/2
- RL Theory
- Dynamic Programming I
- Markov Decision Processes (MDPs) Part 2/2
- Savings
Show 19 moreShow fewer
- Deep Reinforcement Learning
- Model-Free Control — ε-Greedy SARSA & Off-Policy Q-Learning
- Planning by Dynamic Programming — Policy & Value Iteration
- Deep Reinforcement Learning
- Reinforcement Learning Part 1/2
- Exploration of fundamental concepts in deep reinforcement learning
- Deep Reinforcement Learning
- Model-Free Prediction — Monte Carlo, TD(0) & TD(λ)
- Dynamic Programming II
- Acrobots, Cart-poles, and Quadrotors I
- Convex Optimization II: Model Predictive Control
- Offline Reinforcement Learning (CS 285, Lecture 15, Part 3)
- Control as Inference, Part 2
- Policy Gradient Methods
- Dynamic Programming III
- Trajectory Optimization II
- Offline Reinforcement Learning
- Reinforcement Learning Part 2/2
- RL Theory