Opening Exploration of fundamental concepts in deep reinforcement learning
Exploration of fundamental concepts in deep reinforcement learning
- Length
- 19 min
- Level
- Undergraduate
- Recorded
- UC Berkeley2021
Chapter 1 of 3: Neural network architectures and batch sizes
Actor-critic design in deep reinforcement learning turns on choices about network architecture, batch collection and data reuse. Levine, in session 15 of UC Berkeley's CS 285, first compares separate policy and value networks against a shared trunk. The shared network is more feature-efficient, while separate networks are simpler to stabilize and train. He then contrasts synchronized parallel workers with asynchronous ones, which give up strict mathematical equivalence for speed. The second half builds off-policy actor-critic from a replay buffer, shows where the naive version breaks, and fixes it with a Q-function evaluated at current-policy actions. Soft Actor-Critic is cited as a practical descendant, and policy gradients and value functions are assumed.

Your comment is kept while you sign up.Sign in
Shows why naive off-policy actor-critic fails on replay-buffer data and repairs it with a Q-function.
Actor-critic design in deep reinforcement learning turns on choices about network architecture, batch collection and data reuse. Levine, in session 15 of UC Berkeley's CS 285, first compares separate policy and value networks against a shared trunk. The shared network is more feature-efficient, while separate networks are simpler to stabilize and train. He then contrasts synchronized parallel workers with asynchronous ones, which give up strict mathematical equivalence for speed. The second half builds off-policy actor-critic from a replay buffer, shows where the naive version breaks, and fixes it with a Q-function evaluated at current-policy actions. Soft Actor-Critic is cited as a practical descendant, and policy gradients and value functions are assumed.
Soft Actor-Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
This lecture
Lecture 15 of 73 of Deep Reinforcement Learning.
Choose a node to see how it connects to this lecture. What comes before is on the leftat the top. What comes after is on the rightat the bottom. The ring is what it teaches. People and places are above. Other lectures and books are below.
Press a chosen node again to open it.