Deep Reinforcement Learning: Core Methods, Applications, and Policy Gradients

This podcast episode provides a comprehensive introduction to Deep Reinforcement Learning (DRL), defining it as reinforcement learning that leverages neural networks as function approximators. The discussion outlines how neural networks can be used to approximate various components within an RL system, including the agent's policy (how it chooses actions), value functions (measuring state/action goodness), or a dynamics model (predicting next states and rewards). The speaker illustrates DRL's broad applicability with diverse examples, ranging from robotics (e.g., navigation, manipulation, locomotion) and industrial problems like inventory management to more abstract machine learning tasks such as attention mechanisms in image processing and structured prediction in machine translation.\n\nThe episode meticulously distinguishes DRL from other machine learning paradigms like supervised learning and contextual bandits. A key difference highlighted is that RL operates in stateful environments where an agent's actions influence subsequent inputs, leading to delayed effects and making stable algorithm development challenging. Furthermore, unlike supervised learning, RL typically lacks direct analytic access to the loss function, preventing straightforward gradient computation. Contextual bandits are presented as an intermediate step, sharing the lack of loss function access but without the statefulness of full RL.\n\nPractical advice is offered on when to consider DRL and, crucially, when it might be an overkill. For problems with a small parameter space and available simulators, derivative-free optimization methods are suggested as simpler alternatives. Similarly, if a problem can be approximated as a contextual bandit, specialized algorithms with stronger theoretical guarantees might be preferable. Traditional operations research methods, with their well-developed feature engineering techniques, are also presented as viable alternatives. However, the speaker then showcases DRL's unique power through significant success stories, including Deep Q-learning's mastery of Atari games, DeepMind's AlphaGo triumph, real-time robotic manipulation, and general methods for robotic locomotion.\n\nThe latter part of the discussion delves into the technical formalism, introducing the Markov Decision Process (MDP) as the foundational object in RL and defining the episodic setting. A core focus is placed on policy gradient methods, explaining their intuition: to make "good" trajectories (sequences of actions and states leading to high reward) more probable. The score function gradient estimator is introduced as a fundamental technique, notable for its ability to compute gradients even when the function being optimized is discontinuous, unknown, or operates over discrete spaces. This remarkable property allows DRL to tackle complex real-world problems, such as robotic locomotion involving discontinuous contact dynamics, where traditional smooth optimization techniques would fail.

"reinforcement learning... is a branch of machine learning cons uh concerned with taking sequences of actions."

"what's deep reinforcement learning it's pretty simple it's just uh reinforcement learning where you're using uh neural networks uh as function approximators."

"the big difference between this and the supervised learning setting is you don't have access to the function uh the Lost function trying to optimize so in particular you can't differentiate through it."

"reinforcement learning is almost the same as the contextual Bandit setting except now the environment is stateful."

"if your problem has a small number of parameters you're trying to optimize over and you have a simulator that you can like just do lots of experiments on then derivative free optimization methods are likely to be better than reinforcement learning."

"it's pretty remarkable that you can just use a simple uh that a simple algorithm can solve them all." (referring to Atari games)

"the basic object uh in uh the field of reinforcement learning is the markof decision process."

"the policy is just the function that the agent uses to choose its actions."

"the intuition of how our algorithm is going to work is we we're going to collect a bunch of trajectories... and then we want to make the good trajectories more probable."

"the really nice thing is this is valid even if f(x) is discontinuous or if f(x) is unknown meaning you only you don't get to differentiate it you just get to see the function values."