Deep Reinforcement Learning Fundamentals: From Perceptrons to Q-Learning for Motion Planning

This lecture introduces Deep Reinforcement Learning (DRL) through the lens of the DeepTraffic project, aiming to solve traffic problems using DRL. It begins by distinguishing between supervised, unsupervised, and semi-supervised learning, emphasizing that practical machine learning, including reinforcement learning, relies on some form of "Ground Truth"—whether explicit labels or time-delayed rewards. Supervised learning, often seen as memorization, involves training models on labeled datasets to generalize mappings between inputs and outputs, with neural networks being a highly successful algorithm for this.

The core computational building block of a neural network, the neuron or perceptron, is explained. A perceptron, with its binary output, can approximate a NAND gate, which is a universal logical gate capable of building any computer. The critical advancement from perceptrons to modern neural networks is the introduction of smooth, continuous activation functions (like sigmoid) instead of step functions. This smoothness enables gradual weight adjustments during training, allowing the network to learn complex, non-linear functions through processes like forward pass, backpropagation, and gradient descent, minimizing a loss function (e.g., squared error) over many iterations. Feed-forward networks, which process data without loops, are the primary focus, though recurrent neural networks with memory are briefly mentioned.

Reinforcement learning (RL) is presented as a paradigm closer to "reasoning" than supervised learning's "memorization." An RL agent interacts with an environment by executing actions, receiving observations (new states), and getting occasional, time-delayed rewards or punishments. This interaction can be modeled as a Markov Decision Process (MDP), where the agent's goal is to maximize future discounted rewards. Key components of an RL agent include a policy (how to act in a given state), a value function (how good states/actions are), and a model (representation of the environment). The lecture uses examples like Atari Breakout and a robot in a grid world to illustrate how different reward structures influence optimal policies.

Q-learning, an off-policy RL approach, is detailed as a method to approximate the optimal Q-function through experience, using the Bellman equation for updates. The process involves exploration (taking random actions, often guided by an epsilon-greedy policy) and exploitation (taking optimal greedy actions) to learn the best actions in various states. A significant challenge highlighted is the representation of the world: hand-crafting models is difficult and limited, while using raw pixel data (as in Atari) allows the neural network to learn its own representations, mirroring how humans learn from sensory input. The robustness of these learned models and the sensitivity to reward function design are identified as crucial, less-studied areas, especially when applying DRL to real-world, high-stakes scenarios like autonomous driving, where the state-action space can be astronomically large, demanding extensive simulation and computational power.

"This is what it's like to be human. This is what life is about. You don't know what's good and bad, you got to have to just live it and, every once in a while, you find out that all that stuff you did last week was pretty bad idea. That's reinforcement learning."

"The cool thing about a NAND gate is that it's a universal gate that you can build up any computer you have where you have your phone in your pocket today can be built out of just NAND gates."

"The essential step that makes the neural network work, that a circuit perceptrons doesn't, Is if the output is made smooth, it's made continuous with an activation function."

"The hope for reinforcement learning is that we can extend the knowledge we gain in a supervised way to the huge world outside where we don't have the Ground Truth of how to act, how good a certain state is, or how barristers say it is, this is a kind of brute force reasoning."

"So the goal is to select at any one moment the action that maximizes future reward. Without any knowledge of what a reward is in the greater sense of the word, all you have is an instantaneous reward or punishment, instantaneous response of the world to your actions."

"When you're in a certain state, you have no memory of what happened two states ago. Everything is operating on the instantaneous."

"The fact that it learns anything is amazing. I want you to remember that. That's one of the amazing things about Q-Learning at all and then the Deep neural network version of Q-Learning."

"When you're born, you see there's light coming into your eyes and you don't have any, as far as we know, you don't come with an instruction when you're born. You know there's people in the world then there is good guys and bad guys, and there's this is how you walk. No, all you get is light, sound and the other sensors."