Pieter Abbeel on Deep Reinforcement Learning, Robotics, and the Path to General AI

The conversation with Pieter Abbeel, a leading researcher in robotics and AI, delves into the current state and future challenges of creating intelligent robots. Abbeel emphasizes the dual nature of AI problems, often requiring both hardware and software advancements, as exemplified by the challenge of a robot beating Roger Federer at tennis. He highlights the impressive physical capabilities of robots like Boston Dynamics' creations, even if their underlying intelligence is limited or hard-coded. A significant portion of the discussion revolves around the psychological impact of interacting with robots, noting the human tendency to anthropomorphize them, even when aware of their mechanical nature.

Abbeel explains the mechanics of reinforcement learning (RL), particularly deep reinforcement learning (DRL), and addresses the challenge of sparse and delayed rewards. He offers an intuition for why DRL works despite these challenges, likening neural networks with rectified linear units to piecewise linear feedback control, which leverages the effectiveness of linear control while allowing for more complex, gradual tiling of the control space. A critical distinction is made between current RL successes and the "real world" where time scales and the abstraction gap between low-level actions (muscle contractions) and high-level decisions (doing a PhD) pose immense credit assignment problems, necessitating hierarchical reasoning.

The discussion explores pathways to achieving hierarchical reasoning, including bolting deep learning onto traditional symbolic reasoning systems or using information-theoretic approaches, though the latter has proven difficult. A more promising direction, according to Abbeel, is meta-learning, where the system learns to reinforcement learn, optimizing for faster learning and allowing hierarchical concepts to emerge, as demonstrated in the RL-squared paper. The concept of transfer learning and generalization is also central, with Abbeel acknowledging impressive results in fine-tuning large models (like AlexNet or large language models) for new tasks, but also distinguishing this from a deeper, physics-like generalization that seeks the simplest underlying explanations.

The conversation touches upon the philosophical pursuit of a "general theory of learning," drawing parallels to the modularity observed in the human brain (e.g., neocortex plasticity). Abbeel expresses a preference for mathematical formalization to "leapfrog" experimentation, though he acknowledges that empirical trial-and-error currently drives much of the progress in AI. Finally, he contrasts imitation learning with self-play, advocating for self-play as a powerful mechanism for faster learning due to its inherent ability to generate rich reward signals through contrasting successes and failures, especially in game-like scenarios. He also highlights the breakthrough of third-person learning, where robots learn from observing human demonstrations, akin to "machine translation for demonstrations."

for a lot of the challenges we think about in AI the software is really the missing piece but for something like this the hardware is nowhere near either

it's very hard with bread here we'll give him a name Berkeley robot for the elimination of tedious tasks is very hard to not think of the robot as a person

if you run something like reinforce some Linux about optimizing some objective and there's no reason that D object couldn't be tied into how much there's a person like interacting with this system

much easier is for a person to say okay what you did the last five minutes was much nicer than we did the previous five minutes and that now gives a comparison

if you think about rectified linear units or kind of break the fire type neural networks what do you get you get something that's piecewise linear feedback control

the decision of doing a PhD is like so abstract relative to what you're actually doing in the world and I think that's where credit assignment becomes just completely beyond what any current RL algorithm can do and we need hierarchical reasoning at a level that is just not available at all yet

what we're trying to achieve is non ously hierarchy per se but you could think about what does hierarchy give us what it's we hope it would give us is better credit assignment kind of what is better credit ominous is giving us it gives us faster learning right and so faster learning is ultimately maybe what we're after

if you train a big enough model on enough things it seems to transfer even better

if you use current deep learning techniques what we're doing to predict let's say the relative motion of our planets it would do pretty well but then now if a massive new mass enters our solar system it would prompt predict what will happen right and that's a different kind of journal is a Shahnaz a generalization that relies on the ultimate simplest simplest explanation

whenever you can turn something into self play you're in a beautiful situation where you can naturally learn much more quickly than in most other reinforced learning environments

what's even more interesting if you can now teach robot through third person learning where the robot watches you do something and doesn't experience it but just watches it and says okay well if you're showing me that that means I should be doing this