David Silver on AlphaGo, AlphaZero, and the Reinforcement Learning Path to Artificial Intelligence

This episode features David Silver, lead researcher on AlphaGo, AlphaZero, AlphaStar, and MuZero at DeepMind, discussing his journey and the profound impact of deep reinforcement learning on artificial intelligence. Silver recounts his early fascination with computers, starting with a BBC Model B at age seven, and his father's pursuit of AI, which exposed him to programming languages like Prolog. This early exposure fostered a belief in the limitless possibilities of computers and the ultimate goal of recreating human intelligence as the most significant leap in computer science. He distinguishes between "handcrafted AI" in early game development and the principled, self-learning systems he later pursued.

Silver highlights the game of Go as the ultimate challenge for AI, contrasting its immense complexity and reliance on human intuition with games like chess, which were conquered by brute-force search methods. He explains that Go's astronomical search space and the difficulty of evaluating board positions intuitively made it seem "unbeatable" for traditional AI, leading to a consensus that a fundamentally different approach was required. This realization, that cracking Go would necessitate something akin to human intuition, transformed the problem from a mere game challenge into a quest for a deeper understanding of intelligence itself.

His personal breakthrough came with the discovery of reinforcement learning (RL), particularly through Richard Sutton's seminal textbook. Silver posits that the problem of intelligence can be formally defined as the reinforcement learning problem, where an agent learns to maximize a reward signal by interacting with an environment. He emphasizes that true progress beyond human levels of performance requires systems that learn from first principles, rather than relying on pre-programmed human knowledge, which often leads to brittle systems due to the "knowledge acquisition bottleneck."

The conversation delves into the philosophical implications of RL as a core component of intelligence, suggesting that any future human-level AI would likely be a solution to the RL problem. Silver acknowledges that while current RL methods are powerful, the field is still in its early days, with many discoveries ahead. He describes the basic components of RL solutions, such as value functions (predicting future reward), policies (deciding actions), and models (representing the environment), as essential pieces in decomposing and solving this ambitious problem of intelligence.

I believe alpha zero is one of the most important accomplishments in the history of artificial intelligence and David is one of the key humans who brought alpha zero to life together with a lot of other great researchers at deep mind he's humble kind and brilliant

it was always more than solving puzzles it was something where you know there was this limitless possibilities once you have a computer in front of you you can do anything with it

unless you believe in something metaphysical then what are our brains doing well at some level their information processing systems

I found it immensely satisfying that a system which was able to learn from first principles for itself was able to reach the point that it was understanding this domain better than better than I could and able to outwit me

to me alphago and alpha a girl zero mastery in the game of girl is again to me the most profound and inspiring moment in the history of artificial intelligence

in order to crack go we would need to get something akin to human intuition and if we got something akin to human intuition we'd be able to self you know much many many more problems in AI

I strongly felt that learning would be necessary and that's why my my PhD topic back then was trying to apply reinforcement learning to the game of CO and not just learning of any type but I felt that the only way to really have a system to progress beyond human levels of performance wouldn't just be to mimic how humans do it but to understand for themselves

the more you try to put into something the more brittle the system becomes and and so you just have to have learning you have to have learning that's the only way you're going to be able to get a system which has sufficient knowledge in it

the game of CO was has remarkably simple rules if that's so simple that people have speculated that if we were to meet alien life at some point that we wouldn't be able to communicate with them but we would be able to play hello go with that probably have discovered the same rule set

I see the problem of intelligence I would say it can be formalized as the reinforcement learning problem and that that formalization is enough to capture most if not all of the things that we mean by intelligence

reinforcement learning is the study and the science and the problem of intelligence in the form of an agent that interacts with an environment