Ilya Sutskever on Building AGI: Self-Play, Simulation, Consciousness, and Human Control

Ilya Sutskever, a leading AI researcher, discusses the foundational elements required to build Artificial General Intelligence (AGI), positing that deep learning combined with "another small idea" will be key. He emphasizes the critical role of self-play, highlighting its capacity to generate surprising and creative solutions, a trait currently lacking routinely in AI systems. The conversation delves into the utility of simulation, acknowledging its strengths as a tool for learning while addressing criticisms regarding its transferability to real-world scenarios. Sutskever points to successful sim-to-real transfers, such as OpenAI's Rubik's Cube robot hand, as evidence of deep learning's increasing capability to bridge the gap between simulated and physical environments, allowing systems to learn generalizable "morals of the story."

The discussion extends to the necessity of embodiment and consciousness for AGI. While acknowledging that a physical body is "very useful" for learning unique things, Sutskever suggests it may not be strictly necessary, drawing parallels to human compensation for sensory loss. Regarding consciousness, he finds it "definitely possible" for AI systems to be conscious, arguing from an existence proof that if humans are conscious and neural nets are sufficiently similar to the brain, then conscious neural nets should exist. He also touches upon the challenge of defining and testing intelligence, expressing a desire for AI systems that never make "mistakes a human wouldn't make," which he believes would inspire greater confidence in their intelligence.

Sutskever addresses the societal implications and control mechanisms for AGI, envisioning an ideal future where humanity acts as a "board of directors" to an AGI "CEO." This model suggests a democratic process where AGI systems represent and serve human interests, with the ultimate power to "fire" or "reset" the AGI. Crucially, he asserts that it is "definitely possible to build AI systems which will want to be controlled by their humans," driven by a deep, programmed desire to help humanity flourish, akin to a parent's desire to help their children.

The conversation concludes with a focus on AI alignment, where Sutskever suggests training an "as objective as possible perception system" to internalize human judgments, which would then serve as the base value function for more capable reinforcement learning systems. He expresses a personal aversion to holding sole power over an AGI, finding the scenario "terrifying," and believes that "when it really counts people can be better than we think" in relinquishing such power. This highlights the ethical considerations and the importance of designing AI with inherent drives for human benefit and distributed control.

"I think the deep learning plus may be another small idea."

"self play has this amazing property that it can surprise us in truly novel ways."

"I don't think it's an either/or I think simulation is a tool and it helps you has certain its strengths and certain weaknesses and we should use it."

"I expect the transfer capabilities of deep learning to increase in general and the better the transfer capabilities are the more useful simulation will become."

"I think having a body will be useful I don't think it's necessary but I think it's very useful to have a body for sure because you can learn a whole new you can learn things which cannot be learned without a body."

"humans are conscious and if you believe that artificial neural Nets are sufficiently similar to the brain then there should at least exist artificial neural Nets you should be conscious to."

"I would be impressed by deep learning system which solves a very pedestrian you know pedestrian tasks like machine translation or computer vision tasks or something which never makes mistake a human wouldn't make under any circumstances."

"it's definitely possible to build AI systems which will want to be controlled by their humans."

"it will be possible to program an AGI to design it in such a way that it will have a similar deep drive that it will be delighted to fulfill and the drive will be to help humans flourish."

"I find it trivial to do that I'd finally trivial to little really really this kind of pot I mean you know there's a kind of scenario you are describing sounds terrifying to me that's all I would absolutely not want to be in that position."

"when it really counts people can be better than we think."