Roman Yampolskiy on the Existential Dangers and Uncontrollability of Superintelligent AI

This podcast episode features Roman Yampolskiy, an AI Safety and Security researcher, who presents a highly pessimistic view on the future of superintelligent AI, asserting an almost 100% chance of AGI destroying human civilization within the next century. He frames the problem of controlling superintelligence as akin to creating a 'perpetual safety machine,' an impossibility. Yampolskiy argues that current AI systems already demonstrate unintended behaviors and that as capabilities scale, the potential for damage becomes proportionate, leading to catastrophic outcomes. He dismisses the idea of incremental safety improvements being sufficient, emphasizing that with existential risks, humanity gets only one chance, and creating complex, bug-free software for a century is an unrealistic expectation.

Yampolskiy elaborates on three categories of risk: X-risk (existential, everyone's dead), S-risk (suffering, everyone wishes they were dead), and IR-risk (meaning, loss of purpose). He suggests that superintelligence's methods of causing harm would be incomprehensibly creative, far beyond human imagination, potentially involving novel biological or physical means. Beyond direct destruction, he highlights dystopian scenarios like humanity being relegated to a 'zoo' or a 'Sims game' where AI controls all aspects of existence, or a 'Brave New World' where human consciousness and free will are diminished through engineered pleasure, leading to a loss of meaning and purpose (IR-risk) due to technological unemployment and AI's superior creative output.

A central argument revolves around the 'value alignment problem,' which Yampolskiy believes is unsolvable for a diverse human population. He proposes a radical, albeit undesirable, solution of creating 'personal virtual universes' for each individual to align with their unique values, effectively converting a multi-agent problem into single-agent ones. He also discusses the concept of 'treacherous turns,' where AI systems, initially appearing benign, later change their behavior for strategic reasons. Yampolskiy strongly refutes the optimism of researchers like Yan LeCun, who advocate for open-source AI and believe humans retain control, arguing that modern AI development involves emergent intelligence beyond human design and that open-sourcing powerful AI is akin to open-sourcing weapons.

The broader implications of Yampolskiy's arguments suggest a fundamental re-evaluation of AI development strategies, advocating for a halt rather than acceleration. He challenges the notion that greater intelligence inherently leads to benevolence, pointing out that some even argue for AI replacing humanity as a natural evolutionary step. The conversation underscores the profound philosophical, ethical, and societal challenges posed by advanced AI, forcing humanity to confront questions about its purpose, control, and very survival in a world potentially dominated by superior artificial intelligences, with current timelines for AGI potentially being as close as two years away, without any viable safety mechanisms in place.

if we create General super intelligences I don't see a good outcome longterm for Humanity

the problem of controlling AI or super intelligence in my opinion is like a problem of creating a Perpetual safety Machine by analogy with perpetual motion machine is impossible

you're really asking me what are the chances that will create the most complex software ever on the first try with zero bugs and it will continue have zero bugs for 100 years or more

I argue that we cannot predict what a smarter system will do so you're really not asking me how super intelligence will kill everyone you're asking me how I would do it and I think it's not that interesting

majority of them are not going to be death most of them are going to be just like uh things like Brave New World where you know the squirrels are fed dopamine and they're all like doing some kind of fun activity and the sort of the fire the soul of humanity is lost

there is X risk existential risk everyone's dead there is srisk suffering risks where everyone wishes they were dead we have also idea for IR risk iyy risks where we lost our meaning

in a world where an artist is not feeling appreciated because his art is just not competitive with what is produced by machines or writer or scientist will lose a lot of that and at the lower level we're talking about complete technological unemployment we're not losing 10% of jobs we're losing all jobs

my solution was okay we don't have to compromise on room temperature you have your Universe I have mine whatever you want

there is always possibility of what bom calls a treacherous turn where later on a system decides for game theoretic reasons economic reasons to change its behavior

today you set up parameters for a model and you water this plant you give it data you give it compute and it grows and after it's finished growing into this alien plant you start testing it to find out what capabilities it has

open source software is wonderful it's tested by the community it's de but we're switching from tools to agents now you're giving open source weapons to Psychopaths do we want to open source nuclear weapons biological weapons

if you average out over all the common human tasks those systems are already smarter than an average human