Marcus Hutter on AIXI, Kolmogorov Complexity, and the Computable Universe

The conversation with Marcus Hutter, a senior research scientist at Google DeepMind, delves into his foundational work on Artificial General Intelligence (AGI), particularly the AIXI model, which integrates Kolmogorov complexity, Solomonoff induction, and reinforcement learning. Hutter posits a profound hypothesis that the universe is inherently computable, beautiful, and simple, drawing parallels with fundamental physical theories like general relativity and quantum field theory. He champions Occam's Razor as the most important principle in science, arguing that its success in finding simple, predictive models suggests an underlying simplicity in the world, a bias that Solomonoff induction formalizes.

Hutter elaborates on Solomonoff induction as a theory that solves the philosophical problem of induction by seeking the shortest program that reproduces observed data, weighing simpler models higher using Bayesian techniques. He defines Kolmogorov complexity as the ultimate measure of information content, representing the length of the shortest self-extracting archive for a dataset. A key nuance is the distinction between the universe's overall simplicity—potentially describable by a very short program—and the emergent complexity of its subsets, such as planet Earth, which cannot be easily compressed. He also discusses the role of noise and chaotic systems in making scientific understanding harder, yet potentially enabling phenomena like free will.

The Hutter Prize, initially 50,000 euros and now increased to 500,000 euros, incentivizes the development of intelligent compressors as a direct path to AGI, based on the principle that superior compression implies higher intelligence. Hutter's definition of intelligence as "an agent's ability to perform well in a wide range of environments" provides a robust framework for AGI development, suggesting that traits like creativity, memorization, and planning are emergent from this core ability. He highlights AlphaZero as a remarkable example of a reinforcement learning algorithm achieving general intelligence within specific domains (chess, Go) through self-play, even rediscovering complex strategies from scratch.

The discussion underscores the profound connection between theoretical computer science, physics, and the quest for creating truly intelligent machines. Hutter believes general intelligence is possible and that machines can achieve it, citing the progress of conversational AI like Google's Meena and the Alexa Prize. The conversation ultimately suggests that the human endeavor of science is a form of compression, seeking simple rules to understand and predict the world, and that this pursuit, driven by an evolutionary bias towards pattern recognition, might culminate in a single, simple equation describing intelligence itself.

the universe is inherently beautiful elegant and simple and described by these equations and we're not just picking that

Occam's razor says that you should not multiply entities beyond necessity which sort of if you translate it to proper English means... if you have two series or hypotheses or models which equally well describe the phenomenon your study or the data you should choose the more simple one

I believe that Occam's razor is probably the most important principle in science

if we start with the assumption that the world is governed by simple rules then there's a bias toward simplicity and pliant Occam's razor is the mechanism to finding these rules

Solomonoff induction... solves the big philosophical problem of induction

you are looking for the shortest program which if you run this program reproduces the data you have it will not stop it will continue naturally and this you take for your prediction

compression means for me finding short programs for the data or the phenomena at hand

Kolmogorov complexity... is the length of the shortest possible self-extracting archives you could produce for a certain data set... arguably that is the information content in the data set

intelligence measures an agent's ability to perform well in a wide range of environments