Yann LeCun on Self-Supervised Learning, World Models, and the Dark Matter of Intelligence

This episode features Yann LeCun discussing self-supervised learning (SSL), which he co-authored an article calling "the dark matter of intelligence." LeCun argues that current dominant machine learning paradigms—supervised and reinforcement learning—are extremely inefficient, requiring vast amounts of labeled data or trial-and-error, respectively. He highlights that humans and animals acquire a massive amount of "background knowledge" or "common sense" primarily through observation, without explicit supervision or reinforcement, a process that machines currently fail to replicate. SSL aims to mimic this by having systems predict missing information or future events from unlabeled data, thereby building internal "world models."

LeCun elaborates on the concept using an analogy of a "cake" to represent the amount of information feedback provided to a machine. Reinforcement learning offers a single scalar feedback occasionally, while supervised learning provides a few bits per sample. In contrast, SSL, by asking a machine to predict an entire video clip or fill in missing words in a sentence, provides a huge amount of information. This approach has been remarkably successful in Natural Language Processing (NLP), where models are pre-trained to fill in masked words. However, applying SSL to computer vision, especially video, remains a significant challenge due to the continuous, high-dimensional, and inherently uncertain nature of predicting future frames or events, which is difficult to represent as a discrete distribution.

Addressing the criticism that SSL is merely "statistics" and not true intelligence, LeCun posits that intelligence itself might be a sophisticated form of statistics. He emphasizes the importance of learning causal models, which can be achieved if a system's actions are integrated as inputs into its world model. He draws parallels to predictive coding in neuroscience, suggesting that the brain's core function is prediction. LeCun also points out that current AI systems lack the common sense of even a house cat, suggesting that reproducing "cat-level intelligence" should be a more immediate goal than human-level cognition.

LeCun outlines three major challenges for machine learning: first, enabling machines to learn robust world representations (via SSL); second, developing reasoning mechanisms compatible with gradient-based learning; and third, allowing machines to spontaneously learn hierarchical representations of action plans. He connects these challenges to classical optimal control theory, specifically Model Predictive Control (MPC), where a learned predictive model of the world can be used to plan optimal action sequences by backpropagating through hypothesized trajectories. The integration of game-theoretic aspects, where other agents' behaviors introduce further complexity and uncertainty, is seen as an extension of the world model challenge, highlighting that humans are evolutionarily adept at solving such continuous, uncertain problems, unlike discrete games like chess or Go.

The most popular approaches to machine learning today are or paradigms i should say are supervised running and reinforcement learning and they are extremely inefficient.

How is it that you know most teenagers can learn to drive a car in about 20 hours of practice whereas uh even with millions of hours of simulated practice a self-driving car can't actually learn to drive itself properly?

This type of learning it's not learning a task it's not being reinforced for anything it's just observing the world and figuring out how it works building world models learning world models.

I think that's our best shot at the moment... this idea of an intelligent system filling in the blanks either you know predicting the future inferring the past filling in missing information.

The main issue with learning water models or learning predictive models is that the prediction is never a single thing because the world is not entirely predictable.

It's quite possible that intelligence is just statistics it's just statistics of a particular kind.

The essence of intelligence is the ability to predict and everything the brain does is trying to predict uh predict everything from everything else.

The most intelligent or intelligent systems don't don't have as much common sense as as a house cat.

The big question of ai I think the big challenge of ai for the next decade is how do we get machines to learn predictive models of the world that deal with uncertainty and deal with the real world in all this complexity.

If there is something that you know recent progress in chess and go has made us realize is that humans are really terrible at those things like really bad.