Yann LeCun on the Fundamental Limitations of Auto-Regressive LLMs and the Path to Advanced Machine Intelligence via Joint Embedding Predictive Architectures

Yann LeCun, a seminal figure in AI and Meta's Chief AI Scientist, critically assesses auto-regressive Large Language Models (LLMs) like GPT-4 and Llama, arguing they are fundamentally limited in their path towards human-level intelligence. He contends that LLMs lack essential characteristics of intelligent behavior, such as understanding the physical world, persistent memory, reasoning, and planning. LeCun highlights the vast disparity in data input, noting that a four-year-old processes orders of magnitude more sensory information than LLMs trained on the entire internet's text, suggesting that real-world interaction, not just language, is crucial for learning.

LeCun emphasizes that intelligence must be grounded in reality, whether physical or simulated, as the environment is far richer than what can be expressed in language. He distinguishes between language, which is a compressed and low-bandwidth representation, and the rich, redundant sensory input from the physical world. He introduces the "Moravec Paradox," explaining why complex tasks like chess are easier for computers than seemingly simple physical tasks like driving or clearing a dinner table. He further differentiates LLMs' auto-regressive, token-by-token generation from human thought processes, which involve abstract, language-independent planning and mental models before articulation.

LeCun proposes Joint Embedding Predictive Architectures (JEPAs) as a more promising path. Unlike generative models that struggle to predict every pixel in high-dimensional continuous data like video, JEPAs aim to predict abstract representations of inputs. This approach allows the system to extract predictable information while filtering out unpredictable noise, thereby learning hierarchical, abstract representations of the world. He explains the evolution from contrastive learning to non-contrastive methods in JEPAs, which avoid the need for negative samples and prevent model collapse, offering a more robust self-supervised learning mechanism.

The discussion extends to the broader implications of AI development, particularly the debate around AGI and open-source AI. LeCun, a strong proponent of open-sourcing AI, views the concentration of power in proprietary AI systems as a significant danger, advocating for open access to empower human goodness. He maintains an optimistic stance on AGI, believing it will be beneficial and controllable, contrasting with "doomers" who fear existential threats. His work with JEPAs suggests a paradigm shift from purely language-centric AI to systems that learn from rich, redundant sensory data, potentially unlocking more robust and human-like intelligence.

"I see the danger of this concentration of power to to proprietary AI systems as a much bigger danger than everything else."

"Auto-regressive LLMs are uh not the way we're going to make progress towards superhuman intelligence."

"LLMs can do none of those [understand the physical world, persistent memory, reason, plan] or they can only do them in a very primitive way."

"Most of what we learn and most of our knowledge is through our observation and interaction with the real world not through language."

"Intelligence cannot appear without some grounding in uh some reality."

"This is the old Moravec Paradox from the pioneer of robotics and SMC we said you know how is it that with computers it seems to be easy to do high level complex tasks like playing chess and solving integrals and doing things like that whereas the thing we take for granted that we do every day... we can't do as computers."

"The kind of thinking that we're doing and the answer that we're planning to produce is not linked to whether we're going to see it in French or Russian or English."

"Building World models means observing the world and uh understanding why the world is evolving the way the way it is and then uh the the extra component of a world model is something that can predict how the world is going to evolve as a consequence of an action you might take."

"In a jepa you're not trying to predict all the pixels you're only trying to predict an abstract representation of of the inputs right and that's much easier in many ways."

"Language is already to some level abstract and already has eliminated a lot of information that is not predictable and um so we can get away without doing the tring without you know lifting the abstraction level and by directly predicting words."