The Underestimated Complexity of Computer Vision, AI, and Autonomous Driving with Jitendra Malik

The conversation with Jitendra Malik, a leading figure in computer vision, delves into the profound and often underestimated complexity of the field. Malik highlights that human vision's subconscious nature leads early AI researchers and even current practitioners to misjudge the difficulty of replicating it computationally. He introduces the "fallacy of the successful first step," where initial progress (e.g., 50% solution) is easily achieved, but reaching higher accuracy (99.99%) becomes exponentially harder, often beyond a lifetime. This inherent difficulty is evident when comparing the vast neural resources dedicated to vision in the human brain to the current capabilities of AI systems.

Malik distinguishes between lower and mid-level vision tasks (like basic perception) where current deep learning excels, and higher-level "cognitive" tasks that require common sense reasoning, predictive modeling, and understanding of intentionality. He argues that these higher-level aspects are crucial for real-world applications like fully autonomous driving, where error tolerance is near zero. He critiques the "tabula rasa" approach of current deep learning, contrasting it with human learning, which builds upon years of pre-existing visual knowledge and intuitive physics before formal instruction (e.g., driver's ed). This suggests a need for richer learning paradigms beyond supervised learning.

The discussion touches on the limitations of current vision-based autonomous driving systems, using Tesla Autopilot as an example. Malik points out that while certain subsets of driving (like freeway conditions) are solvable, the "edge cases" requiring sophisticated cognitive reasoning remain a significant hurdle. He emphasizes the need for systems to build predictive models of other agents' behaviors, which he includes under the umbrella of perception blending into cognition. For long-form video understanding, he advocates for incorporating "schemas" or "scripts" – structured knowledge about typical event sequences – which humans acquire through observation, suggesting a learning-based approach rather than hand-coding.

Malik connects computer vision to its biological roots, asserting that perception's fundamental purpose is to guide action, a principle evident from the earliest multicellular organisms. He notes that historical constraints on compute and storage led to simplified approaches (like edge detection), but modern capabilities allow for more holistic processing. While current GPUs might match the brain's raw computational power (as per Hans Moravec's calculations), they lack the brain's energy efficiency and style of computing. The conversation implies that true general-purpose AI vision will require not just more data and compute, but fundamentally new learning architectures that can accumulate and leverage general visual knowledge and common sense in a way that mimics human cognitive development.

most of what we do in vision we do unconsciously or subconsciously in human vision so that gives us this that effortlessness gives us the sense that oh this must be very easy to implement on a computer

the fact is that a very large part of the the cerebral cortex is devoted to visual processing

I think why people fall for this is because of what I call the fallacy of the successful first step

getting 50 of the solution you can get in one minute getting to 90 percent can take you a day getting to 99 percent may take you five years and 99.99 may be not in your lifetime

perception blends into cognition and building predictive models of other agents in the world... that is part of the task of perception because perception always has to uh not tell us what is now but what will happen because what's now is boring it's done it's over with

the current learning techniques that we have yeah is our supervised learning you're given lots of examples xiy pairs and you you learn the functional mapping between them I think that human learning is far richer than that

perception is always to guide action perception in a for a biological system does not give any benefits unless it is coupled with action

we must at the fundamental level connect perception to action

the style of computing that we have in our gpus is far far more power hungry than the style of computing that is there in the human brain or other biological uh entities

long-term understanding requires a notion of this notion of schemas that I talked about perhaps some notions of goals intentionality functionality and so on and so forth