Jim Keller on Elon Musk's First Principles, Tesla Autopilot's Future, and the Craft of Engineering

This podcast clip features Jim Keller, a veteran chip architect, discussing his experiences with Elon Musk and Tesla's Autopilot, delving into the philosophical and technical challenges of autonomous driving. Keller highlights Musk's "first principles thinking"—a radical approach to problem-solving that involves deconstructing problems to their fundamental truths rather than relying on existing paradigms. This mindset, he explains, is crucial for tackling complex challenges like building rockets or self-driving cars, pushing engineers to question deeply held assumptions and embrace intellectual discomfort to achieve breakthroughs. The conversation underscores the distinction between incremental improvements and foundational innovation, suggesting that true progress often requires a complete re-evaluation of what is possible.

A significant portion of the discussion revolves around the complexity of vehicle autonomy, particularly the debate between human and machine capabilities. Keller acknowledges the incredible sophistication of the human vision system and our ability to infer and predict, but also points out that computers excel at sustained attention and memory—areas where humans are prone to error. He introduces the concept of cars as "ballistic things with tracks and probabilistic changes," suggesting that from a computational perspective, driving can be broken down into solvable data problems. However, he also concedes that human behavior, with its inherent unpredictability and reliance on "theories" about other drivers, adds a layer of complexity that current AI systems struggle with, making the problem more than just a simple ballistics calculation.

Keller also sheds light on the engineering challenges of developing specialized hardware for autonomous vehicles. He discusses the tension between building highly specialized AI accelerators for maximum performance and maintaining enough programmability to adapt to rapidly evolving machine learning algorithms. Cost constraints, particularly Elon Musk's vision of putting an Autopilot computer in every Tesla, drive innovative system design to make advanced technology affordable. He likens much of engineering to "craftsmen's work"—a process of intricate, satisfying problem-solving that requires skill, attention to detail, and continuous learning, rather than just pure genius, drawing parallels to a violin maker or even assembly line workers.

Ultimately, Keller expresses confidence in the near-term solvability of autonomous driving, citing the rapid advancements in data collection, computation, and deep learning algorithms. He believes that while human behavior introduces significant hurdles, the machine's superior attention, memory, and ability to process vast amounts of data will lead to autonomous systems that are demonstrably safer than human drivers within years. The discussion concludes by reinforcing the idea that progress, while often disappointing in the short run, frequently delivers surprising breakthroughs in the long run, especially when driven by a relentless pursuit of first principles and a willingness to challenge the status quo.

"first you figure out what configure machine you want the atoms in and then how to put them there"

"you don't have to be especially smart to drive a car so it's not like a super hard problem"

"the big problem with safety is attention which computers are really good at not skills"

"it's not just detecting object it's understanding the scene and it's being able to do it in a way that doesn't make errors"

"if cars are ballistic things with tracks and probabilistic changes of speed and direction and roads are fixed and given by the way they don't change dynamically right you can map the world really thoroughly you can place every object really thoroughly right you can calculate trajectories of things really thoroughly right"

"autonomous cars were always on autopilot but the cars have no theories about why they got cut off or why they're in traffic so they'll never stop paying attention"

"progress disappoints in the short run the surprises in the long run"

"I'm hoping that there's a lot of things that humans aren't good at that machines are definitely good at said attention and things like that will they'll be so much better that the overall picture of safety in autonomy will be obviously cars will be safer"

"I want the acceleration afforded by specialization without being over specialized so that the new algorithm is so much more effective that you'd have been better off on a GPU"

"most engineering is craftsmen's work"

"you know he has a deep belief that no matter what you do is a local maximum"

"you know that you know ability to look at it without assumptions and and how constraints is super wild"

"I imagine 99 percent of your thought process is protecting your self conception and 98% of that's wrong"

"when people write books they often take 20 years of their life where they passionately did something reduce it to to 200 pages that's kind of fun"

"autonomous driving is something we can solve on a timeline of years so 1 2 3 5 10 years as opposed to a century yeah definitely"