Waymo's Decade-Long Journey: Engineering Self-Driving Perception with Deep Learning

Sacha Arnoud, Director of Engineering at Waymo, details the company's decade-long journey from a Google project to a leader in autonomous driving, emphasizing the profound engineering challenges and the pivotal role of deep learning. He highlights Waymo's mission to enhance safety, accessibility, and efficiency in mobility, citing that 94% of crashes involve human error. A key takeaway is the "90% done, 90% to go" principle, illustrating the exponential effort required to transition a working demo into a safe, production-ready system, demanding 10x improvements in technology, team size, sensor capabilities, and testing rigor.

The presentation traces the evolution of deep learning within Alphabet, from the Google Brain team's foundational research to its early application in Google Street View for mapping addresses and business listings. This early success in understanding real-world scenes, particularly with the first production deep learning system in 2012 for address mapping, laid the groundwork for autonomous vehicle development. Arnoud distinguishes between basic collision avoidance and the much deeper semantic understanding and behavior prediction necessary for truly driverless operation, such as anticipating a car swerving to avoid a cyclist.

Waymo's approach to perception relies on a suite of complementary sensors—vision systems, radar, and lidar—many of which are custom-designed in-house to achieve superior performance. A significant challenge lies in processing noisy real-world sensor data, such as reflections or exhaust smoke, which requires sophisticated machine learning-driven filtering. Deep learning, particularly convolutional layers, is crucial for extracting features and building a high-level semantic understanding of the scene, all processed in real-time on embedded systems within the vehicle, independent of cloud connectivity.

The broader implications of Waymo's work extend beyond just driving, promising to reshape urban environments, traffic management, and parking. The continuous testing and expansion of driverless operations in areas like Phoenix underscore the commitment to a product launch. The collaboration between Google Research and Waymo ensures the system remains at the bleeding edge of AI capabilities, constantly pushing the boundaries of what autonomous systems can perceive, understand, and predict for safe and reliable operation.

"94% of us crashes today involve human errors a lot of those errors are around distraction and things that could be avoided so safety is a big piece of it"

"our mission at Waco is fundamentally to to make it safe and easy to move people and things around"

"the first objective of the project was to try and assemble first product a vehicle take off-the-shelf sensors assemble them together and try to go and decide if self-driving is even a possibility"

"when you are 90% done you still have 90 percent to go right so 90% of the technology takes only 10% of the time"

"you need to 10x the capabilities of your technology you need to 10x your team size and find ways for more engineers and more researchers to collaborate together you need to 10x the capabilities of your sensors you need to 10x fundamentally the overall quality of the system"

"the first deepening application that that succeeded in production and that's all the way back to 2012 that we had the first system in production was really the first breakthrough that we had across across alphabet on our ability to properly understand real scene situations"

"driving requires you to take the conditions that they are and you have to deal with it"

"the better your sensors are the better your perception system is gonna be right so that's why at way more we we took the path of designing our own sensors in-house"

"if you take that car too seriously all the way to impacting your behavior obviously you're gonna make mistakes"

"only then only when you have that that depth of understanding you can start to come up with realistic behavior predictions and trajectory predictions for all those agents in the in on the scene so that you can come up with a proper strategy for your planning control"