Taming the Long Tail: Waymo's Machine Learning Approach to Autonomous Driving Challenges

Drago Anguelov, a Principal Scientist at Waymo, details the company's journey in autonomous driving, marking a decade of development and over 10 million miles driven on public roads, culminating in the launch of their commercial service in Phoenix. The central theme of his discussion is "taming the long tail" – the immense challenge of handling the vast number of rare, unexpected, and complex scenarios that autonomous vehicles (AVs) encounter beyond common driving situations. He illustrates this with vivid examples such as a bicyclist carrying a stop sign, objects falling onto the road, navigating complex construction zones, and reacting to red-light runners, underscoring the necessity for AVs to be exceptionally capable, safe, and robust across an inherently diverse and unpredictable real-world environment.

Anguelov dissects the core AI tasks crucial for self-driving: perception, which involves mapping sensory inputs and prior knowledge into a comprehensive scene representation; prediction, focused on anticipating the future behavior of other agents, particularly humans, who introduce significant complexity; and planning, which generates safe, comfortable, and communicative vehicle actions. He emphasizes that understanding and predicting human behavior, often requiring long-term anticipation and the interpretation of subtle cues, is one of the most challenging aspects. Anguelov advocates for machine learning (ML) as the indispensable tool for modeling these intricate functions and effectively addressing the long tail, contrasting it with traditional, artisan-like system development by conceptualizing modern ML as a scalable "factory" that continuously refines models through iterative data processing.

The "ML factory" paradigm at Waymo operates through a continuous feedback loop: deploying software, collecting vast amounts of driving data, intelligently selecting data (especially focusing on long-tail cases via active learning and data mining), labeling this data, training ML models, and rigorously testing and validating them. Waymo leverages Google's extensive computing infrastructure, including TensorFlow, and benefits from deep learning expertise, having contributed to pioneering neural net architectures like Inception and SSD. A significant innovation highlighted is the application of AutoML for neural architecture search, which automates the design of high-performance, low-latency models, frequently surpassing human-engineered designs and adapting to evolving data requirements.

To ensure resilience in situations where ML models might lack confidence or sufficient training examples, Waymo implements a hybrid system approach. This strategy incorporates redundant and complementary sensors (camera, lidar, and radar), each with distinct strengths and failure modes, and integrates expert domain knowledge to introduce biases that constrain representations, facilitating more efficient model learning with fewer examples. The system is also designed to drive more conservatively when predictions are uncertain. Large-scale simulation, simulating billions of miles, is paramount for testing, enabling the exploration of rare scenarios, generating variations of real-world logs, and critically, accounting for how the AV's actions influence the simulated environment and other agents. This multi-faceted strategy aims to continually expand ML capabilities while simultaneously safeguarding against current limitations, thereby ensuring both safety and sustained progress in autonomous mobility.

"taming the long tail of autonomous driving challenges"

"the world's first fully autonomous ride on public roads"

"we have driven 10 million miles on public road"

"the long tail of situations you it's it's it's one type of effort to get yourself driving for the common cases and then it's another effort to tame this the rest and they really really matter"

"perception is mapping from sensory inputs in potentially prior knowledge of the environment to seen representation"

"people is honestly what makes driving quite challenging"

"machine learning is a tool to handle the long tail"

"modern machine learning is like a factory"

"we simulate the equivalent of 25,000 cars virtual cars driving ten million miles a day and seven over seven billion miles simulated"

"in simulation our actions affect the environment and it need to be accounted for"

"we found better models than the human engineers did"

"our system is this hybrid it's an example of what that looks for perception is well with no matter if the there's cases where the machine learning system may be not confident we still have tracks and obstacles from leather and radar scans and we make sure that we we drive relative to those safely"