Deep Learning for Human Sensing in Autonomous Vehicles: Data, Annotation, and Human-Centered AI

This lecture from MIT's 6.S094 course delves into the application of deep learning methods, specifically computer vision, for understanding human behavior in the context of autonomous driving. The core argument emphasizes that while algorithms are exciting, the "painful boring stuff" of collecting, cleaning, and annotating real-world data is paramount for creating successful, robust, and generalizable systems. The speaker highlights the immense challenges of data collection, which involves capturing diverse real-world scenarios, identifying rare but critical events (the "1%"), and developing sophisticated annotation tools tailored to specific tasks like glance classification or body pose estimation, often leveraging human computation for labeling. Large-scale distributed computing and storage are also identified as essential hardware requirements for processing the vast datasets, such as the MIT group's five billion images of driving data. A significant portion of the discussion addresses human imperfections in driving, such as distraction (texting, eating), drunk driving, and drowsy driving, which contribute to tens of thousands of fatalities annually. The lecture contrasts two approaches to autonomous vehicles: full autonomy, which seeks to remove the imperfect human from the loop, versus a human-centered approach that advocates for collaboration between humans and AI. The speaker argues against the full autonomy path, citing the human tendency to over-trust and misuse technology, as exemplified by drivers circumventing steering wheel touch sensors. Instead, the lecture champions AI systems that perceive and work with human beings to enhance safety and alleviate distraction, acknowledging that humans are "amazing" drivers despite their flaws. The MIT group's naturalistic driving data set, comprising 25 instrumented vehicles (21 Teslas) collecting thousands of miles daily, serves as the foundation for this human-centered research. This data, captured by multiple cameras monitoring the driver's face, body, and the external scene, is synchronized with GPS and vehicle data to understand driver behavior and train deep neural networks. The goal is not only to improve safety but also to ensure an "enjoyable" driving experience, as evidenced by the high usage rates of Tesla Autopilot. Initial findings from glance classification algorithms suggest that drivers using Autopilot maintain similar on-road attention distributions as manual drivers, indicating a potential "sweet spot" where users gain value without excessive over-trust. The lecture then details specific computer vision tasks, starting with the "easier" pedestrian detection using techniques like R-CNN and its variants (Fast R-CNN, Mask R-CNN) for bounding box localization and segmentation. It progresses to the more complex body pose estimation, which identifies joint landmarks for understanding driver alignment (e.g., slouching) and pedestrian nonverbal communication. Finally, glance classification is presented as a critical, learnable machine learning problem that classifies driver gaze into broad regions (on-road, off-road, rearview mirror, etc.) rather than precise geometric gaze estimation. This region-based approach is crucial because real-world ground truth for exact gaze is unavailable, making it amenable to data-driven training and robust deployment in variable lighting and human behavior scenarios.

data is everything

data collection is the hardest part and the most important part

the painful boring stuff of collecting data of cleaning that data of annotating that data in order to create successful systems is much more important than good algorithms or great algorithms

humans are amazing

eyes off road for five seconds is the average time your eyes off the road while texting

the tempting path is towards full autonomy where we removed this imperfect flawed human from the picture altogether

we work together human and machine to improve the safety to alleviate distraction to bring drive our attention back to the road and use artificial intelligence to increase safety through collaboration human robot interaction versus removing the human completely from the picture

if there's any sensing that I advocate and is has the most impact in the driving context is for the car to know where the driver is looking

it's not possible to design systems that deal with lighting variability and the full variability of human behavior without being able to learn

people are driving 33% of miles using Tesla autopilot what does that mean that people are getting value from it