Convolutional Neural Networks for End-to-End Learning of the Driving Task: Computer Vision, Image Classification, and Self-Driving Car Automation

This lecture delves into Convolutional Neural Networks (CNNs) as a cornerstone for modern computer vision tasks, particularly their application in enabling end-to-end learning for autonomous driving. It begins by contrasting CNNs with traditional neural networks, emphasizing their specialized architecture designed to process spatially consistent data like images. Key concepts such as receptive fields, shared weights, and pooling layers are introduced as mechanisms that allow CNNs to efficiently extract meaningful features from raw pixel data, leading to outputs like steering angles for vehicles or discrete class labels for objects. The discussion highlights the shift towards data-driven approaches in computer vision, where machine learning algorithms learn directly from vast image datasets.

The speaker meticulously outlines the inherent difficulties of computer vision for machines, including challenges like viewpoint and size variation, occlusions, background clutter, and fluctuating illumination – problems that human perception handles with ease. The lecture differentiates between regression tasks (predicting continuous values like steering angles) and classification tasks (assigning discrete labels like "cat" or "dog"). A foundational image classification method, K-Nearest Neighbors (KNN), is presented, demonstrating its limited accuracy (38% on CIFAR-10) compared to human performance (94%) and the superior capabilities of state-of-the-art CNNs (95.4%), underscoring the transformative power of deep learning when sufficient data is available.

Practical aspects of training these sophisticated models are thoroughly explored. The importance of diverse datasets such as Mnist, ImageNet, CIFAR-10, and Places is stressed, along with the critical process of hyper-parameter tuning (e.g., selecting the optimal 'K' for KNN). The lecture details the rigorous methodology of splitting data into training, validation (for cross-validation and tuning), and test sets to ensure models generalize well and avoid overfitting. The architectural components of CNNs – input, convolutional, pooling, and fully connected layers – are explained in sequence, illustrating how they collectively enable advanced functionalities like object segmentation and detection within complex visual scenes.

Finally, the broader implications of CNNs for autonomous driving are contextualized within societal challenges. The lecture highlights the staggering number of annual driving fatalities and the prevalence of human distractions (texting, impairment, drowsiness) as compelling reasons for automation. Tesla's Autopilot serves as a real-world case study, prompting a discussion on the psychological aspects of transferring control and trust from human drivers to machines operating at high speeds. The immense need for extensive real-world driving data, exemplified by MIT's collection of 70,000 miles from instrumented Tesla vehicles, is emphasized as crucial for understanding, refining, and ultimately proving the safety and efficacy of autonomous systems, suggesting that automation may already be contributing to safer roads.

Today we'll continue to talk about neural networks that work with images, convolutional neural networks, and see how those types of networks can help us drive a car.

Computer vision is, these days, dominated by data driven approaches by machine learning where all of the same methods that are used on other types of data are used on images where the input is just a collection of pixels and pixels are numbers from 0 to 255 discrete values.

Regression is when given an image we want to produce a real value of output put back. So if we have an image of the four roadway, we want to produce a value for the steering wheel angle...

I mentioned this yesterday but it bears saying again, computer vision is hard. We, once again, take it for granted. As human beings, we're really good at dealing with all these problems.

Human beings perform at about 94, slightly above 94% accuracy for CIFAR-10... And the currently state-of-the-art convolutional neural networks is ninety five, it's 95.4% accuracy and, believe it or not, it's a heated battle but the most important, the critical fact here, is it's recently surpassed humans.

Convolutional neural networks take inputs that have some spatial consistency, have some meaning to the spatial- Has some spatial meaning in them like images.

The assumption there is that a powerful feature for detecting a cat is just as important no matter where in the image it is. And this allows you to cut away a huge number of connections between neurons...

It's a fascinating trade-off, of transferring control from the human to the car. It's a transfer of trust and it's a chance for us to study the psychology of human beings as they relate to machines at >60 miles an hour.

The thing is, we don't understand automation, because we don't have the data: we don't have the data on the forward roadway video, we don't have the data on the driver and we just don't have that many cars on the road today that drive themselves.