Deep Learning State of the Art (2019): Breakthroughs in NLP, Applied AI, and Reinforcement Learning

This podcast episode provides a comprehensive overview of the state of the art in deep learning as of 2019, highlighting significant breakthroughs and conceptual advancements from 2017 and 2018. The discussion emphasizes that while deep learning has achieved remarkable feats, it is still in its nascent stages, with much potential yet to be realized. The host deliberately avoids focusing on benchmark-specific algorithms, instead delving into fundamental ideas across various subfields. Key areas covered include natural language processing, applied deep learning in autonomous systems, automated machine learning, innovative data handling techniques, and advancements in generative models and reinforcement learning. A major focus is placed on Natural Language Processing (NLP), which is described as experiencing its "ImageNet moment" in 2018. The evolution from encoder-decoder recurrent neural networks to attention mechanisms, self-attention, and the transformative BERT model is detailed. BERT, with its richly bi-directional transformer architecture and masked language modeling, is presented as a significant leap, enabling highly contextualized word embeddings and superior performance across various NLP tasks like classification and question answering. Beyond theoretical advancements, the episode explores the practical impact of deep learning, exemplified by Tesla's Autopilot system, which leverages neural networks for real-time perception and control in consumer vehicles, accumulating over a billion miles of data and continuously improving through weekly updates. The discussion also delves into methods for democratizing and optimizing deep learning. Automated Machine Learning (AutoML) initiatives like Google AutoML and AdaNet are presented as efforts to automate architecture search and ensemble creation, making deep learning more accessible by reducing the need for manual parameter tuning. Furthermore, innovative approaches to data augmentation, such as AutoAugment, and the use of synthetic data by NVIDIA are highlighted as crucial for learning effectively from limited real-world data and reducing costly manual annotation processes. The DAWNBench competition is cited as a significant development, pushing for efficiency and affordability in deep learning training, with Fast.ai demonstrating remarkable results in achieving high accuracy with minimal time and cost. Finally, the episode touches upon advancements in generative adversarial networks (GANs), noting that 2018 was a year of scaling and parameter tuning rather than new conceptual breakthroughs, leading to incredibly high-resolution image generation. NVIDIA's work on video-to-video synthesis is also discussed, emphasizing the challenge and importance of temporal consistency. The segment on deep reinforcement learning highlights the progression from DQN's superhuman performance in Atari games to AlphaGo's mastery of Go, culminating in AlphaGo Zero's ability to learn complex strategies from scratch through self-play, beating world-class engines like StockFish in chess with minimal training, marking a pivotal moment in the pursuit of general intelligence.

"Here we stand in 2019 really at the height of some of the great accomplishments that have happened. But also stand at the beginning."

"I think 2018 in terms of deep learning is the year of natural language processing. Many have described this year as the ImageNet moment."

"BERT is the thing that did the big leap in performance."

"Tesla has a system called Autopilot where the hardware version 2 of that system is a newer implementation of the NVIDIA Drive PX 2 system which runs a ton of neural networks."

"The dream of automating some aspects or all aspects or many aspects as possible of the machine learning process where you can just drop in a dataset that you're working on and the system will automatically determine all the parameters..."

"There's an area of machine learning that's heavily under studied and I think it's extremely exciting area... on the data side there's been very few ideas about how to do data augmentation."

"I think if we look at the history of AI when you're sitting on a porch hundred years from now sort of reminiscing back Alpha Zero will be a thing that people will remember as an interesting moment in time, as a key moment in time."

"Fast AI... have been able to train on ImageNet in 3 hours. So this is for training process for 25 bucks."

"2018 for GANs was a year of scaling and parameter tuning as opposed to breakthrough new ideas."

"The key idea behind DeepLabv3 that is the state of the art is the multi-scale processing."