Jeremy Howard on Deep Learning Frameworks: PyTorch, TensorFlow, fast.ai, and the Future with Swift

This podcast clip features Jeremy Howard discussing the evolution and comparative strengths of various deep learning frameworks, including Theano, Keras, TensorFlow, PyTorch, and fast.ai, while also looking ahead to Swift for TensorFlow. He highlights the shift from frameworks that define a computational graph upfront, like early TensorFlow and Theano, to those offering interactive, Pythonic execution, with PyTorch being the strongest proponent. This interactive approach significantly improved the ability to conduct research, development, and teaching by allowing step-by-step debugging and interactive programming in environments like Jupyter notebooks, which was critical for rapid experimentation in benchmarks like Dawn Bench.

Howard then introduces fast.ai, a multi-layered API built on PyTorch, designed to address PyTorch's accessibility issues for newcomers and reduce boilerplate for researchers. fast.ai enables users to train state-of-the-art neural networks with just a few lines of code, while also providing deep-dive capabilities for those who need more control. This library has been instrumental for students and researchers, leading to wins in deep learning competitions and academic publications. However, he notes that Python itself still presents limitations, particularly for performance-critical tasks like recurrent neural networks, leading to exploration of alternatives like Swift.

He offers a strong critique of TensorFlow 2.0's Python implementation, particularly its "TF Eager" mode, which attempted to mimic PyTorch's interactive computation. Howard argues that TensorFlow's failure to retool its foundational runtime components resulted in a system that is often ten times slower than PyTorch for actual computation steps. He attributes this to TensorFlow's "horribly complex" and "disorganized" codebase, which has accumulated significant technical debt, making fundamental improvements extremely difficult without a complete rewrite.

Looking to the future, Howard expresses optimism for Swift for TensorFlow, seeing it as a potentially transformative layer built on MLIR and leveraging Swift's compiler strengths with LLVM. While acknowledging that Swift's data science community and numeric computing support are currently lacking, he believes the involvement of Chris Lattner (Swift's creator) at Google on Swift for TensorFlow could change this. For new students, he strongly recommends fast.ai and PyTorch, emphasizing that understanding foundational concepts is more important than mastering a specific library, as switching between frameworks becomes a matter of days once the basics are solid.

"Theano and TensorFlow were great but were much harder to teach and do research and development on because they define what's called a computational graph upfront."

"PyTorch was not the first... but was certainly the strongest entrant to come along and say let's not do it that way, let's just use normal Python and everything you know about in Python is just going to work."

"The problem with PyTorch is it's not at all accessible to newcomers because you have to like write your own training loop and manage the gradients and all this stuff."

"We ended up writing this very multi-layered API that at the top level you can train a state-of-the-art neural network in three lines of code."

"We're still limited though by Python and particularly this problem with things like recurrent neural Nets where you just can't change things unless you accept it going so slowly that it's impractical."

"TensorFlow 2.0 Python... they tried to replicate the bits that people were saying they like about PyTorch... but it's 10 times slower than PyTorch to actually do a step."

"When you actually look in the code as I do often I'm always just like oh god what were they thinking it's just it's pretty awful."

"Swift for TensorFlow can be a different beast altogether... it can basically be a layer on top of MLIR that takes advantage of you know all the great compiler stuff that Swift builds on with LLVM."

"Honestly, it doesn't much matter what library you learn because switching from China to MX net to TensorFlow to PyTorch is gonna be a couple of days work as long as you understand the foundations well."

"The Swift community has a total lack of appreciation and understanding of numeric computing so like they keep on making stupid decisions."