Optimizing Neural Network Training: Super Convergence, Learning Rates, and Academic Publishing Challenges

The discussion centers on Leslie Smith's discovery of "super convergence," a method that dramatically accelerates the training of certain neural networks—up to ten times faster—by employing significantly higher learning rates than traditionally thought possible. This breakthrough not only speeds up the training process but also leads to better model generalization and accuracy, as fewer epochs are required, reducing the model's exposure to the training data. The core insight challenges conventional wisdom regarding optimal learning rate settings in deep learning.

A key distinction highlighted is the unique challenge Leslie Smith faced in publishing this finding within academia. Unlike fields such as physics, where unexplained experimental observations can be published and later theorized, deep learning academia often demands theoretical explanations alongside empirical results. This bias against "I don't know why" findings meant Smith's paper, despite its profound practical implications, remained largely unpublished, underscoring a significant hurdle in the dissemination of practical, experimental deep learning insights. The practical trick involves starting with a very low learning rate and gradually increasing it throughout the training process, taking increasingly larger steps.

Practically, the technique involves a nuanced approach to learning rate scheduling: beginning with a very low learning rate and progressively increasing it during training, eventually reaching much higher rates than typically considered feasible. The conversation also touches upon "discriminative learning rates," where different parts of a model are trained at varying rates, particularly beneficial for transfer learning. These practical insights offer direct methods for practitioners to significantly enhance the efficiency and performance of their deep learning models, moving beyond static, fixed learning rate strategies.

The broader implications of this discussion point to a paradigm shift in how deep learning models are optimized. The current understanding of optimizers, weight decay, and learning rates is still nascent, with many interactions remaining mysterious. However, recent advancements suggest a future where hyperparameter tuning, especially for learning rates, becomes increasingly automated, potentially disappearing as a manual adjustment. This evolution promises more robust and efficient AI development, but also calls for a re-evaluation of academic publishing standards to better accommodate and encourage the sharing of crucial experimental findings, even those without immediate theoretical explanations.

Leslie's a researcher who like us cares a lot about just the practicalities of training neural networks quickly and accurately which i think is what everybody should care about but almost nobody does

he discovered something very interesting which he calls super convergence which is there are certain networks that with certain settings of high parameters could suddenly be trained 10 times faster by using a 10 times higher learning rate

no one published that paper because it's not an area of kind of active research in the academic world no academics recognized this is important

deep learning in academia is not considered a experimental science

it's literally impossible for Leslie to publish a paper that says I've just seen something amazing happen this thing trained ten times faster than it should have I don't know why

I read a lot more unpublished papers and published papers because that's where you find the interesting insights

if you can train these things ten times faster they also generalize better because your your doing less epochs which means you look at the data less you get better accuracy

a big part of the trick is starting at a very low learning rate very gradually increasing it so as you're training your model you would take very small steps at the start and it gradually makes them bigger and bigger and troll eventually you're taking much bigger steps than anybody thought as possible