Foundations of Deep Learning: Feedforward Neural Networks, Backpropagation, and Training Techniques

Hugo Larochelle's presentation lays out the foundational concepts of deep learning, focusing on feedforward neural networks. He describes their architecture, explaining how an input vector, such as an image or text, is processed through a series of hidden layers to produce an output, primarily for classification tasks. The core components discussed include pre-activations, activation functions, and the role of weights and biases in performing linear transformations. A key emphasis is placed on the introduction of nonlinearity via activation functions, which is essential for enabling neural networks to capture and perform sophisticated classification functions.

The discussion delves into various activation functions for hidden layers, such as sigmoid, hyperbolic tangent (tanh), and the rectified linear activation function (ReLU), highlighting their distinct properties like saturation behavior and output ranges. For the output layer in classification problems, the softmax activation function is introduced as a mechanism to produce a multinomial probability distribution over potential classes. Larochelle draws a crucial distinction between the theoretical capacity of single-hidden-layer networks to approximate any continuous function (the Universal Approximation Theorem) and the practical challenge of effectively finding the optimal weights and biases through the training process.

The primary training methodology presented is empirical risk minimization, which is solved using stochastic gradient descent (SGD). Practical aspects of SGD are detailed, including the selection of appropriate loss functions (e.g., negative log-likelihood, often referred to as cross-entropy), the application of regularization techniques like L2 regularization (or weight decay), and strategies for parameter initialization. A significant portion of the presentation is dedicated to backpropagation, explaining its efficiency in computing gradients using the chain rule, propagating information from the output layer backward through the network. A critical practical insight shared is the "vanishing gradient" problem, where activation functions like sigmoid and tanh can cause gradients to become extremely small when units saturate, thereby impeding effective parameter updates. ReLU is highlighted as an alternative that helps mitigate this issue for positive pre-activations.

Larochelle also touches upon the conceptual representation of neural network computations as flow graphs, a fundamental concept underlying modern deep learning libraries for automatic differentiation. He briefly mentions advanced deep learning developments such as dropout, batch normalization, and unsupervised pre-training, which are vital for successfully training deeper neural networks. The presentation underscores that while the mathematical derivations of backpropagation can be intricate, contemporary software libraries abstract much of this complexity, allowing practitioners to concentrate on model architecture and hyperparameter tuning, thus broadening access to deep learning technologies.

a multi-layer feed forward neural network is a model that takes as input some vector x which I'm representing here with a different node for each of the dimensions in my input vector and then it eventually produces at its output layer a an output

This will introduce some nonlinearity in the model.

if we have a single hidden layer neural network it's been shown that with a linear output we can approximate any continuous function arbitrarily well as long as we have enough hidden units.

However, this result does not tell us how do we actually find the weights and the bias values such that I can represent a given function. It doesn't essentially tell us how do we train a neural network.

The one algorithm that you'll see constantly in deep learning is stoastic gradient descent.

The basic idea is that we'll compute gradients by exploiting the chain rule and we'll go from the top layer all the way to the bottom computing gradients for layers that are closer and closer to the input as we go and exploiting the chain rule to exploit or reuse previous computations we've made at upper layers to compute the gradients at the layers below.

whenever this gradient here, these partial derivatives come close to zero, that it means the gradient will not propagate well to the next layer, which means that you're not going to get a good gradient to update your parameters.

if my pre-activations are very negative or very positive, so if my unit is very saturated, then gradients will have a hard time propagating to the next layer.

you actually usually don't write down back prop. you just use all of these modules that you've implemented and it turns out there's a way of automat automatically differentiating your loss function and getting gradients for free in terms of effort in terms of programming effort with respect to your parameters.