Theano: A Symbolic Expression Compiler for Deep Learning and GPU Acceleration

The podcast introduces Theano, a mathematical symbolic expression compiler designed for defining and manipulating complex mathematical expressions, particularly useful in deep learning. It leverages NumPy syntax for ease of use and supports fundamental operations as well as larger blocks like neural network layers. A core capability of Theano is its ability to manipulate computation graphs, enabling automatic (symbolic) differentiation, applying optimizations for numerical stability, and compiling these optimized graphs for efficient computation on various hardware, including GPUs. The project, originating from LISA (ancestor of MILA) over eight years ago, has grown into a globally supported tool, driving research papers and industrial applications, and serving as a backend for higher-level deep learning libraries like Keras, Blocks, and Lasagne.

Using Theano involves defining symbolic variables with specified types and dimensions, which form the inputs of a computation graph. These inputs, along with shared variables (often used for model parameters that persist and update across function calls), are then used to build expressions representing the model's forward pass and cost function. A crucial feature is `theano.grad`, which automatically computes symbolic gradients through backpropagation, extending the computation graph to include these derivatives without explicitly forming Jacobian matrices. This symbolic representation allows for further manipulation, such as defining update rules for optimization algorithms like gradient descent, all within the same graph structure.

Once the entire computation graph, including forward pass, cost, and updates, is defined, `theano.function` compiles it into an executable Python object. During compilation, Theano performs extensive graph optimizations to enhance runtime speed and numerical stability. These optimizations include eliminating redundant computations, inserting in-place operations, fusing element-wise operations, and transferring computations to the GPU. Theano generates C++ or CUDA code on-the-fly for optimized execution, significantly boosting performance. Advanced features like `scan` enable the definition of loops within the graph for dynamic sequence models, and robust debugging tools assist in identifying issues in the symbolic graph definition.

The tutorial demonstrates Theano's application through practical examples, including logistic regression on the MNIST dataset and an introduction to convolutional neural networks (LeNet) and Long Short-Term Memory (LSTM) for character-level text generation. It highlights how Theano allows for building custom layers and models, providing a flexible foundation for deep learning research and development. The discussion also touches upon future developments, such as improved GPU operations, faster optimization, and enhanced data parallelism, underscoring Theano's continuous evolution as a powerful tool in the deep learning ecosystem.

Tiano is we can say mathematical symbolic expression compiler.

It makes it possible to define expressions that represent mathematical expression using numpy syntax.

It makes it possible to manipulate those expressions during rough substitutions cloning and replacement things like that and also making possible to go through that graph and perform things like automatic differentiation a symbolic differentiation actually.

Tiano has also been the base of other software projects that build on top of the nose so for instance blocks Kara's Lezyne our machine learning deep learning libraries that used ya know as a back-end.

To define the expression we start by defining inputs so the inputs are symbolic variables that have some type.

Almost all operations in Tiano implement a function that returns that and it actually returns not numbers not a numerical expression for that but it returns a symbolic expression that represents that computation.

What happens also during graph compilation is that the graph that we selected for that particular function gets optimized and what we mean by that is that it's going to be rewritten in parts.

Something else that's really crucial for speed and performance is GPU.

What we have for that in India know is called scan and basically it's one node that encapsulate another whole T&O function.

We initialize the weights using the formula from grow and venture at 2010.