Francesco Cremonesi’s Blog

Reading large datasets for training on Colab

2020-09-18T14:13:00+00:00

Large datasets on Colab

Google Colab provides a fantastic way for anyone to access a powerful GPU runtime on the cloud, especially tailored for exploring and training machine learning models. However, the issue still remains of making sure the training data that you need is available to your models within a reasonable latency. For small datasets, a common approach is to simply store your data on your local computer, and upload it to the Colab runtime everytime via the internet. This approach is not feasible when datasets become large: in our experience, it can take up to 6 hours to upload a <2 GB dataset to the Colab environment.

We need a faster way to access our training data, possibly by storing it already on the cloud to exploit better bandwidth. We consider three approaches:

GDrive: using Google Drive for storage, and mounting it in the Colab runtime;
GCSFuse: using Google Cloud Storage Buckets, and mounting them with gcsfuse;
GCS Manual: using Google Cloud Storage Buckets, and transferring the data with the storage api.

Main results

The results of this investigation can be summarized as follows:

the major bottleneck for the performance of reading the training dataset is the time required to transfer the files from their remote location to the Colab environment during the first epoch;
geographic location plays an important role in affecting the performance, while choosing a cpu, gpu or tpu runtime does not;
after the first epoch of training, caching the files locally on the Colab runtime significantly improves the performance, to the point that other factors (such as gpu transfers) may become the bottlenecks to training performance.

More specifically, in terms of the performance of the approaches considered here, we found that:

GCSFuse does not seem to aggressively exploit the read-only nature of ML workflows, meaning that subsequent reads after the first are almost as slow as the first one;
The GCS Manual approach offers the best performance, but requires payment of a low fee and that you refresh your Colab environment enough times to land in the geographic region that you need;
GDrive offers good performance for free, but can be a bit unstable when dealing with folders containing lots of files.

Performance analysis

When you create a new Colab runtime, it is attached to an hard drive that does not contain your data. Hence, to access the contents of your files, the data needs to be transferred to the Colab runtime, at least once for every file read. In the third approach described above, this transfer happens when we manually copy the files, but do not be fooled: it happens with the other approaches too, even if it is handled automatically under the hood by the software libraries.

In machine learning workflows, re-reading a file happens very often during training, for example at every epoch. Retransferring the data everytime you want to re-read a file would be costly, so a more efficient strategy is to cache the file contents on the Colab runtime filesystem. This is what’s implicitly happening in the third approach, which works well because you already know that those files have not been changed on the remote server. In the first and second approach you are instead relying on the library’s implementations to handle this cache optimization for you.

To measure the performance of reading files, we use read bandwidth in Megabytes per second. In practice, we measure the amount of MB read and the time it took to read them separately, and plot their ratio. We caution against taking these numbers as an absolute indication of real-world performance, although they are valid to have a ballpark idea. The reason is that other operations, such as opening the file, can contribute significant overhead (especially for small files).

Gcsfuse does not provide optimal re-read performance

We measured the speed of re-reading files for the three approaches described above. While the GCS Manual approach and Google Drive appear to exploit some sort of caching to improve the performance, we observed that gcsfuse’s speed was not improved in subsequent reads of the files. The Figure below shows the boxplots for the file read bandwidth, distinguishing the first transfer from the remote servers versus a second (and subsequent) reads. Note the different scales on the x-axis.

During the first read, all bandwidths are the same. The first read is likely bounded by the speed of transferring the file contents from their cloud locations over the internet. The second read is more than 1000x faster for GDrive and the Manual approach, while for GCSFuse the speed remains roughly the same.

In this benchmark, we did our best to ensure that file contents were not kept in memory between the first and second reads, to simulate the case where one would loop over the whole dataset for each epoch, effectively clearing the memory. It remains unclear why the second-read bandwidth of the GDrive approach is worse than the manual approach, but it might be possible that GDrive is performing some checks or contacting the remote server.

Geographic region affects performance

The closer the remote servers are to where the colab is physically running, the better we expect the performance to be. We measured the first-read bandwidth and found that geographic proximity can give a 2x-10x boost to performance. The Figure below shows the read bandwidth broken down by region where the Colab runtime is executing.

For the GCSFuse and GCS Manual approaches, performance was improved when the colab runtime was running in the Netherlands. This is consistent with the fact that my GCS Bucket is located in the EU Multiregion. The GDrive approach, on the other hand, has better performance when the Colab runtime is in the US. One can only assume that GDrive servers are located there.

Runtime type does not affect performance

As expected, we also confirm that runtime type does not affect first read bandwidth.

This confirms our understanding that the performance of reading files for the first time is bounded by the time required to transfer the file contents over the internet.

Implementation guide: storing and reading your data from the cloud

This section presents the code and some technical details for implementation.

GDrive: Google Drive for storage and mounting

I used the UI to upload my dataset to my personal Google Drive. Then, the code to mount your drive in the Colab runtime is relatively simple.

from google.colab import drive
drive.mount('/content/gdrive')

Note that calling drive.mount will prompt you for an authorization code, for which you manually need to click on a link. After authentication and successful mounting, you are able to access all the files on your Google Drive from the Colab runtime.

images = os.listdir('/content/gdrive/path/to/training/data')
with open(path, 'r') as f:
  content = f.readlines()

Google Drive can be a bit clumsy and unstable when dealing with folders containing many files. An efficient approach to uploading large datasets to Drive is to upload a zipped folder, and unzip it directly on the Drive. Sometimes, reading the files would fail with an OSError exception or a timeout. I found that listing the directories containing the training data and catching the exceptions in that moment with the following code can help in later on making the training process more stable.

retry = True
while retry:
  retry=False
  try:
    next(os.walk('/content/gdrive/path/to/training/data'))
  except StopIteration:
    print('Exception Raised. Retry')
    retry = True

GCSFuse: Cloud Storage and gcsfuse

You can use the Google Cloud Storage service to store your dataset. This is a viable option even if you’re not within a corporation or a well-funded research institution, as the prices are not so high. I’ve been storing about 10 GB of data for a couple of datasets, and paying less than 3 EUR per month. A simple way of accessing your data from Colab is then to mount the Storage Bucket on the runtime, using gcsfuse. First, you need to install the tool

!echo "deb http://packages.cloud.google.com/apt gcsfuse-bionic main" > /etc/apt/sources.list.d/gcsfuse.list
!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
!apt -qq update
!apt -qq install gcsfuse

Then you need to authenticate yourself and prepare the directory

from google.colab import auth
auth.authenticate_user()
import os
os.makedirs('/content/bucket-data')
os.chdir('/content')

Now you are ready to mount the drive

!gcsfuse --implicit-dirs my_bucket_name bucket-data

The --implicit-dirs option is really important, as it tells gcsfuse to recreate the directory structure from the Cloud Storage Bucket (very important if, for example, train and validation datasets are distinguished by being in different folders). Note that gcsfuse seems to have some caching options, but when I mounted the drive with the options below I found no significant performance difference

# finer control on caching
!gcsfuse --implicit-dirs --stat-cache-ttl 5h --type-cache-ttl 5h --stat-cache-capacity 65536 ml_datasets_checco_1 bucket-data

After a successful mount, it is again possible to open the files as usual

images = os.listdir('/content/bucket-data/my/path')
with open(path, 'r') as f:
  content = f.readlines()

GCS Manual: Cloud Storage and manual file copy

For performance reasons, we devised a third option in which we copy the files manually from the Storage Bucket using the provided API. Authentication and some preparation is again required

from google.colab import auth
auth.authenticate_user()
import os
if not os.path.exists('/content/gcs-api/my/data'): os.makedirs('/content/gcs-api/my/data')

Then the API can be used to retrieve the files

from google.cloud import storage
client = storage.Client(project='my_project_name')
for bucket in client.list_blobs('my_bucket_name'):
  bucket.download_to_filename('/content/gcs-api/my/data/this_file_name.ext')

As shown above, performance is affected by the region of your GCS Bucket as well as the region where the Colab runtime is executing. The former can be found out through the Google Cloud Console, while the latter can be found using

import requests
ipinfo = requests.get('http://ipinfo.io')
region = ipinfo.json()['country'] + ': ' + ipinfo.json()['region']

Code

The code for the benchmarks can be found here, while the analysis code is here.

We used python’s line profiler %lprun to extract timings for single lines of code. For GDrive and GCSFuse, the read benchmark looks like this:

N=50
images = os.listdir('/content/gdrive/path/to/data')
for i in range(N):
  filename = images[i]
  path = '/content/gdrive/My Drive/ML_data/snakes/valid/venomous/' + filename
  with open(path, 'rb') as f:
    f.seek(0,2)
    length_of_file = f.tell()
    f.seek(0,0)
    content = f.read(length_of_file)

To obtain the read bandwidth, we divided the total size of the files in MB by the sum of all the timings for the f.read function call, as measured by lprun.

For GCS Manual, the benchmark looks like this:

N=50
i=0
for b in client.list_blobs('gcs_bucket_name'):
  i += 1
  if i >=N:
    break
  filepath = '/content/gcs-api/'+ '/'.join(b.name.split('/')[-4:])
  b.download_to_filename(filepath)

Here we measured instead the time of the b.download_to_filename function. Technically this doesn’t include reading the file, but once it has been downloaded the file contents can be considered to be in memory, and reading them right away would happen at lightining speed compared to the cost of transferring them from the remote servers.

My first attempt at publishing a python package

2019-08-18T09:45:10+00:00

Lately I wanted to publish my first package to the PyPI repository, but I encountered some difficulties which I’d like to share here. The main problems I encountered are:

[SOLVED] how to specify a github repository as part of the dependencies for your package;
[SOLVED] how to test your package distribution;
[WORKAROUND] how to publish your package when you have specified a github repository as dependency.

The package: doi4bib

In the frantic last days before handing in my thesis, someone told me that reviewers really appreciate it when clickable DOI links are placed in the references. Getting this in the latex manuscript was as simple as using the doi package, but I quickly realized that the DOI information was missing from most of my references! So I quickly hacked together a utility that would read a .bib file and add the DOI information where needed.

This python tool can be found on my github page. It makes extensive use of other people’s work, namely the OpenAPC DOI Importer and aclement’s biblib library for parsing bib files.

Specifying dependencies

As I mentioned, my tool relies on two dependencies: the OpenAPC DOI Importer and aclement’s biblib library.

OpenAPC DOI Importer

Since the dependency in this case was for a single file from the repository, I simply copied that file and stripped parts of it that weren’t relevant to me. This may not be the best practice in general, but for such a simple situation it works quite well.

Aclement’s biblib library

In this case, I needed the specify the dependency on a full library. In the ideal case, there would be a python package my-dependency which can be specified in the setyp.py file as such:

# file setup.py
setup(...
    install_requires=['my-dependency',...]
    ···]

Unfortunately, this was not the case. Even more unfortunately, a conflict occurred:

a package with the same name but completely different API exists on PyPI.

Given the bad luck that always accompanies programmers who are starting out, the existing biblib package on PyPI did not work for me. I urge you to test it out for yourself, as it offers similar functionalities to what I describe here.

So the first problem I encountered is that I needed to explictly point to aclement’s repository as a dependency.

Specifying a github repo as a dependency

The basic idea is that you need to use the PEP-508 syntax. The thing that worked for me was to specify the dependency both in install_requires and dependency_links, as such:

# file setup.py
setup(...
    install_requires=[
        'biblib@git+https://github.com/aclements/biblib.git#egg=biblib-0.1.0',
        ...
    ],
    dependency_links=[
        'git+https://github.com/aclements/biblib.git#egg=biblib-0.1.0',
    ...
    ]

In my case, it was necessary to embed the github url in both install_requires and dependency_links because otherwise the created wheel did not have the knowledge of the url, and during a pip install would routinely fall back to the other existing package with the same name.

The syntax is

    <package name> @ git+<URL>#egg=<package name>-<version>

You can read the details in the “Dependencies that aren’t in PyPI” section of the setuptools docs.

Testing the distribution

Before publishing your package you probably want to test it. One nice service is provided by TestPyPI which allows you to try out the distribution tools and process without worrying about affecting the real index.

However, I often found that I could run python setup.py install locally without issue, but pip install from the testpypi repository would fail. So another quick test that allowed me to reproduce at least some of the issues I was having with pip was to create a clean virtual environment and pip-installing the newly created wheel directly. Thus my workflow was, before testing publication to TestPyPI:

# in package directory
python setup.py sdist bdist_wheel
virtualenv venv
source venv/bin/activate
pip install -e dist/doi4bib-0.1.10-py2.py3-none-any.whl

I found that the last pip install command often reproduced the errors I got when trying to install from TestPyPI.

Publishing the package

This is where I ran into a wall. I managed to publish several versions of the package to TestPyPI, but none of them could later on be installed correctly via pip install. The main problem was always that the incorrect biblib package was picked up by pip. Finally, as described above I managed to create a wheel with the correct url to the github repository. However, this has the side-effect of making the package impossible to upload to TestPyPI and PyPI. Attempts to upload fail with

HTTPError: 400 Client Error: Invalid value for requires_dist

For the moment, it seems that PyPI doesn’t accept direct references, period. Therefore, I opted for a simple, yet effective, option. For now, I will only publish my package directly through github link, using the now familiar PEP-508 syntax:

pip install -e git+https://github.com/sharkovsky/doi4bib@0.1.10#egg=doi4bib-0.1.10

A Simplified Model for Load Imbalance in Distributed Spiking Neural Networks

2018-03-20T09:45:10+00:00

Simulating networks of spiking neurons at large scale requires the implementation of some strategy to distribute neurons across different compute nodes. As the network evolves during the simulation some compute nodes may be required, at a given time instant, to process more spikes than other neurons, simply because of the chaotic activity of the network. This introduces a negative effect on simulation performance called load imbalance, by which the total wallclock time of the simulation is increased because some compute nodes must waste cycles waiting for others (who were tasked with a larger amount of computation) to finish.

The load balance problem has been tackled differently in the literature. The developers of NEST, one of the state-of-the-art spiking point-neuron simulators available open-source, have recently stated that “one needs to investigate how to map the spatial structure of a neuronal network to the topology of modern HPC systems. This is not an obvious design choice as the currently employed round-robin distribution scheme is crucial for load balancing. Any non-random distribution of neurons could incur significant performance penalties due to unbalanced work load.” As part of a software engineering effort, the group of Modha et al. have developed a closed-source simulator based on non-blocking communication that is able partially to mitigate the load balancing problem. Finally, a 2014 study from Gerstner’s lab found that synaptic plasticity can greatly impact simulation performance by introducing a significant load imbalance among parallel processes.

In this post, I will introduce a simple model that allows to get an idea of the potential load imbalance that may arise during a simulation under some simplified assumptions.

Results

Under reasonable assumptions, the load imbalance \(\Lambda = max~\{W_p\} - min~\{W_p\}\) arising from the different spiking workloads in a distributed spiking neural network simulation can be estimated by

\[Pr( \Lambda = k ) = \sum_{x=0}^\infty Pr( max~\{W_p\} = x, min~\{W_p\} = x - k ),\]

where \(W_p i.i.d.\) and

\[Pr( max~\{W_p\} = x, min~\{W_p\} = y ) = \begin{cases} 0, &x<y \\ [ F_W(y) - F_W(y-1)]^P, &x = y \\ [ F_W(y+1) - F_W(y-1)]^P - [ F_W(y+1) - F_W(y)]^P - [ F_W(y) - F_W(y-1)]^P, & x = y+1\\ [ F_W(x) - F_W(y-1)]^P + [ F_W(x-1) - F_W(y)]^P - [ F_W(x) - F_W(y)]^P - [ F_W(x-1) - F_W(y-1)]^P, &x > y + 1 \end{cases}\]

where \(F_W\) is the cdf of \(W_p\).

Application

This result was obtained with the application to the distributed simulation of spiking neural networks in mind. In this case \(W_p\) represents the number of spikes that a given parallel process must integrate in a network minimum delay period. According to the model defined below, \(W_p \sim Poiss( \frac{NK}{P}f\delta)\).

However, the problem is framed in the general setting of finding the distribution of the difference between the maximum and minimum of a collection of i.i.d variables.

Validation

I tested the formula above for \(P=16\) and different values of the Poisson parameter \(\mu\). The numerical experiments yield counts that are very similar to those obtained during the formula.

I have observed for \(\mu > 20\) results start being incorrect. ~~I suspect this may be due to roundoff errors.~~ This is due to the fact that I approximated the range of values for the sum in the definition of the pmf of \(\Lambda\). In the new version of the code I have called this variable range_for_convolution and set it to a much larger value. With a value of range_for_convolution=200., results appear correct at least for \(\mu \leq 100\). Obviously, this results in a much more computationally expensive process. In the future, it would be interesting to explore better ways of fixing this problem.

The code for generating this image is provided here.

Problem Definition

As is often done, I will simplify the spiking activity of a neuron and consider it as a poisson process with certain firing rate \(f\). Moreover, I will introduce the simplifying assumption that all neurons are independent from each other. Note that this is a very unrealistic assumption, however it is often done in literature and in any case one may assume that if the neurons are randomly distributed across compute nodes it may still be a valid approximation. Finally, I will assume for simplicity that all neurons have the same firing rate. Again, this is not a very realistic assumption. However this assumption can provide a best-case scenario where the load imbalance originates only from the chaotic and possibly stochastic nature of the spiking neural network, and if we introduce additional imbalance by, for example, having different firing rates than the resulting load imbalance could only get worse.

Let me introduce some notation:

\(N\) is the total number of neurons in the simulation;
\(P\) is the number of parallel distributed processes;
\(f\) is the firing rate of each neuron;
\(K\) is the number of synapses per neuron;
\(\delta\) is the interval of simulated time at the end of which we wish to assess the potential load imbalance.

The number of spikes that a neuron must process at a given instant can be modeled by assuming that each of the neuron’s synapses receives events independently from the others and according to a Poisson process as described above. In the case of multisynaptic connections or of small networks this may not be a realistic assumption, but in the case of a large, random network this may be a good approximation.

As a consequence of the assumptions above and assuming that neurons are randomly and evenly distributed across compute nodes, we have the following model.

The number of spikes to be processed by all the neurons pertaining to a given parallel process is distributed like a Poisson variable: \(W \sim Poiss(\frac{NK}{P}f\delta).\)

Moreover, since we want to estimate the load imbalance, we are interested in the difference between the parallel process with the maximum number of spikes to process and the one with the minimum.

We define the load imbalance as \(\Lambda = max~W - min~W\)

where the \(max\) and \(min\) are taken over the different parallel processes.

Probabilistic Setting

We can frame the problem in a general probabilistic setting as:

Given a set of \(P\) i.i.d. variables \(\{W_p\}\), what is the distribution of \(\Lambda = max~\{W_p\} - min~\{W_p\}\)?

This is an interesting problem for two reason: first it requires estimating the joint distribution of the maximum and minimum of a set of i.i.d. variables; second it requires estimating the distribution of the difference of two variables that are not independent.

Marginal Distributions of the Minimum and Maximum

Finding the distribution of the maximum (minimum) of a set of i.i.d. variables is relatively trivial. The idea is to estimate the cumulative distribution function (cdf) of the maximum (minimum) using an insight into the properties of these functions. For the maximum, we have that if the maximum should be smaller than a certain value \(x\), than all the variables in the set should be smaller than \(x\), which translates to \(Pr( max~\{W_p\} \leq x ) = Pr( W_0 \leq x \cap W_1 \leq x \cap \dots )\).

For the minimum, we use some transformations to obtain that \(Pr( min~\{W_p\} \leq y ) = 1 - Pr( W_0 > y \cap W_1 > y \cap \dots )\).

If \(F_W\) is the cdf of \(W\), the insights above lead to

\[\begin{align*} Pr( max~\{W_p\} \leq x ) &= F_W(x)^P, \\ Pr( min~\{W_p\} \leq y ) &= 1 - [ 1 - F_W(y) ]^P. \end{align*}\]

Joint Distribution of the Minimum and Maximum

Finding the joint distribution is not so trivial. This stack exchange answer provides the correct path to follow, but contains some typos. These stack exchange answers also provides some insight, but are applied only to special cases. For completeness and correctness, I will repeat the procedure here.

Given a collection of P i.i.d. variables \(\{W_p\}\), we seek the joint distribution of \(( max~\{W_p\}, min~\{W_p\})\).

By drawing a little diagram, you can convince yourself that

\[Pr( max~\{W_p\} \leq x, min~\{W_p\} \leq y ) = Pr( max~\{W_p\} \leq x ) - Pr( max~\{W_p\} \leq x, min~\{W_p\} > y ).\]

From the marginals above, we have that \(Pr( max~\{W_p\} \leq x ) = F_W(x)^P\). To estimate the other term, we need to use the same insight as for the marginals: assuming \(y < x\), if the minimum should be larger than \(y\) and the maximum should be less or equal to \(x\), than all the variables in \(\{W_p\}\) should respect this property.

\[\begin{align*} Pr( max~\{W_p\}\leq x, min~\{W_p\} > y ) &= Pr( W_0 > y \cap W_0 \cap \leq x \cap W_1 > y \cap W_1 \cap \leq x \cap \dots ) \\ &= [ F_W(x) - F_W(y) ]^P. \end{align*}\]

On the other hand, if \(y \geq x\), then \(Pr( max~\{W_p\}\leq x, min~\{W_p\} > y ) = 0\).

Therefore, we have that

\[Pr( max~\{W_p\} \leq x, min~\{W_p\} \leq y ) = \begin{cases} F_W(x)^P - [ F_W(x) - F_W(y) ]^P, & y < x \\ F_W(x)^P, & y \geq x. \end{cases}\]

Joint Probability Mass Function

The stack exchange answer cited above proceeds to compute the pmf directly by differentiation. Given that the variables of interest here are discrete and non-negative, I found that this approach did not work. Instead, I will use the following relationship that is valid for integer, non-negative variables:

\[\begin{align*} Pr(X=x,Y=y) &= Pr( x-1 < X \leq x, y-1 < Y \leq y) \\ &= F_{(X,Y)}(x,y) - F_{(X,Y)}(x-1,y) - F_{(X,Y)}(x,y-1) + F_{(X,Y)}(x-1,y-1) \end{align*}\]

After some computations and taking special care of the corner cases we have:

Distribution of the Difference

Finally we are ready to compute our quantity of interest. Here, special care must be taken to pick the correct formula for the difference of two non-independent variables. This formula is a special case of the invertible mappings formula (thank you Francesco Casalegno for pointing this out). According to the formula we have

\[Pr( max~\{W_p\} - min~\{W_p\} = k ) = \sum_{x=0}^\infty Pr( max~\{W_p\} = x, min~\{W_p\} = x - k )\]

Pratical Boltzmann Machines

2017-06-30T11:53:10+00:00

This post collects a couple of tricks that I found useful for implementing, training and debugging Boltzmann Machines. If you’re more interested in the theory, checkout my Boltzmann Machines (BMs) post.

In what follows, I have tried to collect as many doubts as possible among those that came up while implementing a BM (see ).

Everything is so slow

Yes. There doesn’t seem to be much you can do about that. Training BMs is a costly business, and is very slow. And after getting a residence permit in Switzerland, believe me, I know what slow means.

Neuron activations

On the internet I have seen some people use \({0,1}\) as activation values, and others use \({-1,1}\). From the little that I can tell, \({0,1}\) seem to be used more often in practice, whereas \({-1,1}\) appears often in the theoretical presentations. Moreover, \({0,1}\) has a biological interpretation, corresponding to whether the neuron is active/inactive during that stimulus presentation. However, \({-1,1}\) could somehow be related to excitatory/inhibitory neurons. For both approaches all the neuron dynamics, learning rules and so on remain the same, so in the end it shouldn’t make much of a difference. However, who knows if it makes a difference in terms of convergence speed?

Debugging your energy computation

You can reproduce the analytical example from slide 35.

initialize a Boltzmann machine with 2 visible and 2 hidden units;
initialize the weight matrix with

\[W = \begin{matrix} 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 1 \\ 2 & 0 & 0 & -1 \\ 0 & 1 & -1 & 0 \end{matrix}\]

and

\[\mathbf{b} = \mathbf{0}\]

reproduce the table in the slide (also in my BM blog post ), computing the energy for every combination of visible and hidden unit values.

See the code under the section “Debugging your energy computation” for a prototypical implementation.

Debugging your sampling/simulated annealing step

You can use the same analytical example from above. In this case, I suggest computing the conditioned probabilities ( e.g. given that \(\mathbf{h} = \left [ 1,1 \right ]^T\) ).

Looking at the relevant rows of the table from slide 35:

v1	v2	h1	h2	-Energy	exp( -Energy)	joint probability \(p(\mathbf{v}, \mathbf{h})\)
1	1	1	1	2	7.39	.186
1	0	1	1	1	2.72	.069
0	1	1	1	0	1	.025
0	0	1	1	-1	0.37	.009

allows us to compute the conditioned partition function

\[Z_{|[1,1]} = 7.39 + 2.72 + 1 + 0.37 = 11.48\]

and the conditioned probabilities

v1	v2	conditioned probability
1	1	7.39/11.48 = 0.643
1	0	2.72/11.48 = 0.237
0	1	1/11.48 = 0.087
0	0	0.37/11.48 = 0.032

Then you let your MCMC chain run to convergence (i.e. the BM is dreaming) many times, but always clamping the hidden state to \([1,1]\), and you compute a histogram of the outcomes.

In my code you’ll find an implementation of this idea under the section “Test sampling/dreaming procedure”, although I was super lazy and computed the probability of \(v1+v2\) (a scalar) instead of the vector \([v1,v2]\), so that I could use numpy.histogram easily.

A second way to do this is to randomly initialize the network at every time, and making sure that the joint probabilities respect the ones given in the table.

Finally, a third way is to randomly initialize the hidden state to a randomly chosen value, and checking whether the outcome of the dream step respect the marginal probabilities of \(\mathbf{v}\). To be honest, though, I’m not sure if this way is correct, because it sounds like you are making an assumption that the posterior probability of \(\mathbf{h}\) is uniform.

Simulated annealing schedule

Here again I have found some contradictory information on the internet. I have seen both the phrase

as the temperature becomes a very small value

and

as the temperature tends to one.

In the end, the interpretation that I settled on is the following: as the temperature tends to 0, Boltzmann Machines become a deterministic model very similar to Hopfield nets. I don’t think this is generally what you want, so I settled on the idea that the temperature should start at a value \(T_0 >> 1\) and tend to \(T_{end} = 1\).

Keep in mind that if your starting temperature is too large than the first few iterations will be essentially random. This can be useful if you have a large Boltzmann Machine for which the randomly chosen initial value could be very far the steady-state distribution, but can be annoying for small Boltzmann Machines.

Another concept that I struggled with for some time is where the simulated annealing is used: is it “outside” the training loop, or “inside”? In the end, I decided for the following interpretation: simulated annealing is used to help with the optimization process related to inference, and not the gradient descent on the parameters. Therefore, simulated annealing happens every time you do an inference step, i.e. every time you process a new data point or every iteration of the negative phase. In this interpretation, the simulated annealing happens inside the training loop, but outside the Gibbs sampling loop.

A few tips that I found useful are:

if your input data is rescaled between -1 and 1 and you have a small-medium BM, then a decent starting value for the temperature can be around 20;
temperature should decrease pretty slowly: I found that making jumps larger than 5 was making convergence harder;
running about 20-40 Gibbs sampling loops per temperature value turned out to be pretty good for me;
running at least double the number of Gibbs sampling loops at the final temperature, to make sure everything is stable, seemed to help.

Training

Training has been, in my experience, extremely slow. Learning rates larger than 0.1 seemed to pretty consistently end up exploding the model, and often I ended up using \(\eta = 0.01\) or even \(\eta = 0.005\).

In the code I knew the exact solution so I could compute the error. Convergence was extremely slow:

and what’s more, even though the error curve looks pretty promising, the model itself does not yet reproduce the probability distribution in a particularly satisfactory way:

condition	\(p(\mathbf{v} = [1,1])\)	\(p(\mathbf{v} = [1,0]~or~\mathbf{v} = [0,1] )\)	\(p(\mathbf{v} = [0,0])\)
analytical model \(W_{analytic}\)	0.15	0.48	0.37
setting \(W=W_{analytic}\)	0.16	0.47	0.37
random initial conditions	0.29	0.51	0.2
after convergence	0.16	0.5	0.34

Discussion

My experience with understanding and training BMs has been somewhat difficult. There is a lot of information on the internet for RBMs, but not so much original BMs. However, the latter are the more biologically realistic, which is ultimately what I was interested in.

I have reached the point where the lowering of my standards due to exhaustion has allowed the current status quo to be sufficient for me to feel satisfied. So overall, a good experience and a Goldmanesque metaphor for life.

Boltzmann Machines

2017-06-23T11:53:10+00:00

In this post belonging to a series dedicated to machine learning methods that are in some way closer to biological networks than your run-of-the-mill fully connected network I will present Boltzmann Machines (BMs). These neural networks are characterized by some properties that make them seem biologically plausible:

arbitrary recurrent connectivity;
inference is equivalent to energy minimization;
binary (and possibly stochastic) neuron dynamics;
unsupervised learning.

For more details on why BMs can be considered biologically plausible, see my other post. Obviously, we are far from understanding how the brain works, and these are all mathematical abstractions only loosely related to the physical/biological phenomenon.

The most prominent author of Boltzmann machine related papers is G. Hinton (see his practical guide ), but Y. Bengio, I. Goodfellow, A. Krizhevsky and others have made significant contributions. Moreover, although I had to wander all the way to the second page of Google’s results to find it, Gorayni’s blog post cleared many of my doubts about the actual implementation, and the MacKay book he mentions turned out to be a gem of clarity for the theoretical aspects.

Why Boltzmann machines?

Boltzmann machines are a generative model. This means that they can learn, in an unsupervised way, the probability distribution of your data \(p(data)\), or in a supervised way \(p(data | label)\). As always, Andrew Ng’s explanation is worth a thousand of mine.

Moreover, there are three ways (see chapter 16 ) in which one could use Boltzmann machines as a discriminative model instead: train a classifier on top of a subset of the neurons, train a different BM for each class, or train a single BM with two sets of input neurons.

Network topology

Boltzmann machines are characterized by arbitrary recurrent, symmetric connections. Usually, self connections are also disallowed.

The neurons (or units) are split in two non-overlapping groups: visible and hidden units.

This distinction is important because it allows the boltzmann machine to learn a latent representation of the data, so the BM can learn not only \(p(data)\) but also \(p(data|hidden~units)\). Moreover, hidden units allow the Boltzmann machine to model distributions over visible state vectors that cannot be modelled by direct pairwise interactions between the visible units (see section learning with hidden units). This distinction is also at the basis of Restricted Boltzmann Machines.

Inference as energy minimization

This is the idea that the inference process consists of fixing some part of the network and letting the rest evolve naturally to a state of lower energy. This contrasts with a more static view of neural networks, where the activation of each neuron is computed only once, statically, as a function of the input values. This point of view is not specific to BMs, actually is used to define a broad range of models under the umbrella term energy based models and in particular was used before in Hopfield networks. Y. Le Cun’s tutorial is a good starting point for understanding energy based models.

I still find it difficult to understand the relationship between inference and energy in an intuitive way. I will now try to explain the explanation that I’ve been giving to myself:

in a stochastic neural network, performing inference means letting the network evolve, possibly under some constraints, until a state that is in some sense stable is reached;
in this sense, inference is then equivalent to sampling from the probability distribution of possibile network states;
the relationship between the probability distribution of the network states and the energy has a simple formulation \(p(\mathbf{x}) = \frac{e^{ -E(\mathbf{x}) }}{Z}\);
given the formula above, states with lower energy have a higher probability.

Note that in this setting energies can have negative values.

Finally, the example on slide 35 is what really made the idea click for me. This exmaple is very useful to understand how connection weights define a probability distribution. Consider the following, very simple, Boltzmann machine with an 2-dimensional input vector \(\mathbf{v} = (v1,v2)\) and two hidden units.

Then we can compute the probability distribution \(p(\mathbf{v})\) induced by the weights by considering all possible combinations of inputs and hidden units, following this process:

compute the energy associated with every network configuration (formula in section below);
compute the partition function, i.e. the normalizing factor \(Z\), as the sum over all possible combinations \(Z = \sum\limits_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}\);
compute the probability associated with a particular network configuration \(( \mathbf{v_0}, \mathbf{h_0})\)as \(\frac{ e^{-E(\mathbf{v_0}, \mathbf{h_0})} }{Z}\);
the probability distribution of the input data is given by the total probability \(p(\mathbf{v}) = \sum\limits_{\mathbf{h}} p(\mathbf{v}, \mathbf{h})\).

And we can build a table exactly as in slide 35

v1	v2	h1	h2	-Energy	exp( -Energy)	joint probability \(p(\mathbf{v}, \mathbf{h})\)
1	1	1	1	2	7.39	.186
1	1	1	0	2	7.39	.186
1	1	0	1	1	2.72	.069
1	1	0	0	0	1	.025
1	0	1	1	1	2.72	.069
1	0	1	0	2	7.39	.186
1	0	0	1	0	1	.025
1	0	0	0	0	1	.025
0	1	1	1	0	1	.025
0	1	1	0	0	1	.025
0	1	0	1	1	2.72	.069
0	1	0	0	0	1	.025
0	0	1	1	-1	0.37	.009
0	0	1	0	0	1	.025
0	0	0	1	0	1	.025
0	0	0	0	0	1	.025

which allows us to compute the marginalized probabilities

\(\mathbf{v}\)	marginal probability \(p(\mathbf{v})\)	formula
(1,1)	.466	.186 + .186 + .069 + .025
(1,0)	.305	.069 + .186 + .025 + .025
(0,1)	.144	.025 + .025 + .069 + .025
(0,0)	.084	.009 + .025 + .025 + .025

Definition of energy

How did we compute the enrgy values in the table above? The formula is actually pretty simple: consider the symmetric connection weight matrix \(W\), biases \(\mathbf{b}\) and the state vector \(\mathbf{s} = (\mathbf{v}, \mathbf{h})\), then the (scalar) energy is given by:

\[E = -\frac{1}{2}\mathbf{s}^TW\mathbf{s} - \mathbf{b}^T\mathbf{s}.\]

N.B. vectors are column vectors here. To see this formula in action, consider the BM above. Then

\[W = \begin{matrix} 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 1 \\ 2 & 0 & 0 & -1 \\ 0 & 1 & -1 & 0 \end{matrix},\]

and the biases are all zero. The energy of configuration \(\mathbf{s} = ( 1, 0, 1, 1)\) would be \(E = -\frac{1}{2} ( 2w_{13} + 2w_{14} + 2w_{34}) = -2 - 0 - (-1) = -1\).

The neuron dynamics during inference

BM neurons can take only two values: 1 and 0. The crucial idea behind letting BMs states’ evolve is that it must be done in a sequential way, one neuron at a time. In theory, one can select a candidate transition for the \(i^{th}\) neuron, compute the difference in energy between the original and the transitioned state, and decide whether to reject or keep the transition based on the probability

\(p ( keep~transition) = \frac{1}{1 + e^{E(transition) - E(original)} }\).

Incidentally, this is what I do in my code.

Otherwise, the wikipedia page gives a nice derivation of the formula for how energy relates to the probability of the \(i^{th}\) neuron being on or off and which also explains where the formula for the above probability comes from. The gist is the following: we rely on the property of Boltzmann distributions that the energy of a state is proportional to the negative log probability of that state:

\(E( \mathbf{s}_i = 0 ) - E( \mathbf{s}_i = 1 ) = ln( p( \mathbf{s}_i = 1 ) ) - ln( p( \mathbf{s}_i = 0 ) )\),

and by doing some algebra we get to

\[p( \mathbf{s}_i = 1 ) = \frac{1}{1 + e^{ E( \mathbf{s}_i = 1 ) - E( \mathbf{s}_i = 0 ) } } .\]

So far we have stated the formulas that allow to decide whether the state transition of a single neuron should happen, so how does this fit in the global picture?

Enter MCMC

No, MCMC is not a rapper suffering from Palilalia.

Markov Chain Monte Carlo (MCMC) methods, and in particular Gibbs sampling, are the answer to the above problem. The idea is the following: the overall state of the network’s dynamics follow a Markov Chain, where each state transition’s probability is defined as above; to compute the dynamics of the network, we need to run this Markov Chain to convergence. The concept of convergence here is defined mystically as the moment when the probability distribution of the states does not depend anymore on the initial value, but is only a function of the energy of each state. In practice, I have not found a single resource explaining how to understand whether your network has converged or not, and in a later section I’ll explain how I did it for a very very very simple case.

To run the Markov Chain to convergence, we use the MCMC method which consists in repeatedly sampling from the distribution of possible states. In particular, we repeatedly use Gibbs sampling:

until convergence, do:
- for each neuron in the network, do:
  - compute transition probability for this neuron, keeping the rest fixed
  - randomly decide to keep or reject transition according to probability

As we said before, given the mystical definition of convergence, we often run the above loop for a fixed number of iterations. As a cheaper alternative, we can resort to simulated annealing instead.

Simulated annealing

Running the MCMC to convergence everytime can be very resource consuming. To mitigate this problem, we adopt an optimization technique called simulated annealing.

Simulated annealing introduces the concept of temperature \(T\), a value which modifies the transition probability to:

\[p ( keep~transition) = \frac{1}{1 + e^{ \frac{ E(transition) - E(original)}{T}} }.\]

This is how the the Markov Chain loop is modified in the presence of simulated annealing:

given an annealing schedule
for each (temperature value \(T\), number of iterations \(N\)) tuple in the annealing schedule, do:
- repeat the full Gibbs sampling procedure \(N\) times, at a temperature value of \(T\).

Training… Finally!

If inference means energy minimization, then we want to find a training procedure that will arrange the connection weights such that the observed data have very low energy. An equivalent way to put it is to say that we want to find weights and biases that define a Boltzmann distribution in which the training vectors have high probability. Following the procedure described on the scholarpedia page, assuming the training data were sampled i.i.d. we can write a gradient ascent formula for the log likelihood as

\[\Delta W = \sum\limits_{\mathbf{v}} ln \left ( \frac{ d p(\mathbf{v},\mathbf{h}) }{dW} \right ).\]

To compute the expression above we need to define two quantities:

\(\mathbf{s}^{\mathbf{v}}\) is a sample of the network state vector that was obtained by running the MCMC procedure to convergence, while keeping the input vector \(\mathbf{v}\) fixed;
\(\mathbf{s}^m\) is a sample of the network state vector that was obtained by running the MCMC procedure to convergence having left all units unclamped.

Given that we know the relationship between the probability \(p(\mathbf{v},\mathbf{h})\) and the energy, i.e. \(p(\mathbf{v},\mathbf{h}) = \frac{ e^{-E(\mathbf{v})}}{\sum\limits_{\mathbf{u},\mathbf{g}} e^{-E(\mathbf{u},\mathbf{g})} }\) we can write:

\[\Delta W = \sum\limits_{\mathbf{h}} p(\mathbf{h} | \mathbf{v} )\mathbf{s}^{\mathbf{v}}(\mathbf{s}^{\mathbf{v}})^T - \sum\limits_{\mathbf{h},\mathbf{v}} p(\mathbf{v}, \mathbf{h} )\mathbf{s}^m(\mathbf{s}^m)^T .\]

Computing the exact quantities above requires summing over many many possible configurations. In practice this can rapidly become unfeasible, so we substitute the above quantity with:

\[\Delta W = \frac{1}{N_{\mathbf{v}}}\sum\limits_{\mathbf{v}} \mathbf{s}^{\mathbf{v}}(\mathbf{s}^{\mathbf{v}})^T -\frac{1}{N_m} \sum \mathbf{s}^m(\mathbf{s}^m)^T,\]

which corresponds to the following procedure (with a learning rate \(\eta\)):

initialize updates \(\Delta W = 0, \Delta \mathbf{b} = 0\);
for every training data point, do:
- keeping the visible units clamped to the values of the training data point, run MCMC to convergence;
- update \(\Delta W = \Delta W + \mathbf{s}\mathbf{s}^T, \Delta \mathbf{b} = \Delta \mathbf{b} + \mathbf{s}\);
update \(W = W+ \frac{\eta}{N_{\mathbf{v}}} \Delta W, \mathbf{b} = \mathbf{b} + \frac{\eta}{N_{\mathbf{v}}} \Delta \mathbf{b}\);
reset updates \(\Delta W = 0, \Delta \mathbf{b} = 0\);
for a certain number iterations \(N_m\), do:
- allow the MCMCM to run to convergence, without clamping anything;
- update \(\Delta W = \Delta W + \mathbf{s}\mathbf{s}^T, \Delta \mathbf{b} = \Delta \mathbf{b} + \mathbf{s}\);
update \(W = W - \frac{\eta}{N_m} \Delta W, \mathbf{b} = \mathbf{b} - \frac{\eta}{N_m} \Delta \mathbf{b}\);

A Summary of Biologically Plausible Deep Learning and Equilibrium Propagation

2017-06-20T11:53:10+00:00

In the past couple of years, Y. Bengio has been interested in the question of whether and how could backpropagation be implemented in a biologically-realistic framework. I find his work extremely interesting, and have decided to summarize my own personal understanding of it in this blog post.

In particular, I will try to combine information from these three papers:

[1] Towards Biologically Plausible Deep Learning (v3 on ArXiv) ;
[2] Early Inference in Energy-Based Models Approximates Back-Propagation (v2 on ArXiv);
[3] Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation (v5 on ArXiv).

The problem of credit assignment

One of the main motivations for Bengio’s work (see [1]) is finding a

credible machine learning interpretation of the learning rules that exist in biological neurons […] accounting for credit assignment through a long chain of neural connections.

In other words, answering the question of how does the brain propagate an error?

There are many aspects to this question, all of which represent interesting unsolved problems:

how does the brain propagate the signal that a certain prediction was wrong, through its many hidden layers and recurrent connections?
how does the brain propagate the signal of the magnitude of a given error?
how does the brain interpret that signal and produce a valid update of its synaptic weights?

Now is a good time to make an important distinction: obviously there are several physiological implementations of all the mechanisms cited above (e.g. NDMA recruitment in Long Term synaptic Potentiation), and studying these biological mechanisms is an extremely important aspect of neuroscience. However, in this work we are going to be taking a more abstract view, and asking ourselves the question of what are the relevant computational (mathematical) aspects related to the questions above?

Biologically plausible, whatever do you mean?

Well, this is where the boundaries of objectivity start to become blurred like the mists of Avalon. There are many many properties observed experimentally in biological neural networks, and ideally a biologically-plausible artificial neural network would implement them all. Since this is proving to be very hard, researchers must make decisions on which experimentally observed properties they’d like to preserve in their models. Enter subjectivity.

Reasons why backprop is not biologically plausible

The paper [1] cites 6 reasons why backprop is not biologically plausible:

backpropagation is purely linear (I’m not sure in what sense);
if backpropagation were a thing (biologically), the feedback paths would require exact knowledge of the derivatives of the activation functions of neurons, and to this time this has never been observed experimentally;
similary, feedback paths would be required to have the same exact weights (but transposed) of the feedforward paths, a.k.a the weight transport problem;
backpropagation is based on the idea that communication between neurons happens via (smooth) continuous values, but experimental evidence shows that neurons communicate via binary discontinuous events (spikes) although there is still a lot of discussion on how the brain interprets these events;
backpropagation happens in two phases (feedforward and backprop), which would require the brain to have a perfectly synchronized clock that alternates between the two;
there is no widely-accepted theory of what the output targets (i.e. cost function) of the brain should be, nor what its biological substrate would look like.

Boltzmann machines as a biologically plausible model

In Bengio’s point of view, Boltzmann machines represent the most biologically plausible learning algorithm for deep architectures. I will try to summarize what I think can be some explanations for this:

Boltzmann machines are based on (stochastic) binary units, so they address concern number 4 above. These units are often divided in visible and hidden units, and in the case of Restricted Boltzmann Machines this division gives rise to two separate layers without inter-layer connections.
Boltzmann machines do not implement backprop but an unsupervised learning algorithm, which addresses concern 6 above. Indeed, Boltzmann machines are a generative model in the sense that they learn a probability distribution of the input dataset conditioned on some internal representation. In order to use BMs as a classifier, one needs to additionally train a classifier on top of the hidden units, an approach that is reminiscent of reservoir computing. In this case, however, the concerns raised in point 6 are still valid, as it is not clear what the output targets of the braing would be.
Training Boltzmann machines does not require explicitly computing any derivative, which addresses point 2.
I’m not sure how point 1 above should be interpreted, but Bengio’s work implicitly assumes that it is also addressed by Boltzmann machines.

Unfortunately, two issues remain with the biological plausibility of BMs: the weight transport problem (point 3) and the two-phase training (point 5).

Equilibrium propagation: an even more biologically plausible model

Bengio proposes in [2] a new learning framework that addresses the issue of two-phase learning. Bengio’s framework doesn’t completely remove the second phase, but still addresses the issue in the sense that only one kind of neural computation is required for both phases, something that is not true for standard Boltzmann machines.

The main steps of the Equilibrium Propagation algorithms are:

inference is energy minimization, just like in Boltzmann machines, although the actual formual used for the energy is different;
we distinguish three subsets of the units in the network, namely input, hidden and output units;
a total energy is defined as the sum of an internal energy the models the interactions within the network and an external energy (modulated by a constant) that models how the targets influence the output units;
learning happens in two phases: the free phase where all the neurons are performing inference, and the weakly clamped phase where the output units are nudged towards an (observed) target;
the input units are always clamped, the hidden units are always free, the output units are free during the first phase and weakly clamped during the second phase;
weights are updated according to a rule with a formulation similar to classical Boltzmann machines, but which is shown [2,3] to approximate gradient descent on the least squares error.

There are some additional compelling arguments for the biological plausibility of Equilibrium Propagation:

although the two-phases are not completely removed, the fact that they require the same kind of neural computation means that they are more open to biologically-plausible interpretations, for example
- when you move your arm you make a prediction of where it is going to be, and your eyes allow you to see if the prediction is correct;
- you see an image and you are trying to understand what it contains, and after a while a “teacher” tells you its contents;
Bengio et al. show in [1,3] that their learning rule can be interpreted as the classic STDP learning rule;

A naive explanation of backprop through time for recurrent neural networks

2017-06-12T11:53:10+00:00

In the second blog post of this series, I would like to repeat the same detailed computations for backpropagating the error in the case of recurrent neural networks.

As I mentioned before there are many other explanations of backprop out there, much better than mine. In addition to M. Nielsen’s book and the deep learning book, I’d like to mention a few other blogs that helped me understand things specifically in the case of recurrent networks. A. Karpathy’s unreasonable effectiveness of recurrent neural networks is one of the most exciting resources on RNNs out there. Chen’s arXiv paper is a nice, academic style, introduction to the topic, Lipton’s review paper gives a nice overview of the challenges related to training recurrent networks, while H. Jaeger’s tutorial is a mathematically intense but complete presentation which then moves on to talk about reservoir computing. Finally, I took the idea for the training set from P. Roelants’ blog post.

What is a recurrent neural network?

Recurrent Neural Networks (RNN) are special networks in which the concept of time is explicitly modeled. Originally, they were developed to deal with input data in the form of sequences, where each timestep corresponds to the processing of one element of the input sequence. The main difference with feedforward networks is that:

in RNNs, each neuron is characterized by a state (a variable) whose value can change according to some rules, but whose presence is persistent through time.

In practice this means that we need an uglier notation to represent neurons, as shown in this table:

Network Type	Notation	Description
feedforward	\(h^l_i\)	hidden state of the \(i^{th}\) neuron in the \(l^{th}\) layer
feedforward	\(\mathbf{h}^l\)	hidden state of the vector of neurons in the \(l^{th}\) layer
recurrent	\(h^l_{n,i}\)	hidden state of the \(i^{th}\) neuron in the \(l^{th}\) layer at timestep \(n\)
recurrent	\(\mathbf{h}^l_n\)	hidden state of the vector of neurons in the \(l^{th}\) layer at timestep \(n\)

Typically we represent RNNs like this:

If you compare this to the representation of a feedforward network you’ll notice the recurrent arrows connecting a neuron to itself. These arrows can be misleading because one may be tempted to think that a neuron’s state is immediately affected by the recurrent connections. Instead, what happens is that

recurrent connections affect the state of a neuron through some delay (typically, a single timestep).

The layer-wise point of view

As we saw in the post on feedforward networks, it is often useful to take the layer-wise point of view by considering the full vector of neurons in each layer. First, consider the visualization of a feedforward network which I have stretched out using my mad graphic skillz in order to give the 3D impression. Here, I am going to use a nomenclature that no-one else uses, but I promise it’s only for a minor thing: I am going to refer to the different layers of a network as the space dimension of the network. The reason behind this is simply to contrast it to the time dimension of recurrent neural networks (a nomenclature that everybody likes).

In the image, each ball is a neuron and each rectangle is a layer. I have drawn all the connections, but only higlighted those originating from a specific neuron. Keep an eye on that neuron for later.

Now compare this to the image of a recurrent neural network, which has both a space and a time dimension.

In the image, the same neural network from before is now repeated through time, in the sense that the state of each neuron is is computed ad each timestep. The highlighted neuron from the previous image now not only sends connections to the next layer, but also sends connections within the same layer, to the next timestep. This image would have been a mess had I drawn every single recurrent connection, so as an example I concentrated on the red neuron and drew all the incoming connections (from the previous layer at the same timestep, and from the same layer at the previous timestep) and all the outgoing connections (to the next layer at the same timestep, and to the same layer at the next timestep). With your third eye, try to imagine that every recurrent layer sends connections both to its downstream layer in space, and to itself at the next timestep.

In terms of notation, we need to distinguish between the weight matrix of recurrent connections, and the weight matrix of feedforward connections. I am going to use a notation that is nonstandard, so let me apologize now. Sorry. I am going to denote the weight matrix of recurrent connections with an \(T\) left-subscript. The \(T\) stands for the time dimension. Similary, the feedforward weight matrix will be denoted with an \(S\) left-subscript, where \(S\) stands for (you guessed it) space. See how I niftly combined all the nonstandard notations in one mess? Here is a summary for you:

Weights	Notation	Alternative Notations
feedforward from layer \(l-1\) to layer \(l\)	\({}_SW^l\)	\(W^l, W^l_{xh}\)
recurrent layer \(l\)	\({}_TW^l\)	\(W^l_hh\)

Obviously, it is possible to mix non-recurrent and recurrent layers, and in this case only the recurrent layers would behave as I have been describing so far. Also, it is possible to prescribe recurrent layers that do not connect to the future selves, but connect to future other layers.

Formulas are not just for babies

If you think a formula can be more expressive than a thousand words, here is the algorithm to perform inference on a recurrent layer \(l\)

\[\begin{align*} &\text{for every timestep } n\\ &\mathbf{z}^l_n = {}_SW^l\mathbf{h}^{l-1}_n + {}_TW^l\mathbf{h}^l_{n-1} \\ &\mathbf{h}^l_n = \sigma ( \mathbf{z}^l_n ) \end{align*}\]

BPTT: Backpropagation Through Time

How do we apply the principles of backpropagation in a recurrent neural network? The principle is the same as for a feedforward network, and indeed many sources claim that training recurrent networks is just like training feedforward networks with unrolling. I found this statement quite obscure, and preferred writing out the formulas explicitly.

Simple example

As we did for feedforward nets, let’s start by considering the very simple example of a scalar network.

In this uber-simple network, the output \(y\) is simply the hidden state of the neuron in the last layer.

Everyone has to start somewhere, so suppose we initialize the hidden states with some arbitrary initial values \(h^1_0, h^2_0\). At the first timestep, the networks receives the first element of the sequence \(x_1\), and processes it by:

\[\begin{align*} h^1_1 &= \sigma \left ( {}_Sw^1x_1 + {}_Tw^1h^1_0 \right ) \\ h^2_1 &= \sigma \left ( {}_Sw^2h^1_1 + {}_Tw^2h^2_0 \right ) \\ y_1 &= h^2_1. \end{align*}\]

At the next timestep, we have:

\[\begin{align*} h^1_2 &= \sigma \left ( {}_Sw^1x_2 + {}_Tw^1h^1_1 \right ) \\ h^2_2 &= \sigma \left ( {}_Sw^2h^1_2 + {}_Tw^2h^2_1 \right ) \\ y_2 &= h^2_2, \end{align*}\]

which by the magic of substitution is nothing other than

\[\begin{align*} h^1_2 &= \sigma \left ( {}_Sw^1x_2 + {}_Tw^1 \sigma \left ( {}_Sw^1x_1 + {}_Tw^1h^1_0 \right ) \right ) \\ h^2_2 &= \sigma \left ( {}_Sw^2\sigma h^1_2 + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_1 + {}_Tw^2h^2_0 \right ) \right ) \\ y_2 &= h^2_2, \end{align*}\]

where I have substituted the expressions of \(h^1_1, h^2_1\). Note that I could have also substituted \(h^1_2\), but I didn’t because the process for backpropagating the error through the layers is not in the scope of this post.

If we kept going for \(N\) timesteps, the formulas would look like this

\[\begin{align*} h^1_N &= \sigma \left ( {}_Sw^1x_N + {}_Tw^1 \sigma \left ( {}_Sw^1x_{N-1} + {}_Tw^1 \sigma \left ( {}_Sw^1x_{N-2} + \dots {}_Tw^1 h^1_0 \right ) \right ) \right )\\ h^2_N &= \sigma \left ( {}_Sw^2h^1_N + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right ) \right ) \right )\\ y_N &= h^2_N, \end{align*}\]

If you feel like this is too familiar for comfort, don’t worry you don’t suffer from Fregoli delusion. The same exact recursive substitutions happen if we move thorugh layers and not through time. As a matter of fact, we almost already know how to backprop through time thanks to our knowledge of backprop through layers!

Gradients galore

Let’s compute some derivatives. Suppose we have a cost function \(C = C (y)\) that we want to minimize. Question: how does the recurrent weight of the last layer influence the cost function at time \(N\)?

\[\begin{align*} \frac{dC(y_N)}{d{}_Tw^2} &= \frac{dC}{dy} \Big |_{y_N} \frac{ dy_N}{d{}_Tw^2} \\ &= \frac{dC}{dy} \Big |_{y_N} \frac{d\sigma}{dz} \Big |_{z^2_N} \frac{d z^2_N }{d{}_Tw^2}, \end{align*}\]

to compute the last term \(\frac{d z^2_N }{d{}_Tw^2}\) we have to calculate the derivative of the long expression

\[\frac{d \left [ {}_Sw^2h^1_N + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right )\right ]} {d{}_Tw^2}.\]

Fortunately, \(h^1_N\) does not depend on \({}_Tw^2\), nor does \(h^1_{N-1}\) and so on. Therefore we can already simplify

\[\begin{align*} &\frac{d \left [ {}_Sw^2h^1_N + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right )\right ]} {d{}_Tw^2} =\\ &\frac{d \left [ {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right )\right ]} {d{}_Tw^2}. \end{align*}\]

The annoying bit here is that we are facing the derivative of a product: both \({}_Tw^2\) and \(\sigma ( \dots )\) depend on \({}_Tw^2\)! So we must write

\[\begin{align*} &\frac{d \left [ {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right )\right ]} {d{}_Tw^2} = \\ & \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right ) \\ & + {}_Tw^2 \frac{d \left [ \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right )\right ]}{d{}_Tw^2}. \end{align*}\]

The expression that remains under the derivative sign here is the same as before, only one timestep in the past. That’s good news. We can recursively simplify the expression for the derivative into:

\[\begin{align*} \frac{dC(y_N)}{d{}_Tw^2} &= \frac{dC}{dy} \Big |_{y_N} \frac{d\sigma}{dz} \Big |_{z^2_N} \\ & \Bigg ( \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right )\right ) \\ & + {}_Tw^2 \frac{d\sigma}{dz} \Big |_{z^2_{N-1}} \Big ( \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right ) + \dots + \frac{d\sigma}{dz} \Big |_{z^2_{1}} h^2_0 \Big ) \Bigg ). \end{align*}\]

Writing the full expression for what’s inside the calls to the \(\sigma\) function was useful to understand exactly who depends on what in order to compute the derivatives. To make the notation lighter, but also to gain insight on backprop, let’s substitute back the directly the values of the hidden states. We obtain the recursive formulation for the derivative:

\[\frac{dC(y_N)}{d{}_Tw^2} = \frac{dC}{dy} \Big |_{y_N} \frac{d\sigma}{dz} \Big |_{z^2_N} \left ( h^2_N + {}_Tw^2 \frac{d\sigma}{dz} \Big |_{z^2_{N-1}} \left ( h^2_{N-1} + \dots + \frac{d\sigma}{dz} \Big |_{z^2_{1}} h^2_0 \right ) \right ).\]

The question now is, how does this extend to the upstream layers? We start off pretty easily:

\[\begin{align*} \frac{dC(y_N)}{d{}_Tw^1} &= \frac{dC}{dy} \Big |_{y_N} \frac{ dy_N}{d{}_Tw^1} \\ &= \frac{dC}{dy} \Big |_{y_N} \frac{d\sigma}{dz} \Big |_{z^2_N} \frac{d z^2_N }{d{}_Tw^1}, \end{align*}\]

and we have to now compute the annoying derivative

\[\frac{d z^2_N }{d{}_Tw^1} = \frac{ d \left [ {}_Sw^2h^1_N + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right ) \right ) \right ]}{d{}_Tw^1}.\]

The caveat is that, in contrast to what happened above, many terms in this expression depend on \({}_Tw^1\), namely \(h^1_N, h^1_{N-1}, h^1_{N-2}, \dots\). We continue with our computations, trying not to make transcription mistakes:

\[\begin{align*} &\frac{ d \left [ {}_Sw^2h^1_N + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right ) \right ) \right ]}{d{}_Tw^1} \\ &= {}_Sw^2 \frac{d h^1_N }{d{}_Tw^1} + {}_Tw^2 \frac{ d \left [ \sigma \left ( {}_Sw^2h^1_{N-1} + {}_Tw^2 \sigma \left ( {}_Sw^2h^1_{N-2} + \dots {}_Tw^2 h^2_0 \right ) \right ) \right ]}{d{}_Tw^1} \end{align*}\]

As before, we have a nice recursive property: the expression that remains under the second derivative sign is simply the same as the one we wanted to compute, but one timestep in the past. Before we move on to the fully recursive formulation, we need an expression for \(\frac{d h^1_N }{d{}_Tw^1}\). As before, \({}_Sw^1x_N\) doesn’t depend on \({}_Tw^1\), and for the second term we have to compute the derivative of a product:

\[\begin{align*} \frac{d h^1_N }{d{}_Tw^1} &= \frac{d \left [ \sigma \left ( {}_Sw^1x_N + {}_Tw^1 \sigma \left ( {}_Sw^1x_{N-1} + {}_Tw^1 \sigma \left ( {}_Sw^1x_{N-2} + \dots {}_Tw^1 h^1_0 \right ) \right ) \right ) \right ]}{d{}_Tw^1} \\ &= \frac{d\sigma}{dz} \Big |_{z^1_N} \left ( \sigma \left ( {}_Sw^1x_{N-1} + {}_Tw^1 \sigma \left ( {}_Sw^1x_{N-2} + \dots {}_Tw^1 h^1_0 \right ) \right ) + {}_Tw^1 \frac{d \left [ \sigma \left ( {}_Sw^1x_{N-1} + {}_Tw^1 \sigma \left ( {}_Sw^1x_{N-2} + \dots {}_Tw^1 h^1_0 \right ) \right ) \right ]}{d{}_Tw^1} \right ) . \end{align*}\]

And getting the recursive formulation of the derivative:

\[\begin{align*} \frac{dC(y_N)}{d{}_Tw^1} &= \frac{dC}{dy} \Big |_{y_N} \frac{d\sigma}{dz} \Big |_{z^2_N} \left \{ {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_N} \left ( h^1_{N-1} + {}_Tw^1 \frac{d h^1_{N-1} }{d{}_Tw^1} \right ) + {}_Tw^2 \frac{ d h^2_{N-1} }{d{}_Tw^1} \right \} \\ &= \frac{dC}{dy} \Big |_{y_N} \frac{d\sigma}{dz} \Big |_{z^2_N} \Bigg \{ {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_N} \left ( h^1_{N-1} + {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-1}} \left ( h^1_{N-2} + {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-2}} \left ( h^1_{N-3} + \dots \right ) \right ) \right ) \\ & + {}_Tw^2 \frac{d\sigma}{dz} \Big |_{z^2_{N-1}} \Bigg [ {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_{N-1}} \left ( h^1_{N-2} + {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-2}} \left ( h^1_{N-3} + {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-3}} \left ( h^1_{N-4} + \dots \right ) \right ) \right ) \\ & + {}_Tw^2 \frac{d\sigma}{dz} \Big |_{z^2_{N-2}} \left ( \dots + {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{1}} h^1_0 \right ) \dots \Bigg] \Bigg \}. \end{align*}\]

That’s a pretty ugly-looking expression. Lets’s try to group together terms by factors of \(h^1_{*}\):

\[\begin{align*} \frac{dC(y_N)}{d{}_Tw^1} &= \frac{dC}{dy} \Big |_{y_N} \Bigg \{ \\ & \frac{d\sigma}{dz} \Big |_{z^2_N} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_N} h^1_{N-1}\\ & + \left (\frac{d\sigma}{dz} \Big |_{z^2_N} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_N} {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-1}} + \frac{d\sigma}{dz} \Big |_{z^2_N} {}_Tw^2 \frac{d\sigma}{dz} \Big |_{z^2_{N-1}} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_{N-1}} \right ) h^1_{N-2} \\ & + \left (\frac{d\sigma}{dz} \Big |_{z^2_N} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_N} {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-1}} {}_Tw^1 \frac{d\sigma}{dz} \Big |_{z^1_{N-2}} + \frac{d\sigma}{dz} \Big |_{z^2_N} {}_Tw^2 \frac{d\sigma }{dz} \Big |_{z^2_{N-1}} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_{N-1}} {}_Tw^1 \frac{d\sigma}{d z} \Big |_{z^1_{N-2}} +\frac{d\sigma}{dz} \Big |_{z^2_N} {}_Tw^2 \frac{d\sigma}{dz} \Big |_{z^2_{N-2}} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_{N-2}} \right ) h^1_{N-3} \\ & + \dots \Bigg \}. \end{align*}\]

The nice thing about rearranging the terms in this way is that you get a visual idea of the error terms trickling back through layers and through time. For example, look at the line that ends with \(h^1_{N-1}\): it already contains the seed for the following line’s term \(\frac{d\sigma}{dz} \Big |_{z^2_N} {}_Sw^2 \frac{d\sigma}{dz} \Big |_{z^1_N} {}_Tw^1\). This is the essence of backpropagation, where we can compute the errors at time \(N-1\) and backpropagate an error signal to compute the error at time \(N-2\). In the end, the total error will be the sum of the errors at each instant.

A similar derivation can and should be done for \(\frac{dC(y_N)}{d{}_Sw^2}, \frac{dC(y_N)}{d{}_Sw^1}\) but I don’t have the time or space to do it here. Instead, let me go straight to the vector notation and a wonderful visualization!

Vector notation

Imagine now that instead of a single neuron per layer, we had multiple neurons.

Doing the same tricks as for a feedforward network, we can write the backprop algorithm in vector notation:

\[\begin{align*} &\text{ backwards for each layer } l = L, \dots, 1 \\ &\text{ backwards in time } n = N, \dots, 1 \\ &\delta = {}_T\delta^l_{n+1} + {}_S\delta^{l+1}_n \\ &\Delta {}_SW^l = \Delta {}_SW^l + \left [ \delta \odot \frac{d\sigma}{dz} \Bigg |_{\mathbf{z}^l_n} \right ] \otimes \mathbf{h}^{l-1}_n \\ &\Delta {}_TW^l = \Delta {}_TW^l + \left [ \delta \odot \frac{d\sigma}{dz} \Bigg |_{\mathbf{z}^l_n} \right ] \otimes \mathbf{h}^{l}_{n-1} \\ &{}_S\delta^l_n = ({}_SW^l)^T \left [ \delta \odot \frac{d\sigma}{dz} \Bigg |_{\mathbf{z}^l_n} \right ] \\ &{}_T\delta^l_n = ({}_TW^l)^T \left [ \delta \odot \frac{d\sigma}{dz} \Bigg |_{\mathbf{z}^l_n} \right ] . \end{align*}\]

It can be easy to visualize it all in sort of grid, with layers (space) on one axis and time on the other.

To formulas in the image may not reflect with 100% accuracy the notation of the text. Please refer to the text as the more up-to-date (and thus, hopefully, correct) version.

TBTTP: Truncated Backpropagation Through Time

it may not be desirable to go back until the dawn of time for every backpropagation step. So, a simple but effective technique is used which consists in truncating the backward step in time after a given amount of steps \(\tau_{max}\).

Another technique often coupled with this one is to avoid performing backprop through time at every time step, but only do it every \(\tau_b\) steps.

Show me the code!

A simple implementation that pretty much follows the notation from this blog post is here. In particular, the code implements a many-to-one architecture where the network ingests a full sequence and produces a single number as output. As such, it only performs BPTT at the end of the sequence, instead of at every time step or every \(\tau_b\) steps as descrbied above.

A naive explanation of backprop

2017-06-07T11:53:10+00:00

I got lost trying to implement some simple neural networks from scratch in python. In particular, I figured out that I didn’t really understant how backpropagation works and what is the magic behind it. I decided to inaugurate a blog with my thoughts on backpropagation.

Now, there are many other explanations of backprop out there, much better than mine. Chapter 2 of M. Nielsen’s book is where I learned all this stuff in the beginning. I then went to the deep learning book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville for a deeper but less gentle explanation. Finally, C. Olah’s blog, A. Karpahty’s articles and S. Raval’s hilarious videos all allowed me to get more and more comfortable with different aspects of this problem.

The purpose of this blog post is two fold: first of all, several of the resources above encourage the readers to do the math. This is my attempt at detailing some mathematical expressions, and in turn I encourage everyone to check their calculations against mine and point out any errors/inconsistencies. Second of all, I wanted to get comfortable with the idea of having a blog, and thought this would be a good starting point.

The handwaving backprop explanation

Everyone is talking about it, and everyone has their own, two-sentence explanation that they casually drop in conversation when they want to look cool at a conference. I like to think of myself as a funny guy, so I usually tell the joke:

Backpropagation is merely a rebranding of the chain rule. Yes, I find it quite….. derivative.

if you haven’t fallen off your chair, let me restate what pretty much everybody says about backprop: “in neural networks, the error is associated with the gradient of the cost function. Thanks to the backpropagation algorithm, we have a fast and efficient way of computing this gradient w.r.t every parameter of the model.”

Why Backprop?

Why do we go backwards in the backprop algorithm?

Consider the following simple example.

To compute \(y\), we have the following relation:

\[y = \sigma(z) = \sigma( w_2 h_1 ) = \sigma( w_2 \sigma( w_1 x ) ).\]

Suppose we want to compute \(\frac{dy}{dw_1}\) and \(\frac{dy}{dw_2}\), applying the chain rule we get:

\[\begin{align*} \frac{dy}{dw_1} & = \frac{d \sigma(z_2) }{dz_2} \frac{ dz_2 }{dw_1 } \\ & = \frac{d \sigma(z_2) }{dz_2} \frac{ dz_2 }{dh_1} \frac{dh_1}{dw_1} \\ & = \frac{d \sigma(z_2) }{dz_2} \frac{ dz_2 }{dh_1} \frac{dh_1}{dz_1} \frac{dz_1}{dw_1} \\ & = \frac{d \sigma(z_2) }{dz_2} w_2 \frac{dh_1}{dz_1} x, \\ \frac{dy}{dw_2} & = \frac{d \sigma(z_2) }{dz_2} \frac{ dz_2 }{dw_2} \\ & = \frac{d \sigma(z_2) }{dz_2} h_1. \end{align*}\]

All these formulas look complicated, yet simple enough for our high-school selves to understand them. Isn’t that wonderful!

The trick in backpropagation is noticing that the term \(\frac{d \sigma(z_2) }{dz_2}\) appears several times. First, we need it to compute \(\frac{dy}{dw_2}\). Then, we back-propagate it via \(\frac{d \sigma(z_2) }{dz_2} w_2\) because we also need it to compute \(\frac{dy}{dw_1}\).

The idea is that each layer has the necessary information to compute this error signal, and can pass it backwards to the previous layers which needs it to compute its own derivative (and its own error signals to pass on, if it has previous layers).

Cost function

In the previous example, we wanted to compute the derivative of \(y\). The question arises naturally, \(y\) do you want that? (ah ah).

Well, ideally you have a goal in mind for your network, some sort of cognitive function that you wish it would emulate(e.g. learning to classify handwritten digits from an image). So how do you translate this into some beautiful math that your NN can digest? You postulate the existence of a cost function \(C\). The purpose of a cost function is, in the words of Dr. Evil, to throw us a friggin’ bone. In this simplified point of view, a cost function is a way to massage the neural network’s output in something that is:

scalar;
differentiable;
related to the error in the cognitive function you want to emulate.

So what’s the relationship with learning? The idea is that you are going to tweak and adjust the weights in your network (and generally jump through several hoops) in order to ensure that your neural network does one thing really well: minimize its cost function.

The way you’ll do this is by computing error signals in the form of gradients, and updating network weights accordingly. Therefore you would change the formulas above to compute:

\[\begin{align*} \frac{dC}{dw_1} & =\frac{dC}{dy} \frac{dy}{dw_1}, \\ \frac{dC}{dw_2} & =\frac{dC}{dy} \frac{dy}{dw_2}. \end{align*}\]

Why gradients?

This might seem like a pretty silly question, but it still required me some thinking: why exactly do we use the gradient as an error signal?

There are obviously some intuitive explanations for this: as a first approximation, a small change in a parameter corresponds to a change in the output that is proportional to the gradient w.r.t. that parameter. However, the moment of truth that finally put my mind at ease was the realization that using gradients as an error signal is only a consequence of the arbitrary decision of using Gradient Descent as an optimization algorithm. If we were to use different methods (e.g. Newton’s) we would be computing more than just gradients (see e.g. section 8.6 of the deep learning book).

A gentle reminder, the update formula for gradient descent is

\(w_1 \leftarrow w_1 - \eta \frac{dC}{dw_1}\).

We call \(\eta\) the learning rate, a parameter that can have a major effect on convergence.

Scary math in the form of vectors

Let’s take a leap of imagination here and imagine that in the network from the previous example there was at each layer a vector of neurons instead of single ones. I know I’m familiar with this concept, since I applied the same technique at age 14 so I could tell mom I had friends and a girlfriend. Because the devil is in the details, let’s set some dimensions!

\[\mathbf{x} \in \mathbb{R}^{N_{in}},~~~ \mathbf{h}_1 \in \mathbb{R}^{N_1},~~~ \mathbf{h}_2 = \mathbf{y} \in \mathbb{R}^{N_2} .\]

Let me state this clearly here, all vectors here are column vectors.

To compute the feedforward part of the neural network, just apply the simple recipe:

\[\begin{align*} \mathbf{z}_1 = & W_1\mathbf{x} \\ \mathbf{h}_1 = & \mathbf{ \sigma } (\mathbf{z}_1) \\ \mathbf{z}_2 = & W_2 \mathbf{h}_1 \\ \mathbf{y} = & \mathbf{ \sigma } (\mathbf{z}_2) \\ cost = & C(\mathbf{y}) \end{align*}\]

Now, let me fill you in on some details: \(W_1,W_2\) are matrices whose dimensions are \(N_{this~layer} \times N_{previous~layer}\), i.e.:\(W_1 \in \mathbb{R}^{ N_1 \times N_{in} }, W_2 \in \mathbb{R}^{ N_2 \times N_1 }.\) The function \(\mathbf{ \sigma}\) transforms a vector into another vector with same dimensions \(\mathbf{ \sigma}: \mathbb{R}^N \to \mathbb{R}^N\) whereas \(C\) transforms a vector into a scalar \(C: \mathbb{R}^{N_2} \to \mathbb{R}\).

How do we rewrite the gradient descent formula for vectors? Easy, just look at it component-wise:

\[w_{1,ij} \leftarrow w_{1,ij} + \frac{dC}{dw_{1,ij}}.\]

Now, evaluating the derivative requires some math-fu. We start by writing the feedforward relationship component-wise:

\[\begin{align*} y_u & = \sigma( z_{2,u} ) \\ & = \sigma \left ( \sum\limits_m^{N_1} w_{2,um}h_{1,m} \right ) \\ & = \sigma \left ( \sum\limits_m^{N_1} w_{2,um} \sigma \left ( \sum\limits_{p}^{N_{in}} w_{1,mp}x_p \right ) \right ) \end{align*}\]

so we can write:

\[\begin{align*} \frac{dC}{dw_{1,ij}} & = \sum\limits_u^{N_2} \frac{dC}{dy} \Bigg|_{y_u} \frac{dy_u}{dw_{1,ij}} \\ & = \sum\limits_u^{N_2} \frac{dC}{dy} \Bigg|_{y_u} \frac{ d\sigma }{dz} \Bigg|_{z_{2,u}} \frac{d z_{2,u} }{dw_{1,ij}} \\ & = \sum\limits_u^{N_2} \frac{dC}{dy} \Bigg|_{y_u} \frac{ d\sigma }{dz} \Bigg|_{z_{2,u}} \left [ \sum\limits_m^{N_1} w_{2,um} \frac{ d h_{1,m} }{dw_{1,ij}} \right ] \\ & = \sum\limits_u^{N_2} \frac{dC}{dy} \Bigg|_{y_u} \frac{ d\sigma }{dz} \Bigg|_{z_{2,u}} \left [ \sum\limits_m^{N_1} w_{2,um} \frac{ d\sigma }{dz} \Bigg|_{ z_{1,m} } \left ( \frac{d}{dw_{1,ij}} \sum\limits_{p}^{N_{in}} w_{1,mp}x_p \right ) \right ]. \end{align*}\]

And now we simplify:

\[\frac{d}{dw_{1,ij}} \sum\limits_{p}^{N_{in}} w_{1,mp}x_p = \begin{cases} x_j & \text{if } m = i\\ 0 & \text{otherwise}, \end{cases}\]

and we are left with

\[\frac{dC}{dw_{1,ij}} = \sum\limits_u^{N_2} \frac{dC}{dy} \Bigg|_{y_u} \frac{ d\sigma }{dz} \Bigg|_{z_{2,u}} w_{2,ui} \frac{ d\sigma }{dz} \Bigg|_{ z_{1,i} } x_j.\]

Now we’re gonna be fancy and write this back into matrix notation. We’re going to use this rule

\[\sum\limits_u k_ua_{ui} \to \text{the } i^{th} \text{ component of } A^T\mathbf{k},\]

and introduce the outer product \(\otimes\) between \(\mathbf{x} \in \mathbb{R}^{N_x}\) and \(\mathbf{y} \in \mathbb{R}^{N_y}\) such that

\[\begin{align*} & \mathbf{x} \otimes \mathbf{y} \in \mathbb{R}^{N_x \times N_y} \\ & \left [ \mathbf{x} \otimes \mathbf{y} \right ]_{ij} = x_iy_j, \end{align*}\]

and introduce also the simpler element-wise product \(\left [ \mathbf{x} \odot \mathbf{y} \right ]_i = x_iy_i\).

We can finally say that \(\frac{dC}{dw_{1,ij}}\) will be the \((ij)^{th}\) component of the matrix

\[\left [ W_2^T \left ( \nabla C \Bigg|_{\mathbf{y}} \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{2}} \right ) \right ] \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{1}} \otimes \mathbf{x}.\]

By repeating the exact same computations, but simpler, we can get the expression for \(\frac{dC}{dw_{2,ij}}\) as the \((ij)^{th}\) component of the matrix

\[\left ( \nabla C \Bigg|_{\mathbf{y}} \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{2}} \right ) \otimes \mathbf{h}_1.\]

As a validation, you can check that the above formula is indeed an \((N_2 \times N_1)\) matrix.

The backprop insight

This is where the magic happens! We can summarize our gradient descent formulas as

\[\begin{align*} W_2 \leftarrow & W_2 - \eta \left ( \nabla C \Bigg|_{\mathbf{y}} \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{2}} \right ) \otimes \mathbf{h}_1 \\ W_1 \leftarrow & W_1 - \eta \left [ W_2^T \left ( \nabla C \Bigg|_{\mathbf{y}} \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{2}} \right ) \right ] \odot \frac{ d \sigma }{ dz} \Bigg|_{\mathbf{z}_{1}} \otimes \mathbf{x}. \end{align*}\]

As we noted before, one term is repeated in the two expressions, namely \(\left ( \nabla C \Bigg|_{\mathbf{y}} \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{2}} \right )\). Moreover, this repeated term can be computed using only knowledge from the cost function and layer 2. We can use this insight to define an error signal that gets backpropagated through the network.

We are left with the following backprop algorithm:

compute the gradient of the cost function, evaluated on the activations of the last layer;
set the error signal \(\mathbf{ \delta } = \nabla C \Bigg|_{\mathbf{y}}\)
for each layer \(l\) going backwards:
- update the weight matrix \(W_l \leftarrow W_l - \eta \left ( \mathbf { \delta } \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{l}} \right ) \otimes \mathbf{h}_{l-1}\);
- backpropagate the error signal \(\mathbf { \delta } = W_l^T \left ( \mathbf { \delta } \odot \frac{ d \sigma }{dz} \Bigg|_{\mathbf{z}_{l}} \right )\).

Show me the code!

A simple implementation that pretty much follows the notation from this blog post is here.