A Gentle Introduction to RNN Unrolling

 Recurrent neural networks are a type of neural network where the outputs from previous time steps are fed as input to the current time step.

This creates a network graph or circuit diagram with cycles, which can make it difficult to understand how information moves through the network.

In this post, you will discover the concept of unrolling or unfolding recurrent neural networks.

After reading this post, you will know:

• The standard conception of recurrent neural networks with cyclic connections.
• The concept of unrolling of the forward pass when the network is copied for each input time step.
• The concept of unrolling of the backward pass for updating network weights during training.Let’s get started.

  Unrolling Recurrent Neural Networks

  Recurrent neural networks are a type of neural network where outputs from previous time steps are taken as inputs for the current time step.

  We can demonstrate this with a picture.

  Below we can see that the network takes both the output of the network from the previous time step as input and uses the internal state from the previous time step as a starting point for the current time step.

  

  Example of an RNN with a cycle

  RNNs are fit and make predictions over many time steps. We can simplify the model by unfolding or unrolling the RNN graph over the input sequence.

  A useful way to visualise RNNs is to consider the update graph formed by ‘unfolding’ the network along the input sequence.

  — Supervised Sequence Labelling with Recurrent Neural Networks, 2008.

  Unrolling the Forward Pass

  Consider the case where we have multiple time steps of input (X(t), X(t+1), …), multiple time steps of internal state (u(t), u(t+1), …), and multiple time steps of outputs (y(t), y(t+1), …).

  We can unfold the above network schematic into a graph without any cycles.

  

  Example of Unrolled RNN on the forward pass

  We can see that the cycle is removed and that the output (y(t)) and internal state (u(t)) from the previous time step are passed on to the network as inputs for processing the next time step.

  Key in this conceptualization is that the network (RNN) does not change between the unfolded time steps. Specifically, the same weights are used for each time step and it is only the outputs and the internal states that differ.

  In this way, it is as though the whole network (topology and weights) are copied for each time step in the input sequence.

  Further, each copy of the network may be thought of as an additional layer of the same feed forward neural network.

  

  Example of Unrolled RNN with each copy of the network as a layer

  RNNs, once unfolded in time, can be seen as very deep feedforward networks in which all the layers share the same weights.

  — Deep learning, Nature, 2015

  This is a useful conceptual tool and visualization to help in understanding what is going on in the network during the forward pass. It may or may not also be the way that the network is implemented by the deep learning library.

  Unrolling the Backward Pass

  The idea of network unfolding plays a bigger part in the way recurrent neural networks are implemented for the backward pass.

  As is standard with [backpropagation through time] , the network is unfolded over time, so that connections arriving at layers are viewed as coming from the previous timestep.

  — Framewise phoneme classification with bidirectional LSTM and other neural network architectures, 2005

  Importantly, the backpropagation of error for a given time step depends on the activation of the network at the prior time step.

  In this way, the backward pass requires the conceptualization of unfolding the network.

  Error is propagated back to the first input time step of the sequence so that the error gradient can be calculated and the weights of the network can be updated.

  Like standard backpropagation, [backpropagation through time] consists of a repeated application of the chain rule. The subtlety is that, for recurrent networks, the loss function depends on the activation of the hidden layer not only through its influence on the output layer, but also through its influence on the hidden layer at the next timestep.

  — Supervised Sequence Labelling with Recurrent Neural Networks, 2008

  Unfolding the recurrent network graph also introduces additional concerns. Each time step requires a new copy of the network, which in turn takes up memory, especially for larger networks with thousands or millions of weights. The memory requirements of training large recurrent networks can quickly balloon as the number of time steps climbs into the hundreds.

  … it is required to unroll the RNNs by the length of the input sequence. By unrolling an RNN N times, every activations of the neurons inside the network are replicated N times, which consumes a huge amount of memory especially when the sequence is very long. This hinders a small footprint implementation of online learning or adaptation. Also, this “full unrolling” makes a parallel training with multiple sequences inefficient on shared memory models such as graphics processing units (GPUs)

  — Online Sequence Training of Recurrent Neural Networks with Connectionist Temporal Classification, 2015

  Further Reading

  This section provides more resources on the topic if you are looking go deeper.

  Papers

  • Online Sequence Training of Recurrent Neural Networks with Connectionist Temporal Classification, 2015
  • Framewise phoneme classification with bidirectional LSTM and other neural network architectures, 2005
  • Supervised Sequence Labelling with Recurrent Neural Networks, 2008
  • Deep learning, Nature, 2015

  Articles

  • A Gentle Introduction to Backpropagation Through Time
  • Understanding LSTM Networks, 2015
  • Rolling and Unrolling RNNs, 2016
  • Unfolding RNNs, 2017

  Summary

  In this tutorial, you discovered the visualization and conceptual tool of unrolling recurrent neural networks.

  Specifically, you learned:

  • The standard conception of recurrent neural networks with cyclic connections.
  • The concept of unrolling of the forward pass when the network is copied for each input time step.
  • The concept of unrolling of the backward pass for updating network weights during training.

  Do you have any questions?
  Ask your questions in the comments below and I will do my best to answer.