Multimedia Data Modelling Using Multidimensional Recurrent Neural Networks

Abstract

Modelling the multimedia data such as text, images, or videos usually involves the analysis, prediction, or reconstruction of them. The recurrent neural network (RNN) is a powerful machine learning approach to modelling these data in a recursive way. As a variant, the long short-term memory (LSTM) extends the RNN with the ability to remember information for longer. Whilst one can increase the capacity of LSTM by widening or adding layers, additional parameters and runtime are usually required, which could make learning harder. We therefore propose a Tensor LSTM where the hidden states are tensorised as multidimensional arrays (tensors) and updated through a cross-layer convolution. As parameters are spatially shared within the tensor, we can efficiently widen the model without extra parameters by increasing the tensorised size; as deep computations of each time step are absorbed by temporal computations of the time series, we can implicitly deepen the model with little extra runtime by delaying the output. We show by experiments that our model is well-suited for various multimedia data modelling tasks, including text generation, text calculation, image classification, and video prediction.

1. Introduction

Multimedia data such as text, images, and videos are ubiquitous nowadays. Modelling such data usually involves the analysis, prediction, or reconstruction of them. For instance, text modelling relates to many natural language processing tasks such as sentiment analysis [1], part-of-speech tagging [2], machine translation [3], and question answering [4], image modelling relates to many computer vision tasks such as image segmentation [5], depth reconstruction [6], image generation [7], and super-resolution [8], and video modelling also relates to many computer vision tasks such as object tracking [9], video segmentation [10], motion estimation [11], and video prediction [12]. Although they are diverse, these tasks usually can be formulated as a time series prediction problem, e.g., generating a desired output ${\mathit{y}}_t$ for a given time series ${\mathit{x}}_{1:t}=\{{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_t\}$, for time $t=1,2,\dots ,T$, where ${\mathit{x}}_t\in {\mathbb{R}}^U$ and ${\mathit{y}}_t\in {\mathbb{R}}^V$ are vectors (In this paper, we assume the vectors are in row form). The recurrent neural network (RNN) [13,14] is a popular model that can learn to encapsulate the useful information of the input history ${\mathit{x}}_{1:t}$ into a hidden state vector ${\mathit{h}}_t\in {\mathbb{R}}^M$. By concatenating the input ${\mathit{x}}_t$ to the previous hidden state ${\mathit{h}}_{t-1}$, we first get ${\mathit{h}}_{t-1}^{cat}\in {\mathbb{R}}^{U+M}$:

$$\begin{array}{c}{\mathit{h}}_{t-1}^{cat}=[{\mathit{x}}_t,{\mathit{h}}_{t-1}].\hfill \end{array}$$

(1)

The hidden state ${\mathit{h}}_t$ is then updated by:

$$\begin{array}{c}{\mathit{a}}_t={\mathit{h}}_{t-1}^{cat}{\mathit{W}}^h+{\mathit{b}}^h,\hfill \end{array}$$

(2)

$$\begin{array}{c}{\mathit{h}}_t=\varphi \left({\mathit{a}}_t\right),\hfill \end{array}$$

(3)

where ${\mathit{W}}^h\in {\mathbb{R}}^{(U+M)\times M}$ and ${\mathit{b}}^h\in {\mathbb{R}}^M$ are parameters namely the weight and bias, respectively; ${\mathit{a}}_t\in {\mathbb{R}}^M$ is the activation for ${\mathit{h}}_t$, and $\varphi (·)$ is the tanh function (element-wise). The RNN finally produces an output ${\mathit{y}}_t$ for time t:

$${\mathit{y}}_t=\phi ({\mathit{h}}_t{\mathit{W}}^y+{\mathit{b}}^y),$$

(4)

where ${\mathit{W}}^y\in {\mathbb{R}}^{M\times V}$ and ${\mathit{b}}^y\in {\mathbb{R}}^V$, and $\phi (·)$ is a differentiable transformation that depends on the task.

Nevertheless, the standard RNN is notorious for capturing the long-term dependency caused by the vanishing and exploding gradients [15]. Long Short-Term Memories (LSTMs) [16,17] mitigate this by (i) introducing the memory cell to store information longer, and (ii) utilising the gates for information routing. In a standard LSTM [17], the hidden state ${\mathit{h}}_t$ is updated as follows:

$$\begin{array}{c}[{\mathit{a}}_t^g,{\mathit{a}}_t^i,{\mathit{a}}_t^f,{\mathit{a}}_t^o]={\mathit{h}}_{t-1}^{cat}{\mathit{W}}^h+{\mathit{b}}^h,\hfill \end{array}$$

(5)

$$\begin{array}{c}[{\mathit{g}}_t,{\mathit{i}}_t,{\mathit{f}}_t,{\mathit{o}}_t]=[\varphi \left({\mathit{a}}_t^g\right),\sigma \left({\mathit{a}}_t^i\right),\sigma ({\mathit{a}}_t^f),\sigma \left({\mathit{a}}_t^o\right)],\hfill \end{array}$$

(6)

$$\begin{array}{c}{\mathit{c}}_t={\mathit{g}}_t\odot {\mathit{i}}_t+{\mathit{c}}_{t-1}\odot {\mathit{f}}_t,\hfill \end{array}$$

(7)

$$\begin{array}{c}{\mathit{h}}_t=\varphi \left({\mathit{c}}_t\right)\odot {\mathit{o}}_t,\hfill \end{array}$$

(8)

where ${\mathit{W}}^h\in {\mathbb{R}}^{(U+M)\times 4M}$ and ${\mathit{b}}^h\in {\mathbb{R}}^{4M}$ are parameters, ${\mathit{a}}_t^i,{\mathit{a}}_t^o,{\mathit{a}}_t^g,{\mathit{a}}_t^f\in {\mathbb{R}}^M$ are respectively the activations of the input gate ${\mathit{i}}_t$, the output gate ${\mathit{o}}_t$, the new content ${\mathit{g}}_t$, and the forget gate ${\mathit{f}}_t$, ${\mathit{c}}_t\in {\mathbb{R}}^M$ is the updated memory cell, $\sigma (·)$ is the sigmoid function (element-wise), and ⊙ is the element-wise multiplication. Since LSTM is successful in modelling time series, it is natural to further increase its capacity so that it could be profitably applied to a wider range of tasks.

We consider the width and the depth to compose a network’s capacity, where the former measures how much information could be processed in parallel, while the latter measures how many computation steps are required for processing [18]. Whilst using more hidden units in a layer can widen the LSTM, it scales the parameter number quadratically. On the other hand, the Stacked LSTM (sLSTM) deepens the LSTM by using multiple layers [19]; however, the runtime scales linearly with the layer number and the input information is likely to be lost when it vertically passes through the LSTM layers (caused by vanishing/exploding gradients).

The goal of this paper is to make the LSTM wider and deeper and meanwhile prevent its parameter number and runtime from growing. To summarize, we have the following contributions:

• We represent RNN hidden states as multidimensional arrays (tensors) to allow more flexible parameter sharing, thereby being able to efficiently widen the network without extra parameters.
• We use the temporal computations of the RNN to absorb its deep computations in order that we can deepen it without extra runtime. We call this novel RNN as the Tensor RNN (tRNN).
• We propose a memory cell convolution and apply it to the tRNN in order to mitigate gradient vanishing and explosion, obtaining a Tensor LSTM (tLSTM).
• We generalise the tLSTM so that it can process not only non-structured time series (series of vectors) [20], but also structured time series (series of tensors, such as videos).
• We show by experiments that our model is well-suited for various multimedia data modelling tasks, including text generation, text calculation, image classification, and video prediction.

2. Method

2.1. Tensor Representation

From (2), we can see that the parameter number of an RNN scales quadratically with its hidden size. To widen the network while restricting its parameter number, one can use tensor factorisation, where the parameters are represented by multidimensional tensors that could be factorised as low-rank subtensors containing much fewer elements [21,22,23,24,25,26,27,28,29]. As the hidden state vector would be broadcast when interacting with the parameter tensor, the network is widened implicitly. One can also limit the parameter number of an RNN by spatially sharing a small group of parameters within its hidden state, analogous to the convolutional neural network (CNN) [30,31].

Here, we use parameter sharing to reduce the number of parameters in RNNs. In contrast to tensor factorisation, it provides two benefits: (i) scalability—the size of hidden state would not affect the number of parameters; (ii) separability—we can carefully route the information via the receptive field control so that the RNN deep computations can be shifted into its temporal direction (Section 2.2). In addition, the hidden state vectors of RNN are explicitly tensorised, as tensors are more: (i) flexible—we can choose the dimensions for parameter sharing and then just enlarge the sizes of these dimensions so that no more parameters are introduced; (ii) efficient—by using tensors of higher dimensionality, we can widen the network faster when the number of parameters is fixed (Section 2.3).

To ease explanation, let’s firstly focus on 2D tensors (matrices). Given a hidden state ${\mathit{h}}_t\in {\mathbb{R}}^M$, we tensorise it as ${\mathit{H}}_t\in {\mathbb{R}}^{P\times M}$, where P and M are respectively the tensorised size and channel size. In ${\mathit{H}}_t$, the 1st dimension is locally-connected for parameter sharing, and the 2nd dimension is fully-connected for global interaction—like in CNN where only the last dimension is fully-connected so that different feature planes (e.g., red/green/blue channels for the input image) can be globally fused. In addition, when comparing ${\mathit{H}}_t$ with the hidden state in the Stacked RNN (sRNN) (as in Figure 1a), P can be thought as the layer number, and M can be thought as each layer’s size. To explain our model, we start with 2D tensors, and then demonstrate how to use higher dimensional tensors to enhance the model, and finally show how to extend the model to deal with structured input time series.

2.2. Deep Computation through Time

As RNN is already deep when unfolded in time, we can associate the input ${\mathit{x}}_t$ with a future (delayed) output to also make the input-to-output computation deep. To achieve this, we should guarantee that the output ${\mathit{y}}_t$ is separable, i.e., it is independent of the future input ${\mathit{x}}_{t:T}$. Therefore, we first stack ${\mathit{x}}_t$’s projection on top of ${\mathit{H}}_{t-1}$; then move the input content downwards through the temporal computation, and finally produce ${\mathit{y}}_t$ from the bottom of a future hidden state ${\mathit{H}}_{t+L-1}$, where $L-1$ denotes the delayed time steps and L denotes the depth. Figure 1b shows an example with $L=3$, which can be thought as a skewed sRNN that is mentioned in [7,32]. However, in our implementation, the network structure does not need to be changed, and various interactions are also allowed if the output satisfies the separability. For instance, we can use wider local connections or introduce feedback connections (as in Figure 1c) to improve the model (similar to [33]). In addition, we update ${\mathit{H}}_t$ by convolving it with a learnable kernel so that the parameters can be shared. In doing this, we have increased the input–output mapping complexity (via output delay) and limited the growth of the parameter number (via parameter sharing by convolution).

To define the above described tRNN, we denote the concatenated hidden state as ${\mathit{H}}_{t-1}^{cat}\in {\mathbb{R}}^{(P+1)\times M}$, and the location at a tensor as $p\in {\mathbb{Z}}_{+}$. At location p of ${\mathit{H}}_{t-1}^{cat}$, the channel vector ${\mathit{h}}_{t-1,p}^{cat}\in {\mathbb{R}}^M$ satisfies:

$${\mathit{h}}_{t-1,p}^{cat}=\left\{\begin{array}{cc}{\mathit{x}}_t{\mathit{W}}^x+{\mathit{b}}^x,\hfill & \mathrm{if}p=1\hfill \\ {\mathit{h}}_{t-1,p-1},\hfill & \mathrm{if}p>1,\hfill \end{array}\right.$$

(9)

where ${\mathit{W}}^x\in {\mathbb{R}}^{U\times M}$ and ${\mathit{b}}^x\in {\mathbb{R}}^M$. The hidden state ${\mathit{H}}_t$ is then updated through a convolution:

$$\begin{array}{c}{\mathit{A}}_t={\mathit{H}}_{t-1}^{cat}⊛\{{\mathit{W}}^h,{\mathit{b}}^h\},\hfill \end{array}$$

(10)

$$\begin{array}{c}{\mathit{H}}_t=\varphi \left({\mathit{A}}_t\right),\hfill \end{array}$$

(11)

where ${\mathit{W}}^h\in {\mathbb{R}}^{K\times M^i\times M^o}$ and ${\mathit{b}}^h\in {\mathbb{R}}^{M^o}$ are the kernel’s weight and bias, respectively, with K denoting the kernel size, $M^i=M$ the input channel, and $M^o=M$ the output channel, ${\mathit{A}}_t\in {\mathbb{R}}^{P\times M^o}$ denotes the activation of ${\mathit{H}}_t$, and ⊛ denotes the convolution operation (detailed in Appendix A.1). As kernels convolve across different layers of the hidden state, we call the convolution as a cross-layer convolution, which allows the interaction among layers (from both top-down and bottom-up). Finally, the channel vector ${\mathit{h}}_{t+L-1,P}\in {\mathbb{R}}^M$, located at the bottom of ${\mathit{H}}_{t+L-1}$, is used to generate ${\mathit{y}}_t$:

$${\mathit{y}}_t=\phi ({\mathit{h}}_{t+L-1,P}{\mathit{W}}^y+{\mathit{b}}^y),$$

(12)

where ${\mathit{W}}^y\in {\mathbb{R}}^{M\times V}$ and ${\mathit{b}}^y\in {\mathbb{R}}^V$. To ensure ${\mathit{y}}_t$’s receptive field only covers historical inputs ${\mathit{x}}_{1:t}$ (as in Figure 1c), a constraint among L, P, and K needs to be satisfied:

$$L=\lceil \frac{2P}{K-K\%2}\rceil ,$$

(13)

where % is the modulus operator and ⌈·⌉ denotes the ceil operation. Please see in Appendix B for the derivation of (13).

We call the RNN described in (9)–(12) as a Tensor RNN (tRNN), where one can increase the tensorised size P to widen the model, and meanwhile keep the number of parameters fixed (by using convolution). Moreover, different from the sRNN with a runtime complexity of $O\left(TL\right)$, the runtime complexity of a tRNN is broken down to $O(T+L)$, indicating that the runtime would not be significantly affected by T or L.

2.3. Using LSTMs

To capture the long-term dependency across different time steps, the tRNN can be straightforwardly extended with LSTM by modifying (10) and (11) as:

$$\begin{array}{c}[{\mathit{A}}_t^g,{\mathit{A}}_t^i,{\mathit{A}}_t^f,{\mathit{A}}_t^o]={\mathit{H}}_{t-1}^{cat}⊛\{{\mathit{W}}^h,{\mathit{b}}^h\},\hfill \end{array}$$

(14)

$$\begin{array}{c}[{\mathit{G}}_t,{\mathit{I}}_t,{\mathit{F}}_t,{\mathit{O}}_t]=[\varphi \left({\mathit{A}}_t^g\right),\sigma \left({\mathit{A}}_t^i\right),\sigma \left({\mathit{A}}_t^f\right),\sigma \left({\mathit{A}}_t^o\right)],\hfill \end{array}$$

(15)

$$\begin{array}{c}{\mathit{C}}_t={\mathit{G}}_t\odot {\mathit{I}}_t+{\mathit{C}}_{t-1}\odot {\mathit{F}}_t,\hfill \end{array}$$

(16)

$$\begin{array}{c}{\mathit{H}}_t=\varphi \left({\mathit{C}}_t\right)\odot {\mathit{O}}_t,\hfill \end{array}$$

(17)

where $\{{\mathit{W}}^h,{\mathit{b}}^h\}$ is the kernel with kernel size K, input channel $M^i=M$, and output channel $M^o=4M$, ${\mathit{A}}_t^i,{\mathit{A}}_t^o,{\mathit{A}}_t^g,{\mathit{A}}_t^f\in {\mathbb{R}}^{P\times M}$ are respectively the activations of the input gate ${\mathit{I}}_t$, the output gate ${\mathit{O}}_t$, the new content ${\mathit{G}}_t$, and the forget gate ${\mathit{F}}_t$, and ${\mathit{C}}_t\in {\mathbb{R}}^{P\times M}$ is the updated memory cell. However, as (16) only gates the previous memory cell ${\mathit{C}}_{t-1}$ along the temporal direction (as in Figure 1d), when the tensorised size P grows large, the long-term dependency from the input–output direction is likely to be lost.

2.3.1. Memory Cell Convolution

Here, we propose a novel memory cell convolution (memConv) for capturing the long-term dependency from multiple directions, where, like the hidden state, the memory cell could also have a wider receptive field (as in Figure 1e). In addition, the kernel for memConv is generated on the fly and therefore varies with time and location, flexibly controlling the long-term dependency from different directions. Concretely, we define the tensor update for tLSTM as follows:

$$\begin{array}{c}[{\mathit{A}}_t^g,{\mathit{A}}_t^i,{\mathit{A}}_t^f,{\mathit{A}}_t^o,{\mathit{A}}_t^q]={\mathit{H}}_{t-1}^{cat}⊛\{{\mathit{W}}^h,{\mathit{b}}^h\},\hfill \end{array}$$

(18)

$$\begin{array}{c}[{\mathit{G}}_t,{\mathit{I}}_t,{\mathit{F}}_t,{\mathit{O}}_t,{\mathit{Q}}_t]=[\varphi \left({\mathit{A}}_t^g\right),\sigma \left({\mathit{A}}_t^i\right),\sigma ({\mathit{A}}_t^f),\sigma \left({\mathit{A}}_t^o\right),\varsigma \left({\mathit{A}}_t^q\right)],\hfill \end{array}$$

(19)

$$\begin{array}{c}{\mathit{W}}_t^c\left(p\right)=\mathrm{reshape}\left({\mathit{q}}_{t,p},\left[K,1,1\right]\right),\hfill \end{array}$$

(20)

$$\begin{array}{c}{\mathit{C}}_{t-1}^{conv}={\mathit{C}}_{t-1}⊛{\mathit{W}}_t^c\left(p\right),\hfill \end{array}$$

(21)

$$\begin{array}{c}{\mathit{C}}_t={\mathit{G}}_t\odot {\mathit{I}}_t+{\mathit{C}}_{t-1}^{conv}\odot {\mathit{F}}_t,\hfill \end{array}$$

(22)

$$\begin{array}{c}{\mathit{H}}_t=\varphi \left({\mathit{C}}_t\right)\odot {\mathit{O}}_t,\hfill \end{array}$$

(23)

where, unlike (14)–(17), the kernel $\{{\mathit{W}}^h,{\mathit{b}}^h\}$ contains additional $\langle K\rangle$ output channels ($\langle ·\rangle$ computes the cumulative product of the input variable elements) for generating the activation ${\mathit{A}}_t^q\in {\mathbb{R}}^{P\times \langle K\rangle }$ of the dynamic kernel bank ${\mathit{Q}}_t\in {\mathbb{R}}^{P\times \langle K\rangle }$, ${\mathit{q}}_{t,p}\in {\mathbb{R}}^{\langle K\rangle }$ denotes the vectorised dynamic kernel selected from ${\mathit{Q}}_t$’s entry p, and ${\mathit{W}}_t^c\left(p\right)\in {\mathbb{R}}^{K\times 1\times 1}$ is the dynamic kernel reshaped from ${\mathit{q}}_{t,p}$ (illustrated in Figure 2a), with a size K and a single input/output channel. Equation (21) defines the memConv (detailed in Appendix A.2), where we use ${\mathit{W}}_t^c\left(p\right)$, the value of which varies with p, to convolve every channel of ${\mathit{C}}_{t-1}$, producing a convolved memory cell ${\mathit{C}}_{t-1}^{conv}\in {\mathbb{R}}^{P\times M}$. Analogous to [34], in (19), a softmax function $\varsigma (·)$ is employed to normalise ${\mathit{Q}}_t$ along its channel dimension, which can stabilise the memory cell values and thereby mitigate the vanishing and exploding gradients (please check Appendix C for more discussion).

There are many works [22,23,27,35,36,37] using the concept of dynamically producing the model weights, where [36] also dynamically generates location-dependent convolution kernels for improving the CNN. Unlike these works, we aim to broaden the receptive fields for tLSTM memory cells. Whilst being flexible, fewer parameters are needed for generating the memConv kernel as it can be shared by different channels of the memory cell.

2.3.2. Channel Normalisation

We adapt the recently proposed layer normalisation (LN) [38] to speed-up the training of tLSTM. In [38], LN has been observed unsuitable for the CNN where different statistics are possessed by different channel vectors. Similarly, we have found that LN also performs not well for the tLSTM, in which low-level information is possessed by channel vectors close to the input and vice versa. Therefore, we propose a channel normalisation (CN), which normalises each channel vector independently. The CN operator is defined as:

$$\mathrm{CN}\left(\mathit{Z};\mathsf{\Gamma },\mathit{B}\right)=\widehat{\mathit{Z}}\odot \mathsf{\Gamma }+\mathit{B},$$

(24)

where $\mathsf{\Gamma },\mathit{B},\mathit{Z},\widehat{\mathit{Z}}\in {\mathbb{R}}^{P\times M^z}$, $\mathsf{\Gamma }$ and $\mathit{B}$ are parameters namely the gain and bias, respectively, $\mathit{Z}$ is the input tensor, and $\widehat{\mathit{Z}}$ is the normalised tensor. Let ${\mathit{z}}_{m^z}\in {\mathbb{R}}^P$ be the $m^z$-th channel of $\mathit{Z}$, it is normalised element-wisely:

$${\widehat{\mathit{z}}}_{m^z}=({\mathit{z}}_{m^z}-{\mathit{z}}^{\mu })/{\mathit{z}}^{\sigma },$$

(25)

where ${\mathit{z}}^{\mu },{\mathit{z}}^{\sigma }\in {\mathbb{R}}^P$ are respectively the mean and the standard deviation which are computed along $\mathit{Z}$’s channel dimension, and ${\widehat{\mathit{z}}}_{m^z}\in {\mathbb{R}}^P$ denotes the $m^z$-th channel in $\widehat{\mathit{Z}}$. As the parameter number introduced by CN/LN is quite small in terms of the model parameters, it could be reasonably neglected.

2.3.3. Leveraging Higher-Dimensional Tensors

In (13), we can see that given the kernel size K, the tensorised size P scales linearly w.r.t. the depth L. To widen the tLSTM more efficiently, we resort to using higher-dimensional tensors, where the tensor volume can be expanded more rapidly. Based on the tLSTM defined in previous sections, we can generalise the tensors from 2D to $(D+1)$-dimensional where $D>1$, resulting in ${\mathit{H}}_t,{\mathit{C}}_t\in {\mathbb{R}}^{P_1\times P_2\times \dots \times P_D\times M}$ with the tensorised size $P=\left[P_1,P_2,\dots ,P_D\right]$. As the hidden states are more than 2D, we instead concatenate ${\mathit{x}}_t$’s projection to the corner of ${\mathit{H}}_{t-1}$, thereby extending (9) as:

$${\mathit{h}}_{t-1,\mathit{p}}^{cat}=\left\{\begin{array}{cc}{\mathit{x}}_t{\mathit{W}}^x+{\mathit{b}}^x,\hfill & \mathrm{if}{p}_d=1\mathrm{for}1\le d\le D\hfill \\ {\mathit{h}}_{t-1,\mathit{p}-\mathbf{1}},\hfill & \mathrm{if}{p}_d>1\mathrm{for}1\le d\le D\hfill \\ \mathbf{0},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(26)

where the channel vector ${\mathit{h}}_{t-1,\mathit{p}}^{cat}\in {\mathbb{R}}^M$ is the entry $\mathit{p}\in {\mathbb{Z}}_{+}^D$ of the concatenated hidden state ${\mathit{H}}_{t-1}^{cat}\in {\mathbb{R}}^{(P_1+1)\times (P_2+1)\times \dots \times (P_D+1)\times M}$. Accordingly, the output ${\mathit{y}}_t$ is generated at the opposite corner of ${\mathit{H}}_{t+L-1}$, thus we modify (12) as:

$${\mathit{y}}_t=\phi ({\mathit{h}}_{t+L-1,P}{\mathit{W}}^y+{\mathit{b}}^y).$$

(27)

To update the hidden state, we also tensorise the convolution kernel ${\mathit{W}}^h$ and ${\mathit{W}}_t^c(·)$ so that they have a kernel size of $K=\left[K_1,K_2,\dots ,K_D\right]$, where ${\mathit{W}}_t^c(·)$ is still reshaped from the vector (see Figure 2b). In order to make every dimension of P and K meet the constraint (13) with a same L, we set $P_d=P$ and $K_d=K$ for $d=1,2,\dots ,D$. For CN, it is still applied to normalise the channel dimension of tensors.

Since D is the additional dimensionality introduced by tensorisation, we call D the tensorised dimensionality and denote the resulting tLSTM as a D-tLSTM. For instance, the untensorised LSTM defined in (5)–(8) is a 0-tLSTM, and the tLSTM (with 2D tensors) defined in (18)–(23) is an 1-tLSTM.

2.4. Handling Structured Inputs

Until now, we have limited our discussion to the case where the input at each time step is a vector, which is non-structured. However, as structured data (e.g., image time series) also emerges in many multimedia modelling tasks (e.g., video segmentation, motion estimation, and video prediction), it is essential to generalise the model to handle structured inputs.

We use ${\mathit{X}}_t\in {\mathbb{R}}^{S_1\times S_2\times \dots \times S_E\times U}$ to denote the structured input at time step t, where $E\in {\mathbb{Z}}_{+}$ is the structure dimensionality and $S=[S_1,S_2,\dots ,S_E]$ the structure size, e.g., when ${\mathit{X}}_t$ is a 2D image, then $S=[S_1,S_2]$ is the image size (height and width) and U is the image depth (channel). Correspondingly, we have a hidden state ${\mathit{H}}_t\in {\mathbb{R}}^{P_1\times P_2\times \dots \times P_D\times S_1\times S_2\times \dots \times S_E\times M}$. In contrast to (26), we define the sub-tensor ${\mathit{H}}_{t-1,\mathit{p}}^{cat}\in {\mathbb{R}}^{S_1\times S_2\times \dots \times S_E\times M}$ locating at entry $\mathit{p}$ of ${\mathit{H}}_{t-1}^{cat}$ as:

$${\mathit{H}}_{t-1,\mathit{p}}^{cat}=\left\{\begin{array}{cc}{\mathit{X}}_t⊛\{{\mathit{W}}^x,{\mathit{b}}^x\},\hfill & \mathrm{if}{p}_d=1\mathrm{for}1\le d\le D\hfill \\ {\mathit{H}}_{t-1,\mathit{p}-\mathbf{1}},\hfill & \mathrm{if}{p}_d>1\mathrm{for}1\le d\le D\hfill \\ \mathbf{0},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(28)

where the convolution kernel $\{{\mathit{W}}^x,{\mathit{b}}^x\}$ is used for linear projection and is of size $\mathbf{1}\in {\mathbb{R}}^E$, with U input channels and M output channels.

To update the hidden state tensor, the size of convolution kernels ${\mathit{W}}^h$ and ${\mathit{W}}_t^c(·)$ becomes $K=\left[K_1,K_2,\dots ,K_D,K_{D+1},K_{D+2},\dots ,K_{D+E}\right]$, where the first D dimensions, $K_{1:D}$, are related to the tensorised size $P$, and the succeeding E dimensions, $K_{D+1:D+E}$, are related to the structure size $E$. This also means that $K_{D+1:D+E}$ are free of the constraint (13).

Finally, we generate the output from the sub-tensor ${\mathit{H}}_{t+L-1,P}\in {\mathbb{R}}^{S_1\times S_2\times \dots \times S_E\times M}$. Note that for many tasks such as video prediction, the output usually has the same structure (i.e., a same $S$) as the input. In this case, the output can be generated by:

$${\mathit{Y}}_t=\phi ({\mathit{H}}_{t+L-1,P}⊛\{{\mathit{W}}^y,{\mathit{b}}^y\}),$$

(29)

where ${\mathit{Y}}_t\in {\mathbb{R}}^{S_1\times S_2\times \dots \times S_E\times V}$ is the structured output and $\{{\mathit{W}}^y,{\mathit{b}}^y\}$ is the convolution kernel of size $\mathbf{1}\in {\mathbb{R}}^E$, with M input channels and V output channels. In addition, it is straightforward to generate a non-structured output ${\mathit{y}}_t\in {\mathbb{R}}^V$ from ${\mathit{H}}_{t+L-1,P}$, e.g., by using a CNN or a fully-connected network.

3. Related Work

3.1. Convolutional LSTMs

The Convolutional LSTM (cLSTM) parallelises the computation of LSTM where at each time step the input is structured (as in Figure 3a), such as an array vector [7], a matrix of vectors [39,40,41,42], and a tensor of vectors [43,44]. Different from the cLSTM, tLSTM focuses on increasing the capacity of LSTM where each input can also be non-structured (a single vector), and has the following advantages: (i) the convolution in tLSTM is performed across different hidden layers, the structure of which can be different from the input structure, integrating information top-down and bottom-up, whereas the convolution in cLSTM is only performed within each hidden layer, the structure of which depends on the input structure, thereby falling back to the standard LSTM when each input is a single vector; (ii) by increasing the tensorised size, one can efficiently widen the tLSTM without introducing more parameters, whereas to widen the cLSTM, either increasing the kernel size or kernel channel can significantly increase the parameter number; (iii) by delaying the output, one can deepen the tLSTM with little additional runtime, whereas to deepen the cLSTM, increasing the number of hidden layers can significantly increase the runtime; (iv) with the memConv, tLSTM can capture the long-term dependency of multiple directions, whereas, cLSTM only gates the memory cell along one direction, thereby struggling to capture the long-term dependency of multiple directions.

3.2. Deep LSTMs

The Deep LSTM (dLSTM) improves the sLSTM by further deepening it (as shown in Figure 3b–d). In order to limit the number of parameters as well as make training easy, in [46,47,49,50], another RNN/LSTM is applied to the depth direction of dLSTMs. However, the runtime is still multiplied by the depth. Though deep computations are accelerated in [32,51], they mainly focus on simple architectures, e.g., sLSTMs. Unlike dLSTMs, in tLSTM, deep computations are performed with little extra runtime, and feedback is enabled by cross-layer convolutions. Furthermore, by utilising higher dimensional tensors, one can increase tLSTM’s capacity with higher efficiency, while the whole stacked hidden layers in a dLSTM only compose a 2D tensor, whose dimensionality is fixed.

3.3. Other Parallelisation Methods

When full input and target sequences are available for training, temporal computations of the time series are parallelised (for instance, by using temporal convolutions like in Figure 3e) in [48,52,53,54,55,56]. Nevertheless, for online inference, since inputs are presented sequentially, these methods can no longer parallelise temporal computations, which will also be blocked by deep computations of each time step, rendering themselves not well-suited for the real-time application which requires a high sampling/output frequencies. On the contrary, as tLSTM performs deep computations through temporal computations, it can accelerate both training and online inference for many tasks. This is human-like: when converting the input signal into action, we simultaneously process newly arrived signals in a nonblocking manner. One should also notice that for some tasks (such as autoregressive sequence generation) which take the previous output ${\mathit{y}}_{t-1}$ as the current input ${\mathit{x}}_t$, tLSTM is unable to parallelise the deep computation for online inference, since additional $L-1$ time steps are required to generate ${\mathit{y}}_{t-1}$ for each ${\mathit{x}}_t$.

4. Experiments

To evaluate our tLSTM, we experiment on seven challenging multimedia data modelling tasks, and are interested in the following configurations:

• sLSTM: We implement the sLSTM [45] and share the parameters for different layers. This configuration is served as our baseline.
• tLSTM: tLSTM with 2D tensors, which is defined in (18)–(23).
• 1-tLSTM–M: 1-tLSTM with no memConv (–M), i.e., using (14)–(17).
• 1-tLSTM–F: 1-tLSTM with no feedback connection (–F).
• 2-tLSTM: Tensorising 1-tLSTM by using 3D tensors, which is explained in Section 2.3.3, with $D=2$.
• 2-tLSTM+LN: 2-tLSTM with the LN [38].
• 2-tLSTM+CN: 2-tLSTM with the CN described in Section 2.3.2.

To make different configurations comparable, for sLSTM, we use L and M to represent the layer number and the size of each layer, respectively. Let K be the value of the first D dimensions of the kernel size $K$, we set $K=2$ for 1-tLSTM–F and $K=3$ for other tLSTM configurations, so that according to (13), we have $L=P$.

To check if tLSTM’s performance can be improved without using additional parameters, for each configuration, we use the same amount of parameters and meanwhile increase the tensorised size. We also inspect how the depth can affect the runtime, which is quantified as the averaged milliseconds cost by a single sample’s forward and backward passes over a single RNN time step. Then, we evaluate tLSTM’s ability by comparing it against the state-of-the-art methods. Finally, we analyze the inner working of tLSTM by visualising its memory cells.

The training objective is to minimise the training loss w.r.t. the parameter $\mathbf{\theta }$ (vectorised), i.e.,

$$\underset{\mathbf{\theta }}{min}\frac{1}{N}\sum _{n=1}^N\sum _{t=1}^{T_n}l\left(f({\mathit{x}}_{n,1:t}^d;\mathbf{\theta }),{\mathit{y}}_{n,t}^d\right),$$

(30)

where N is the number of training sequences, $T_n$ is the length of the n-th training sequence, and $l(·,·)$ is the loss between the prediction and the target. We define $l(·,·)$ as the Mean Squared Error (MSE) for regression problems (our video prediction tasks), and as the cross entropy for classification problems (our other tasks). In all tasks, the training objective is minimised by Adam [57] with a learning rate of 0.001. Forget gate biases are set to 4 for image classification tasks and 1 [58] for others. All models are implemented by Torch7 [59] and accelerated by cuDNN on Tesla K80 GPUs (NVIDIA, Santa Clara, CA, USA).

We only apply CN to the output of the tLSTM hidden state as we have tried different combinations and found this is the most robust way that can always improve the performance for all tasks. With CN, the output of hidden state becomes:

$${\mathit{H}}_t=\varphi \left(\mathrm{CN}\left({\mathit{C}}_t;\mathsf{\Gamma },\mathit{B}\right)\right)\odot \mathit{O}.$$

(31)

4.1. Text Generation

The dataset of Hutter Prize Wikipedia [60] is a text file comprising 100 million characters with a vocabulary size of 205, including alphabets, special symbols, and XML markups. This dataset is modelled at character-level, and the goal is generating the next character given all previous ones, e.g.,:

$$\begin{array}{cc}Input:\hfill & \hfill [[\mathtt{J}\mathtt{o}\mathtt{a}\mathtt{c}\mathtt{h}\mathtt{i}\mathtt{m}\mathtt{V}\mathtt{a}\mathtt{d}\mathtt{i}\mathtt{a}\mathtt{n}]],\mathtt{S}\mathtt{w}\mathtt{i}\mathtt{s}\mathtt{s}\mathtt{h}\mathtt{u}\mathtt{m}\mathtt{a}\mathtt{n}\mathtt{i}\mathtt{s},\\ Target:\hfill & \hfill [\mathtt{J}\mathtt{o}\mathtt{a}\mathtt{c}\mathtt{h}\mathtt{i}\mathtt{m}\mathtt{V}\mathtt{a}\mathtt{d}\mathtt{i}\mathtt{a}\mathtt{n}]],\mathtt{S}\mathtt{w}\mathtt{i}\mathtt{s}\mathtt{s}\mathtt{h}\mathtt{u}\mathtt{m}\mathtt{a}\mathtt{n}\mathtt{i}\mathtt{s}\mathtt{t}.\end{array}$$

We evaluate all configurations for the depth $L=1,2,3,4$ and use 10 M parameters, so that the channel size M for sLSTM and 1-tLSTM–F is 1120, for other 1-tLSTMs is 901, and for 2-tLSTMs is 522. Bits-per-character (BPC) are used for performance measuring. As in [33], we split the dataset into 90 M/5 M/5 M for training/validation/test. In each iteration, the model is fed with a mini-batch of 100 subsequences of length 50. During the forward pass, the hidden values at the last time step are preserved to initialise the next iteration. We terminate training after 50 epochs.

Figure 4 shows the results. With a larger M, sLSTM and 1-tLSTM–F perform better than other models when $L\le 2$. When L increases, sLSTM and 1-tLSTM–M boost their performances but get stuck when $L\ge 3$, whereas, with the memConv, the performances of tLSTMs improve, finally surpassing sLSTM and 1-tLSTM–M. With $L=4$, the performance of 1-tLSTM–F is exceeded by that of 1-tLSTM, which is exceeded by that of 2-tLSTM in turn. Whilst LN benefits 2-tLSTM only when $L\le 2$, CN consistently benefits 2-tLSTM with different L.

Note that, in each tLSTM configuration, the runtime is nearly constant and largely unaffected by L, while in sLSTM, the runtime is almost proportional to L.

To compare with the state-of-the-art methods, we evaluate a larger 2-tLSTM+CN on the test set, where $L=6$ and $M=1200$. The results are presented in

Table 1. Test BPCs for Wikipedia text generation.

Method	#Parameters	BPC
MI-LSTM [26]	≈17 M	1.44
mLSTM [29]	≈20 M	1.42
HyperLSTM+LN [37]	26.5 M	1.34
HM-LSTM+LN [61]	≈35 M	1.32
Large RHN [47]	≈46 M	1.27
Large FS-LSTM-4 [50]	≈47 M	1.245
2 × Large FS-LSTM-4 [50]	≈94 M	1.198
2-tLSTM+CN ($L=6$, $M=1200$)	50.1 M	1.264

. With 50.1 M parameters, our model achieves a BPC of 1.264, and is therefore competitive to the best results [47,50] with similar amount of parameters.

4.2. Text Calculation

(i)

Addition: The goal of this task is adding two integers of 15-digit. The model firstly reads both integers, after which it predicts their sum, both in a sequential manner (i.e., one digit per time step). Following [46], we use the symbol ‘-’ to delimit integers and pad the input and target sequences, e.g.,

$$\begin{array}{cc}Input:\hfill & \hfill -\mathtt{694104857461284}-\mathtt{930283746529103}-----------------\\ Target:\hfill & \hfill --------------------------------\mathtt{1624388603990387}-\end{array}$$

(ii)

Copy: The copy task is to reproduce 20 random symbols presented as a sequence, where 65 different symbols are used. As in the addition task, the symbol ‘-’ is also used as a delimiter, e.g.,

$$\begin{array}{cc}Input:\hfill & \hfill -\mathtt{7}\mathtt{h}@\mathtt{P}\}\mathtt{n}\$\mathtt{R}\&+\mathtt{0}^(\#\mathtt{4}?\mathtt{w}>\mathtt{5}\mathtt{C}---------------------\\ Target:\hfill & \hfill ---------------------\mathtt{7}\mathtt{h}@\mathtt{P}\}\mathtt{n}\$\mathtt{R}\&+\mathtt{0}^(\#\mathtt{4}?\mathtt{w}>\mathtt{5}\mathtt{C}-\end{array}$$

For the addition and copy tasks, we set M to 400 and 100, respectively, and evaluate each configuration for $L=1,4,7,10$. The prediction accuracy of symbols are used to measure the performance. Like in [46], for both tasks we randomly generate 5 M training samples and 100 test samples, and set the mini-batch size to 15. Training proceeds for at most one epoch (To simulate the online learning process, we use all training samples only once) and will be terminated if $100\%$ test accuracy is achieved.

Results are shown in Figure 5. In both tasks, the performances of sLSTM and 1-tLSTM–M degrades with larger L. On the contrary, with L increasing, the performance of 1-tLSTM–F continues improving, and can be further boosted by using feedback, tensors of higher dimensionality, and CN, whilst LN improves the performance only when $L=1$. Note that correct solutions can be found (when achieving $100\%$ test accuracies) in both tasks because of their repetitive nature. From the experiments, we find that in the task of addition, 2-tLSTM+CN of $L=7$ performs the best and solves the task using only 298 K training examples, whilst in the task of copy, 2-tLSTM+CN of $L=10$ outperforms other configurations and copies perfectly using only 54 K training examples. Moreover, different from sLSTM, all tLSTMs’ runtime can be largely independent of L.

On both tasks, the best performing configurations are further compared to the state-of-the-art methods.

Table 2. Test accuracies for addition/copy.

Method	Addition		Copy
	#Samples	Accuracy	#Samples	Accuracy
sLSTM [45]	5 M	51%	900 K	>50%
Grid LSTM [46]	550 K	>99%	150 K	>99%
2-tLSTM+CN ($L=7$)	298 K	>99%	115 K	>99%
2-tLSTM+CN ($L=10$)	317 K	>99%	54 K	>99%

reports the results. Our model solves the tasks of both addition and copy significantly faster (with fewer training examples) than others, being the new state-of-the-art.

4.3. Image Classification

The dataset of MNIST [31] comprising 70,000 handwritten digit images sized $28\times 28$, which is divided into 50,000/10,000/10,000 for training/validation/test. For this dataset, there are two tasks:

(i)

Sequential MNIST: In this task, the model first sequentially reads the pixels in a scanline order, and then outputs the class of the digit contained in the image [62]. It is a time series task of 784 time steps, where we generate the output from the last time step, thereby requiring to capture very long term temporal dependencies.

(ii)

Sequential Permuted MNIST: To make the problem even harder, we generate a permuted MNIST (pMNIST) [63] by permuting the original image pixels with a fixed random order so that the long-term dependency can also exist in neighbouring pixels.

In both tasks, we evaluate all configurations with $M=100$ and $L=1,3,5$. We employ the classification accuracy to measure the model performance. We set the mini-batch size to 50, and use early stopping for training. The training loss is calculated at the last time step.

Figure 6 shows the results. Increasing the depth no longer benefits sLSTM and 1-tLSTM–M when $L=5$, while the performance of 1-tLSTM can be boosted by a larger depth and tensorisation. However, the performance of 1-tLSTM seems not to be affected by removing the feedback connections. In addition, CN always improves 2-tLSTM and outperforms LN when $L\ge 3$. With validation accuracies of 99.1% on MNIST and 95.6% on pMNIST, 2-tLSTM+CN with $L=5$ outperforms all other configurations in both tasks. In tLSTMs, the runtime is little affected by L, and when $L=5$, all tLSTMs runs faster than sLSTM.

As presented in

Table 3. Test accuracies (%) for sequential MNIST/pMNIST image classification.

Method	MNIST	pMNIST
iRNN [62]	97.0	82.0
LSTM [63]	98.2	88.0
uRNN [63]	95.1	91.4
Full-capacity uRNN [64]	96.9	94.1
sTANH [65]	98.1	94.0
BN-LSTM [66]	99.0	95.4
Dilated GRU [56]	99.2	94.6
Dilated CNN [54] in [56]	98.3	96.7
2-tLSTM+CN ($L=3$)	99.2	94.9
2-tLSTM+CN ($L=5$)	99.0	95.7

, the best performing configurations are compared against the state-of-the-art methods. On sequential MNIST, 2-tLSTM+CN with $L=3$ achieves 99.2% test accuracy, which is the same as the state-of-the-art one produced by the Dilated GRU [56]. On sequential pMNIST, 2-tLSTM+CN with $L=5$ achieves 95.7% test accuracy, approaching the state-of-the-art one of 96.7% which is obtained from the Dilated CNN [54] in [56].

4.4. Video Prediction

The task of video prediction aims at predicting the future frames of a video given the historical frames. It has a variety of applications such as environment simulation, dataset augmentation, and many computer vision tasks. The main challenge is that the model must capture both the spatial and the temporal relationships among data well. We apply our model to two datasets:

(i)

KTH [67]: The dataset consists of 600 real videos with 25 subjects performing six actions (walking, running, jogging, hand-clapping, hand-waving, and boxing). It has been split into a training set (subjects 1–16) and a test set (subjects 17–25), resulting in 383 and 216 sequences, respectively. We resize all frames to $128\times 128$.

(ii)

UCF101 [68]: The dataset consists of 13,320 real videos of resolution $320\times 240$ with 101 human actions that could be split into five types (sports, playing musical instruments, human-human interaction, body-motion only, and human-object interaction). It is currently the most challenging dataset of actions. Following [69], we train our models on Sports-1M [70] dataset and test them on UCF-101.

On both tasks, we evaluate all configurations with $L=1,3,5$. To process the structured inputs (i.e., video frames), we modify the original sLSTM [45] by replacing each LSTM layer with a Convolutional LSTM [39], where the convolution kernel size is set to $[5,5]$. We also set the last two dimensions (relevant to image structure) of the convolution kernel size $K$ to 5 for tLSTMs. M is set to 100 for KTH and 200 for UCF101. The model performance is measured by three common metrics including MSE, Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM) [71], where SSIM ranges in $[-1,1]$ (larger is better). We set the mini-batch size to 16 and employ early stopping for training. All models are trained by observing 10 frames and predicting the next 10 frames.

Figure 7 shows the quantitative results. When L increases, sLSTM and 1-tLSTM–M improve their performances but get stuck at $L=5$, while with the memConv, the performances of tLSTMs improve and finally exceed both sLSTM and 1-tLSTM–M. The effects of feedback and tensorisation become significant when L is large. Similar to the finding in [38] that LN is not suitable for normalising the convolution layer for images, the performance of 2-tLSTM+LN is even worse than 2-tLSTM. However, CN improves 2-tLSTM with different L. Unlike sLSTM where the runtime increases linearly w.r.t. L, tLSTM can keep its runtime largely unchanged when increasing L.

The best performing configuration is compared with the state-of-the-art methods (their source codes are publicly available) on both datasets (see

Table 4. Test performances for KTH/UCF101 video prediction.

Method	KTH			UCF101
	MSE ↓	PSNR ↑	SSIM ↑	MSE ↓	PSNR ↑	SSIM ↑
Composite LSTM [72]	0.01021	20.893	0.77958	0.16342	9.877	0.50363
Beyond MSE [69]	0.00193	28.465	0.88234	0.00987	22.133	0.81254
PredNet [73]	0.00384	27.954	0.90052	0.01672	18.945	0.80827
MCnet [12]	0.00190	30.179	0.91228	0.00979	22.861	0.84392
2-tLSTM+CN ($L=5$)	0.00148	31.053	0.93316	0.00925	22.785	0.85941

). 2-tLSTM+CN with $L=5$ outperforms all existing models on KTH w.r.t. all metrics, and on UCF101 w.r.t. MSE and SSIM. Sampled qualitative results produced by 2-tLSTM+CN are shown in Figure 8.

4.5. Analysis

It can been seen from the experiments that one can boost the performance of tLSTMs by enlarging the tensorised size or increasing the model depth, whereas almost no extra parameters and runtime are required. The memConv is indispensable to maintain the performance improvement when the network gets wider and deeper. In addition, for tasks with sequential output, feedback connections are useful. In addition, tensorisation or CN can further strengthen the tLSTM.

To inspect the inner working of our tLSTM, the value of memory cells are visualised to show the information routing. On each task, we run the best performing tLSTM with a random sample (here we do not consider video prediction tasks where memory cells of the 2-tLSTM are 5D tensors, which are hard to visualise). At each time step, we record the memory cell’s channel mean (computed by averaging along the channel dimension, for the 2-tLSTM, it has a size of $P\times P$), and visualise its diagonal values from location ${\mathit{p}}^{in}=[1,1]$ (close to the input) to ${\mathit{p}}^{out}=[P,P]$ (close to the output).

As shown in Figure 9, the visualisation result reveals different behaviors of tLSTM when handling different tasks:

• Text Generation: If the next character is largely determined by the current input, the input content can be preserved with less modification when it arrives at the output location, and vice versa.
• Addition: Two integers are gradually compressed into the memory and then interact with each other, generating their summation.
• Copy: The model acts as a shift register, continuing to move the input symbols to their output locations.
• Seq. MNIST: The model seems more sensitive to pixel value changes (which represent contours, or the digit topology); it gradually accumulates evidence to generate the final output.
• Seq. pMNIST: The model seems more sensitive to high value pixels (which come from the digit); our conjecture is that the permutation has destroyed the digit topology, and thereby made each high value pixel potentially important.

In these tasks, there are also some common phenomena:

• At each time step, different locations of the tensor possess markedly different values, which implies that a tensor of a larger size could encode more content, requiring less effort for compressing.
• The value becomes more and more distinct from the input to output and is shifted along the time axis, which reveals that the model indeed simultaneously performs the deep and temporal computations, with the memory cell carrying the long-term dependency.

5. Conclusions

In this paper, we have aimed to deal with multimedia modelling tasks. We have introduced the tLSTM, where tensors are employed to share parameters and temporal computations are utilised to perform deep computations. The main advantage of our tLSTM over other popular methods is that its capacity can be increased with almost no extra parameters and runtime. Another important advantage of the tLSTM is that it can handle a variety of challenging multimedia modelling tasks well as shown in our experiments.

For future work, we would like to: (i) investigate more about the effect of higher-dimensional tensors, e.g., try 3- and 4-tLSTMs, (ii) try increasing the transition depth for tLSTM hidden states (similar to [47]); and (iii) apply tLSTMs to more multimedia modelling tasks such as machine translation, image generation, and video segmentation.