Data-Driven Predictive Modeling of Neuronal Dynamics Using Long Short-Term Memory

Abstract

Modeling brain dynamics to better understand and control complex behaviors underlying various cognitive brain functions have been of interest to engineers, mathematicians and physicists over the last several decades. With the motivation of developing computationally efficient models of brain dynamics to use in designing control-theoretic neurostimulation strategies, we have developed a novel data-driven approach in a long short-term memory (LSTM) neural network architecture to predict the temporal dynamics of complex systems over an extended long time-horizon in future. In contrast to recent LSTM-based dynamical modeling approaches that make use of multi-layer perceptrons or linear combination layers as output layers, our architecture uses a single fully connected output layer and reversed-order sequence-to-sequence mapping to improve short time-horizon prediction accuracy and to make multi-timestep predictions of dynamical behaviors. We demonstrate the efficacy of our approach in reconstructing the regular spiking to bursting dynamics exhibited by an experimentally-validated 9-dimensional Hodgkin-Huxley model of hippocampal CA1 pyramidal neurons. Through simulations, we show that our LSTM neural network can predict the multi-time scale temporal dynamics underlying various spiking patterns with reasonable accuracy. Moreover, our results show that the predictions improve with increasing predictive time-horizon in the multi-timestep deep LSTM neural network.

1. Introduction

Our brain generates highly complex nonlinear responses at multiple temporal scales, ranging from few milliseconds to several days, in response to an external stimulus [1,2,3]. One of the long-time interests in computational neuroscience is to understand the dynamics underlying various cognitive and non-cognitive brain functions by developing computationally efficient modeling and analysis approaches. In the last four decades or so, several advancements have been made in the direction of dynamical modeling and analysis of brain dynamics [4,5,6]. In the context of modeling the dynamics of single neurons, several modeling approaches, ranging from detailed mechanism-based biophysiological modeling to simplified phenomenological/probabilistic modeling, have been developed to understand the diverse firing patterns (e.g., simple spiking to bursting) observed in electrophysiological experiments [7,8]. These models provide a detailed understanding of various ionic mechanisms that contribute to generating specific spiking patterns as well as allowing the performance of large-scale simulations to understand the dynamics underlying cognitive behaviors. However, most of these models are computationally expensive from the perspective of developing novel real-time neurostimulation strategies for controlling neuronal dynamics at single neurons and network levels. In this paper, we investigate purely data-driven long short-term memory (LSTM) based recurrent neural network (RNN) architectures in multi-timestep predictions of a single neuron’s dynamics for the use in developing novel neurostimulation strategies in an optimal control framework.

The availability of an abundant amount of data and advances in machine learning has recently revolutionized the field of predictive data-driven dynamical modeling of complex systems using neural networks (NNs) and deep learning approaches. Various nonlinear system identification approaches have been developed to map static input-output relations using multi-layer perceptrons (MLPs) [9,10,11,12] and their variations [13,14]. Reinforcement learning has recently been explored in robotics dynamical modeling in Reference [15]. NN architectures that make use of vanilla recurrent neural network (RNNs) elements have also been explored for nonlinear system identification and modeling in References [16,17,18]. However, network architectures that make use of vanilla recurrent layers often suffer from the exploding or vanishing gradient problem when used to model dynamics over long time series horizons [19]. In Reference [18], a highly specialized multi-phase training algorithm was used to ensure that the network did not suffer from this problem. LSTM based approaches to modeling dynamical systems [20,21,22] mitigate the vanishing gradient problem but suffer from poor early trajectory predictive performance when using long predictive horizons [20,23]. LSTMs have been used to model high-dimensional chaotic systems [24] but these studies have been limited to single step prediction applications. Additionally, machine learning techniques have begun to be explored in neuroscientific modeling applications. Multi-layer Spiking Artificial Neural Networks (SANNs) for use in spatiotemporal spike pattern transformations have been developed in References [25,26]. This is achieved by using novel approximations and surrogates of the partial derivatives of the spike train functions with respect to the weights. This partial derivative is typically problematic when used in backpropagation, as it is undefined at spike times for many neuronal models, rendering it incompatible with traditional backpropagation-based approaches. In Reference [27], a novel gated recurrent unit (GRU) based encoder/decoder approach is used to learn and predict neuronal population dynamics and kinematic trajectories from single-trial spike train data.

In this paper, we have developed a novel deep LSTM neural network architecture, which can make multi-timestep predictions in large-scale dynamical systems. In particular, we use a reversed sequence-to-sequence mapping technique, developed for language translation applications of multi-layer LSTM networks in Reference [28], and generalize the application of this technique to dynamical systems time-series forecasting. Figure 1 illustrates our overall approach.

In contrast to existing approaches in modeling dynamical systems using neural networks, our architecture uses (1) stacked LSTM layers in conjunction with a single densely connected layer to capture temporal dynamic features as well as input/output features; (2) sequence-to-sequence mapping, which enables multi-timestep predictions; and (3) reverse ordered input and measured state trajectories to the network, resulting in highly accurate early predictions and improved performance over long horizons. We show the efficacy of our developed approach in making stable multi-timestep predictions of various firing patterns exhibited by hippocampal CA1 pyramidal neurons, obtained from simulating an experimentally validated highly nonlinear 9-dimensional Hodgkin-Huxley model of CA1 pyramidal cell dynamics, over long time-horizons. Our approach is contingent on the network being trained on the entire state vector of the neuronal model.

The remaining paper is organized as follows. In Section 2, we describe our developed deep LSTM neural network architecture and methodological approach to data-driven multi-timestep predictions of dynamical systems. We show the efficacy of our approach in making stable multi-timestep predictions over long time-horizons of neuronal dynamics in Section 3, which is followed by a thorough discussion on the limitations of our approach in Section 4.

2. Neural Network Architecture, Algorithm and Approach

In Section 2.1, we describe our developed deep LSTM neural network architecture which combines stacked LSTMs with a fully-connected dense output layer. We describe the sequence-to-sequence mapping with reversed order input sequences used in this paper in Section 2.2. In Section 2.3, we provide the details on the synthetic data used to train our networks. Finally, in Section 2.4, we provide the details on the approach used to train the developed neural network architecture.

2.1. Deep LSTM Neural Network Architecture

Long short-term memory (LSTM) neural networks [29] are a particular type of recurrent neural networks (RNNs) which mitigate the vanishing or exploding gradient problem during the network training while capturing both the long-term and the short-term temporal features in sequential time-series data processing [19]. Specifically, LSTM uses multiple gating variables that control the flow of information of a hidden cell state and assign temporal importance to the dynamical features that are present in the time series data flowing through the cell state. Figure 2 shows a schematic illustrating the internal gating operation in a single LSTM cell.

A forward pass of information through a single LSTM cell is described by the following cell and gating state equations (reference):

$$c_t=f_t\circ c_{t-1}+i_t\circ {\sigma }_c(W_c{x}_t+U_c{h}_{t-1}+b_c),$$

(1a)

$$h_t=o_t\circ {\sigma }_c\left(c_t\right).$$

(1b)

$$f_t={\sigma }_g(W_f{x}_t+U_f{h}_{t-1}+b_f),$$

(1c)

$$i_t={\sigma }_g(W_i{x}_t+U_i{h}_{t-1}+b_i),$$

(1d)

$$o_t={\sigma }_g(W_o{x}_t+U_o{h}_{t-1}+b_o),$$

(1e)

in Equations (1a)–(1e), $c_t\in {\mathrm{I}\mathrm{R}}^h$ and $h_t\in {\mathrm{I}\mathrm{R}}^h$ represent the cell state vector and the hidden state vector, respectively, at time t. $f_t\in {\mathrm{I}\mathrm{R}}^h$, $i_t\in {\mathrm{I}\mathrm{R}}^h$, and $o_t\in {\mathrm{I}\mathrm{R}}^h$ are the “forget gate’’, “input gate’’, and “output gate’’ activation vector, respectively, at time t. $x_t\in {\mathrm{I}\mathrm{R}}^d$ is the input vector to the LSTM unit at time t, and $h_{t-1}\in {\mathrm{I}\mathrm{R}}^h$ is the previous time step hidden state vector passed back into the LSTM unit at time t. The matrices $W_f$, $W_i$, and $W_o$ represent the input weights for the “forget gate’’, “input gate’’, and “output gate’’, respectively. The matrices $U_f$, $U_i$ and $U_o$ represent the weights of the recurrent connections for the “forget gate’’, “input gate’’, and “output gate’’, respectively. The vectors $b_f$ $b_i$, and $b_o$ represent the “forget gate’’, “input gate’’, and “output gate’’ biases, respectively. ∘ represents the element-wise multiplication. The function ${\sigma }_g$ represents the sigmoidal activation function, and ${\sigma }_c$ is the hyperbolic tangent activation functions.

In this paper, we use stacked LSTM network integrated with a fully connected feedforward output layer to make multi-timestep state predictions. The use of a single feedforward dense output layer allows the network to effectively learn the static input-output features, while the stacked LSTM network captures the temporal dynamical features. To appropriately select the optimum dimensionality of the hidden states in a single hidden layer, we systematically varied the number of hidden states in a sequence of {n, $n^2$, $2n^2$, $4n^2$, ⋯}, where n is the dimension of the system’s state and evaluated the training performance for each case. We found that for our application ($n=9$), a hidden state dimensionality of $4n^2=324$ was optimal in learning dynamical behaviors while avoiding overfitting. To select the number of hidden layers, we systematically increased the number of hidden layers of identical hidden state dimensionality (i.e., 324 states) and compared the network performance during the training. We found that increasing the number of hidden layers beyond 3 layers did not improve the network performance on the training and validation dataset. Thus, we fixed the number of hidden layers to 3 in our study. Throughout this paper, we utilized stateless LSTMs which reset the internal cell and hidden states to zero after processing and performing gradient descent for a given minibatch. We initialized the network weights using the Xavier method [30]. Specifically, the initial weights were drawn from a uniform distribution using

$$W_{ij}\sim \mathcal{U}\left(-\frac{6}{\sqrt{n_j+n_{j+1}}},\frac{6}{\sqrt{n_j+n_{j+1}}}\right),$$

(2)

where $n_j$ is the dimensionality of the input units in the weight tensor, and $n_{j+1}$ is the dimensionality of the output units in the weight tensor.

To generate a long time-horizon dynamical prediction beyond the multi-timestep prediction by a single stacked deep LSTM neural network (shown as “Deep LSTM” in Figure 3), we used an iterative approach as described here. We made copies of the trained single stacked LSTM network and connected them in the feedforward manner in a sequence. We concatenated the sequence of predicted output from the previous stacked LSTM network with an equivalent length sequence of new inputs to the system and fed them in the reverse sequence order to the next stacked LSTM network. Figure 3 illustrates this iterative approach.

2.2. Sequence to Sequence Mapping with Neural Networks

To make multi-timestep predictions of dynamical systems’ outputs using the deep LSTM neural network architecture described in the previous section (Section 2.1), we formulate the problem of mapping trajectories of the network inputs to the trajectories of the predicted outputs as a reverse order sequence-to-sequence mapping problem. The central idea of the reverse order sequence-to-sequence mapping approach is to feed the inputs to the network in reverse order such that the network perceives the first input as the last and the last input as the first. Although this approach has been developed and applied in language translation applications [28], it has never been considered in the context of predicting dynamical systems behaviors from time-series data. Figure 4 illustrates the basic idea of the reverse order sequence-to-sequence mapping approach for translating letters (inputs) to their numerical indices (outputs).

As shown in Figure 4, in the forward sequence-to-sequence mapping approach (Figure 4a), that is, $A,B,C\to 1,2,3$, the distance between all mappings is same (i.e., 3 “units”). In the reverse sequence-to-sequence mapping approach (Figure 4b), the network receives the input in a reverse order to map to the target output sequence, i.e., $C,B,A\to 1,2,3$. As noted here, the average distance between the mappings remains the same for both approaches (i.e., 3 “units”) but the reverse order approach introduces short and long-term symmetric temporal dependencies between inputs and outputs. These short and long-term symmetric temporal dependencies provide improved predictive performance over long temporal horizons [28].

2.3. Synthetic Data

Hippocampal CA1 pyramidal neurons exhibit various multi-timescale firing patterns (from simple spiking to bursting) and play an essential role in shaping spatial and episodic memory [31]. In the last two decades, several biophysiological models of the CA1 pyramidal (CA1Py) neurons ranging from single compartmental biophysiological and phenomenological models [32,33,34] to detailed morphology-based multi-compartmental models [35,36,37,38,39,40,41] have been developed to understand the contributions of various ion-channels in diverse firing patterns (e.g., simple spiking to bursting) exhibited by the CA1Py neurons.

In this paper, we use an experimentally validated 9-dimensional nonlinear model of CA1 pyramidal neuron in the Hodgkin-Huxley formalism given in Reference [32] to generate the synthetic data for the network training and validation. The model exhibits several different bifurcations to the external stimulating current and has shown its capability in generating diverse firing patterns observed in electrophysiological recordings from CA1 pyramidal cells under various stimulating currents. Figure 5 shows three different firing patterns generated from this model based on the three different regimes of the applied input currents.

To construct the synthetic training and validation dataset for the deep LSTM neural networks we designed in this paper with different predictive horizons, we simulated the Hodgkin-Huxley model of CA1 pyramidal neuron given in [32] (see Appendix A for the details of the model) for 1000 ms duration for 2000 constant stimulating currents, sampled uniformly between $I=0.0$ nA and $I=3.0$ nA. From these 2000 examples, we randomly and uniformly drew 50 samples (i.e., ${10}^4$ data points) of the desired predictive horizon as the input/output sequence data for training and validation. As described in Section 2.4, we used 1/32 of these data points for validation, that is, 96,875 data points for the training and 3125 data points for the validation. Since our deep LSTM neural network takes an initial sequence of outputs of appropriate predictive horizon length (i.e., $N_p=1,50,100,200$) as an input sequence to make the next time-horizon prediction of equivalent length of sequence, we assume that this initial output sequence data is available to the deep LSTM neural network throughout our simulations.

2.4. Network Training

We formulated the following optimization problem to train a set of network weights $\theta$:

$${\theta }^{*}=\underset{\theta }{arg\; min}\mathcal{L}\left(\theta \right),$$

(3)

where the loss function $\mathcal{L}\left(\theta \right)$ is given by

$$\mathcal{L}\left(\theta \right)=\frac{1}{N_P}\sum _{k=0}^{N_P}{(\overrightarrow{x}\left(k\right)-\widehat{x}\left(k\right|\theta ))}^T(\overrightarrow{x}\left(k\right)-\widehat{x}\left(k\right|\theta )).$$

(4)

Here $N_P$ represents the length of horizon over which the predictions are made, $\overrightarrow{x}\left(k\right)$ is the known state vector at time step k, and $\widehat{x}\left(k\right|\theta )$ is the neural network’s prediction of the state vector at time k, given $\theta$.

To solve the optimization problem (3) and (4), we used the standard supervised backpropagation learning algorithm [42,43,44] along with the Adaptive Moment Estimation (Adam) method [45]. The Adam method is a first-order gradient-based optimization algorithm and uses lower-order moments of the gradients between layers to optimize a stochastic objective function.

Given the network parameter ${\theta }^{\left(i\right)}$ and the loss function $\mathcal{L}\left(\theta \right)$, where i represents the algorithm’s training iteration, the parameter update is given by Reference [45]

$$\begin{array}{c}\hfill m_{\theta }^{(i+1)}\leftarrow {\beta }_1{m}_{\theta }^{\left(i\right)}+(1-{\beta }_1){\nabla }_{\theta }{\mathcal{L}}^{\left(i\right)},\end{array}$$

(5)

$$\begin{array}{c}\hfill {\nu }_{\theta }^{(i+1)}\leftarrow {\beta }_2{m}_{\theta }^{\left(i\right)}+(1-{\beta }_2){\left({\nabla }_{\theta }{\mathcal{L}}^{\left(i\right)}\right)}^2,\end{array}$$

(6)

$$\begin{array}{c}\hfill {\widehat{m}}_{\theta }=\frac{m_{\theta }^{(i+1)}}{1-{\left({\beta }_1\right)}^{i+1}},\end{array}$$

(7)

$$\begin{array}{c}\hfill {\widehat{\nu }}_{\theta }=\frac{{\nu }_{\theta }^{(i+1)}}{1-{\left({\beta }_2\right)}^{(i+1)}},\end{array}$$

(8)

$$\begin{array}{c}\hfill {\theta }^{(i+1)}\leftarrow {\theta }^{\left(i\right)}-\eta \frac{{\widehat{m}}_{\theta }}{\sqrt{{\widehat{\nu }}_{\theta }}+ϵ},\end{array}$$

(9)

where $m_{\theta }$ is the first moment of the weights in a layer, ${\nu }_{\theta }$ is the second moment of the weights in a layer, $\eta$ is the learning rate, ${\beta }_1$ and ${\beta }_2$ are the exponential decay rates for the moment estimates, ∇ is the differential gradient operator, and $ϵ$ is a small scalar term to help numerical stability. Throughout this work, we used ${\beta }_1=0.9$, ${\beta }_2=0.999$ and $\eta =0.001$ [45].

It should be noted that there is a trade-off between the predictive time-horizon of deep LSTM neural network and the computational cost involved in training the network over the predictive horizon. As the predictive horizon increases, the computational cost of training the network over that horizon increases significantly for an equivalent number of examples. To keep the computational tractability in our simulations, all networks with long predictive horizons (i.e., $N_P=50,100,200$) were trained for 200 epochs except the one-step predictive network, which was trained for 1000 epochs.

For all training sets throughout this paper, we used the validation to training data ratio as 1/32. We set the minibatch size for training to 32. We performed all the training and computation in the TensorFlow computational framework on a discrete server running CentOS 7 with twin Nvidia GTX 1080Ti GPUs equipped with 11 Gb of VRAM.

3. Simulation Results

In this section, we present our simulation results on predicting the multi-timescale spiking dynamics exhibited by hippocampal CA1 pyramidal neurons over a long time-horizon using our developed deep LSTM neural network architecture described in Section 2. We trained 4 LSTM networks for making one timestep prediction (equivalently, $0.1$ ms), 50 timesteps prediction (equivalently, 5 ms), 100 timesteps prediction (equivalently, 10 ms), and 200 timesteps prediction (equivalently, 20 ms). Figure 6 shows the training and validation loss for these 4 LSTM networks.

Using the iterative approach described in Section 2.1, we simulated the LSTM networks over 500 ms of time duration under different initial conditions and stimulating input currents between three different regimes of dynamical responses (“Regular Spiking” ($I\in [2.3,3.0]$ nA), “Irregular Bursting” ($I\in [0.79,2.3)$ nA), and “Regular Bursting” ($I\in [0.24,0.79)$) nA) and compared the predicted state trajectories with the Hodgkin-Huxley model. We provide a link to download simulation codes to replicate our presented results in this section in the Supplementary Materials section.

3.1. Regular Spiking

In this section, we demonstrate the efficacy of our trained deep LSTM neural network over the range of external current between $2.3$ nA and 3 nA in predicting the regular spiking dynamics shown by the biophysiological Hodgkin-Huxley model of CA1 pyramidal neuron in response to the external current $I\ge 2.3$ nA. In this range, we observe firing rates between approximately 165 Hz and 187 Hz. For clarity, we here show our results only for the membrane potential traces. We provide the complete set of simulation results on the LSTM network performance in predicting the dynamics of all the 9 states of the Hodgkin-Huxley model in Appendix B.1 (see Figure A1, Figure A2, Figure A3, Figure A4, Figure A5).

Figure 7 shows a comparison of the membrane potential traces simulated using the Hodgkin-Huxley model and the 4 different predictive horizons of the LSTM network (i.e., 1 timestep, 50 timesteps, 100 timesteps and 200 timesteps, which we represent as $N_p=1,50,100,200$) for the external stimulating current $I=3.0$ nA. Note that all the simulations are performed using the same initial condition as provided in Appendix A. Since our LSTM network uses the initial sequence of outputs of appropriate predictive horizon (i.e., $N_p=1,50,100,200$) from the Hodgkin-Huxley model to make future time predictions, the LSTM network predictions (shown by dashed red line) start after $0.1$ ms, 5 ms, 10 ms, and 20 ms in Figure 7a–d, respectively.

As shown in Figure 7, the iterative prediction of the membrane potential traces by the LSTM network did not differ significantly over a short time horizon (up to 200 ms) for $N_p=1,50,100,200$, but it significantly improved afterward with the increased predictive horizon of the LSTM network (i.e., $N_p=1$ to $N_p=200$). In particular, the LSTM network performance significantly improved in predicting the timing of the occurrence of spikes, but the magnitude of the membrane potential traces during spikes degraded as we increased $N_p$ from 1 to 200. For clarity, we also computed the time-averaged root mean squared error (RMSE) of the membrane potential traces between the Hodgkin-Huxley model and the LSTM network for $N_p=1,50,100,200$ over 500 ms of simulation time. Figure 8a shows that the time-averaged RMSE decreased consistently with the increased predictive horizon of the LSTM network. Overall, these results show that our LSTM network with a longer predictive horizon prefers to capture temporal correlations more accurately over the amplitude while an LSTM network with a shorter predictive horizon prefers to capture the amplitude more accurately over the temporal correlations.

To systematically evaluate whether the designed LSTM networks provide reasonable predictions of the membrane potential traces of the regular spiking dynamics across the range of external input currents between $2.3$ nA and $3.0$ nA, we performed simulations for 50 random stimulating currents drawn from a Uniform distribution $\mathcal{U}(2.3,3.0)$. For each stimulating current, we chose 100 initial conditions drawn randomly from the maximum and minimum range of the Hodgkin-Huxley state variables (Note that the network was not trained over this wide range of initial conditions). Figure 8b shows the LSTM network performance, represented in terms of the root mean squared error vs time over 5000 realizations, for $N_p=1,50,100,200$. As shown in this figure, the root mean squared error decreased with the increased predictive horizon of the LSTM network for all time, which is consistent with the result shown in Figure 8a.

In conclusion, these results suggest that our deep LSTM neural network with a longer predictive horizon feature can predict the regular (periodic) spiking patterns exhibited by hippocampal CA1 pyramidal neurons with high accuracy over a long-time horizon.

3.2. Irregular Bursting

In this section, we demonstrate the efficacy of our trained deep LSTM neural network over the range of external current between $0.79$ nA and $2.3$ nA in predicting the irregular bursting dynamics shown by the biophysiological Hodgkin-Huxley model of CA1 pyramidal neuron in response to the external current $I\in [0.79,2.3)$ nA. In this range, we observe firing rates between approximately 53 Hz and 164 Hz. For clarity, we here show our results only for the membrane potential traces. We provide the complete set of simulation results on the LSTM network performance in predicting the dynamics of all the 9 states of the Hodgkin-Huxley model in Appendix B.2 (see Figure A6, Figure A7, Figure A8, Figure A9, Figure A10).

Figure 9 shows a comparison of the membrane potential traces simulated using the Hodgkin-Huxley model and the 4 different predictive horizons of the LSTM network (i.e., $N_p=1,50,100,200$) for the external stimulating current $I=1.5$ nA. Note that all the simulations are performed using the initial condition used for $I=3.0$ nA in Figure 7. Since our LSTM network uses the initial sequence of outputs of appropriate prediction horizon (i.e., $N_p=1,50,100,200$) from the Hodgkin-Huxley model to make future time predictions, the LSTM network predictions (shown by dashed red line) start after $0.1$ ms, 5 ms, 10 ms, and 20 ms in Figure 9a–d, respectively.

As shown in Figure 9, the LSTM performance significantly improved in predicting the timing of the occurrence of spikes up to 100 ms with the increased predictive horizon of the LSTM network from $N_p=1$ to $N_p=200$, but the performance degraded in capturing the magnitude of the membrane potentials during spiking with an increased value of $N_p$. Although the time-averaged root mean squared error of the membrane potential traces between the Hodgkin-Huxley model and the LSTM network for $N_p=1,50,100,200$ showed an improved performance with the increased value of $N_p$ (see Figure 10a), none of the LSTM networks showed a reasonable prediction of the timing of the occurrence of spikes in this regime beyond 100 ms of the time-horizon.

To systematically evaluate whether the designed LSTM networks provide reasonable predictions of the membrane potential traces of the regular spiking dynamics across the range of external input currents between $0.79$ nA and $2.3$ nA, we performed simulations for 50 random stimulating currents drawn from a Uniform distribution $\mathcal{U}(0.79,2.3)$. For each stimulating current, we chose 100 initial conditions drawn randomly from the maximum and minimum range of the Hodgkin-Huxley state variables (note that the network was not trained over this wide range of initial conditions). Figure 10b shows the LSTM network performance, represented in terms of the root mean squared error vs time over 5000 realizations, for $N_p=1,50,100,200$. As shown in this figure, the root mean squared error decreased with the increased predictive horizon of the LSTM network for all time, which is consistent with the result shown in Figure 10a.

In conclusion, these results suggest that our deep LSTM neural network with a longer predictive horizon feature can predict the irregular bursting patterns exhibited by hippocampal CA1 pyramidal neurons with high accuracy over only a short-time horizon.

3.3. Regular Bursting

In this section, we demonstrate the efficacy of our trained deep LSTM neural network over the range of external current between $0.24$ nA and $0.79$ nA in predicting the regular bursting dynamics shown by the biophysiological Hodgkin-Huxley model of CA1 pyramidal neuron in response to the external current $I\in [0.24,0.79)$ nA. In this range, we observe firing rates between approximately 8 Hz and 52 Hz. For clarity, we here show our results only for the membrane potential traces. We provide the complete set of simulation results on the LSTM network performance in predicting the dynamics of all the 9 states of the Hodgkin-Huxley model in Appendix B.3 (see Figure A11, Figure A12, Figure A13, Figure A14, Figure A15).

Figure 11 shows a comparison of the membrane potential traces simulated using the Hodgkin-Huxley model and the 4 different predictive horizons of the LSTM network (i.e., $N_p=1,50,100,200$) for the external stimulating current $I=0.5$ nA. Note that all the simulations are performed using the initial condition used for $I=3.0$ nA in Figure 7. Since our LSTM network uses the initial sequence of outputs of appropriate prediction horizon (i.e., $N_p=1,50,100,200$) from the Hodgkin-Huxley model to make future time predictions, the LSTM network predictions (shown by dashed red line) start after $0.1$ ms, 5 ms, 10 ms and 20 ms in Figure 11a–d, respectively.

By analyzing the results shown in Figure 11, we found that the LSTM network performance in predicting the timing of spikes during bursts as well as tracking the subthreshold potential improved significantly from $N_p=1$ to $N_p=200$, but the performance substantially degraded in capturing the magnitude of the membrane potentials during spiking. In conclusion, the 200 timesteps prediction horizon based LSTM network (see Figure 11d) predicts the temporal dynamics with reasonable accuracy over the first 300 ms of prediction.

Figure 12a shows the time-averaged root mean squared error of the membrane potential traces between the Hodgkin-Huxley model and the LSTM network for $N_p=1,50,100,200$. As noted in this figure, the root mean squared error decreased substantially between 100 timesteps and 200 timesteps prediction horizon compared to the regimes of regular spiking (Figure 8a) and irregular bursting (Figure 10a), which indicates that a longer predictive horizon based LSTM network is necessary to capture the two different timescales (i.e., short intraburst spiking intervals and long interburst subthreshold intervals) presented in these dynamics.

Figure 12b shows the LSTM networks performances, represented in terms of the root mean squared error vs time over 5000 realizations, for $N_p=$ 1, 100, and 200 timestep prediction horizon LSTM network. As shown in this figure, the root mean squared error decreased with the increased predictive horizon of the LSTM network for all time, which is consistent with the result shown in Figure 12a. Note that we have excluded the simulation result for $N_p=50$ as we found out in our detailed analysis that the trained LSTM network for $N_p=50$ led to instability in predicting spiking responses for some of the initial condition values in this regime. The reason for this may be that the network may not have seen these initial conditions during the training.

4. Discussion

In this paper, we developed and presented a new data-driven long short-term memory (LSTM) based neural network (NN) architecture to predict the dynamical spiking patterns of single neurons. Compared to other LSTM-based NN architectures for forecasting dynamical systems behavior reported in the literature, our architecture incorporated a single dense feedforward output layer with an activation function and a reverse-order sequence-to-sequence mapping approach into traditional LSTM based neural networks to enable truly multi-timestep stable predictions of the dynamics over a long time-horizon. We demonstrated the efficacy of our architecture in predicting the multi-time scale dynamics of hippocampal CA1 pyramidal neurons and compared the predictions from our model with the ground truth synthetic data obtained from an experimentally validated biophysiological model of CA1 pyramidal neuron in the Hodgkin-Huxley formalism. Through simulations, we showed that (1) the presented architecture can learn multi-timescale dynamics; and (2) the predictive accuracy of the network increases with the increase in the predictive horizon of the LSTM network.

Our results for irregular bursting regime showed the limitation of the designed deep LSTM neural network architecture in making an accurate prediction of the timing of the occurrence of spikes over a long-time horizon compared to regular spiking and regular bursting regimes. A possible reason for this may be the architecture itself or the dataset used for training these networks, which requires further investigations by training the networks on the dataset explicitly generated from this regime. In addition, it has been shown that this regime of bursting exhibits chaotic dynamics [46], which may provide further explanation for the network’s struggle to accurately predict this bursting behavior, as the system exhibits a high sensitivity to the initial conditions. Hybrid approaches that combine LSTM networks with mean stochastic models (MSM) have been explored for predictively modeling chaotic dynamical systems in Reference [24]. However, this LSTM-MSM approach is limited to iterative single step prediction, and the application of this technique falls beyond the scope of this paper.

Another limitation of our presented approach in modeling neuronal dynamics as currently constructed is the inclusion of the full state vector in both training and predictive evaluation. In experiment, it may be infeasible to have the entire state vector of the neuron measured for any given time. This should provide a valuable direction for future research, as partially observed systems or neuronal spike train recordings are much more feasible to measure in vivo or vitro and merit further consideration in combination with this approach.

In all dynamical regimes, our results showed a degraded performance of the deep LSTM neural network in predicting the amplitude of membrane potentials during the timing of the occurrence of spikes with the increased predictive horizon of the LSTM network. This issue may be related to the equally weighted norm-2 loss function used for training the networks. A further investigation is required by considering different loss functions, such as norm-1 or weighted norm-2, which we consider as our future work.

In addition, we make a note on the computational requirement of our presented approach. The computational cost of inference with an artificial neural network can effectively be boiled down to the number of multiplications and additions needed to complete a forward pass of information. The inference complexity for an LSTM is roughly $\mathcal{O}(d_i·d·h+d·h^2)$, as described in Reference [24], where $d_i$ is the dimensionality of each input, d is the number of inputs, and h is the dimensionality of the hidden states. Using this, we estimated the inference complexity of our LTSM network, represented by $\mathcal{O}\left(I\right)$, for the prediction horizon of 1, 50, 100, and 200. Using a single Nvidia GTX 1080Ti, we have also calculated the average computation time required for each of the predictive horizons used to predict 500 ms of state values from 1000 examples. We report these values in

Table 1. Comparison of computational requirement for the iterative approach presented in this paper.

${\mathit{N}}_{\mathit{P}}$ ($\mathbf{\Delta }\mathit{t}=0.1$ ms)	Prediction Time (ms)	Iterations	On-Line Computation Time (s)	$\mathcal{O}\left(\mathit{I}\right)$
1	500	5000	8.896	$5.3\times {10}^9$
50	500	100	3.778	$1.6\times {10}^8$
100	500	50	3.679	$1.1\times {10}^8$
200	500	25	3.565	$8.0\times {10}^7$

.

Although the data-driven approach developed in this paper showed the ability of the designed LSTM-based neural network in learning multi-timescale dynamics, we note that the network struggles to accurately capture the dynamics of some state variables where the magnitude of the state variable is comparable to the numerical precision of our simulations. This can particularly be seen in Figure A2, Figure A3, Figure A4 and Figure A14, where the network is not able to reconstruct the dynamics of the state variable $q_{sAHP}$ with a reasonable accuracy. One possible way to alleviate this issue may be to increase the tolerance of the numerical errors in simulations, which may increase the overall computational cost during training.

In conclusion, our results showed that a longer predictive horizon-based LSTM network can provide a more accurate prediction of multi-time scale dynamics, but at the expense of extensive offline training cost.