An Introduction to the Non-Equilibrium Steady States of Maximum Entropy Spike Trains

Abstract

Although most biological processes are characterized by a strong temporal asymmetry, several popular mathematical models neglect this issue. Maximum entropy methods provide a principled way of addressing time irreversibility, which leverages powerful results and ideas from the literature of non-equilibrium statistical mechanics. This tutorial provides a comprehensive overview of these issues, with a focus in the case of spike train statistics. We provide a detailed account of the mathematical foundations and work out examples to illustrate the key concepts and results from non-equilibrium statistical mechanics.

1. Introduction

Being the brain one of the most complex systems within the observable universe, it is not surprising that there is still a large number of unanswered questions related to its structure and functions. With the aim of developing new ways of addressing such questions, there is an increasing consensus among neuroscientists in that interdisciplinary approaches are promising. As a prominent example of this, computational neuroscience has been greatly enriched during the last decades by tools, ideas and methods coming from statistical physics [1,2]. Moreover, these methods are recently being revisited with renewed interest due to the arrival of experimental techniques that generate huge volumes of data. In particular, neuroscientists have become progressively aware of the powerful computational techniques used by statistical physicists to analyze experimental data and large scale simulations.

When studying the firing patterns of collections of neurons, one of the most popular principles from statistical mechanics is the maximum entropy principle (MEP), which builds the least structured model that is consistent with average values measured from experimental data. These average values are usually restricted to firing rates and synchronous pairwise correlations, which gives rise to models composed by time independent and identically distributed (i.i.d) random variables, i.e., stochastic processes without temporal structure [3,4,5]. Needless to say, there exists strong evidence in favour of memory effects playing a major role in spike train statistics, and biological process in general [6,7,8,9]. Following this evidence, over the last years the study of complex biological systems has started to consider time-dependent processes where the past has an influence on future behavior [10,11,12]. The corresponding asymmetry between past and future is called the “arrow of time”, which is the unique direction associated with the irreversible flow of time that is noticeable in most biological systems.

Interestingly, the statistical physics literature has a fertile toolkit for studying time asymmetric processes [13]. First, one introduces the distinction between steady states that imply thermal equilibrium, and steady states that still carry fluxes—being called non-equilibrium steady states (NESS). Additionally, the extent to which a steady-state is not in equilibrium (i.e., the strength of its associated currents) can be quantified by the entropy-production rate [14], which is associated with the degree of time-irreversibility in the corresponding process [14]. Several studies have pointed out that being out-of-equilibrium is an important characteristic of biological systems [15,16,17]. Therefore, statistical characterizations consistent with the out-of-equilibrium condition should reproduce some degree of time irreversibility. One popular method that is suitable for studying these issues is Markov chain modeling [11,18,19,20,21,22].

Despite the potential of interdisciplinary pollination related to these fascinating issues, many scientists find it hard to explore these topics because of the major entry barriers, including differences in jargon, conventions, and notations across the various fields. To bridge this gap, this tutorial intends to provide an accessible introduction to the non-equilibrium properties of maximum entropy Markov chains, with an emphasis in spike train statistics. While not introducing novel material, the main added value of this tutorial is to present results of the field of non-equilibrium statistical mechanics in a pedagogical manner based on examples. These results have direct application to maximum entropy Markov chains, and may shed new light on the study of spike train statistics. This tutorial is suitable for researchers in the fields of physics or mathematics who are curious about the interesting questions and possibilities that computational neurosciences offers. The focus on this community is motivated by the growing community of mathematical physicists interested in computational neuroscience.

The rest of this tutorial is structured as follows. First, Section 2 introduces basic concepts of neural spike trains and Markov processes. Then, Section 3 introduces the notion of observable, and explores their fundamental properties. Section 4 introduces the core ideas of MEP, proposing the formal question and exploring methods for solving it. Section 5 studies various properties of interest of MEP models, including fluctuation-dissipation relationships, and their entropy production. Finally, Section 6 summarizes our conclusions.

2. Preliminary Considerations

This section introduces definitions, notations, and conventions that are used throughout the tutorial in order to give the necessary toolkit of ideas and notions to the unfamiliar reader.

2.1. Binning and Spike Trains

Consider a network of N spiking neurons, where time has been binned (i.e., discretized) in such a way that each neuron can exhibit no more than one action potential within one time bin $\Delta t_b$. Action potentials, or “spikes”, are “all-or-none” events, and hence, spike data can be encoded using sequences of zeros and ones. A spiking state is denoted by $x_t^k=1$, and corresponds to the event in which the k-th neuron spikes during the t-th time bin, while $x_t^k=0$ implies that it remains silent.

A spike pattern is defined as the spike-state of all neurons at time bin t, and is denoted by ${\mathit{x}}_t:={\left[x_t^k\right]}_{k=1}^N$. A spike block is a consecutive sequence of spike patterns, denoted by ${\mathit{x}}_{t,r}:={\left[{\mathit{x}}_s\right]}_{s=t}^r$ (see Figure 1). While the length of the spike block ${\mathit{x}}_{t,r}$ is $r-t+1$, it is useful to consider spike blocks of infinite length starting from time $t=0$, which are denoted by $\mathit{x}$. Finally, in this tutorial we consider that a spike train is an infinite sequence of spiking patterns. This assumption turns out to be useful because it allows us to put our analysis in the framework of stochastic processes, and because it also allows us to characterize asymptotic statistical properties.

The set of all possible spike patters (or state space) in a network of N neurons is denoted by $\mathbb{S}$, and the set of all spike blocks of length R in a network of N neurons is denoted by ${\mathbb{S}}^R$.

Even at a single neuron level, for repetitions of the same stimulus, neurons respond randomly, but with a certain statistical structure. This is the main reason to look for statistical characterizations of spike trains. When trying to find a statistical representation considering a whole population of neurons responding simultaneously to a given stimulus, the problem is the following. Consider an experimental spike train from a network of N neurons where sequences of spike patterns are considered time-independent. The spike patterns can take $2^N$ values (state space). For $N>10$ is not possible to observe all possible states in real experimental data nor computer simulations (2 h of recordings binned at 20 ms produce less than $2^{19}$ spike patterns). For $N=100$ the state space is $2^{100}$, therefore the frequentist approach is useless to estimate the invariant measure. Can we learn something about the statistics of spike patterns from data having access only to a very small fraction of the state space? The maximum entropy principle provides an answer to this question. This principle has been used in the context of spike train statistics mainly considering firing rates and synchronous pairwise correlations, which gives rise to trivial stochastic processes composed by (i.i.d) random variables [3,4,5]. However, as mentioned in the introduction, there exists strong evidence in favour of past events playing a role in spike train statistics, and the biological process in general [6,7,8,9,11]. This principle can be generalized considering non-synchronous correlations, affording to build Markov chains from data. This approach opens the way to a richer modeling framework that can afford to model time irreversibility (highly expected in biological systems) and to a remarkable mathematical machinery based on non-equilibrium statistical mechanics which can be used to characterize collective behavior and to explore the capabilities of the system. We focus our tutorial on non-equilibrium steady states in the context of maximum entropy spike trains. In the next section of this tutorial we present the elementary properties of Markov chains (our main object of analysis) which will be used in the next chapters to extract relevant information about the underlying neuronal network generating the data.

2.2. Elementary Properties of Markov Chains

A stochastic process is a collection of random variables $X_t\in \mathbb{S}$ indexed by $t\in T$ that often refers to time. The set $\mathbb{S}$ represents the phase-space of the process; in the case of stochastic processes representing spike trains, one usually takes $\mathbb{S}={\{0,1\}}^N$. Moreover, considering the temporal binning discussed in Section 2.1, usually $T=\mathbb{N}$ (the set of natural numbers) corresponds to the so-called discrete-parameter stochastic processes.

While spike trains can be characterized by stochastic processes dependent on an infinite past [23,24], Markov chains are particularly well-suited for modeling data sequences with finite temporal dependencies. In the next paragraphs we give the precise definition of a Markov process.

A stochastic process $(X_t:t\in \mathbb{N})$ defined on a measure space $\mathsf{\Omega }$ is said to be a $\mathbb{P}$–Markov chain if it satisfies the Markov property (with respect to the probability measure $\mathbb{P}$): if, for every $t\in \mathbb{N}$ and for each sequence of states $x_0,x_1,\dots ,x_{t+1}\in \mathbb{S}$, the following relationship holds:

$$\begin{array}{c}\hfill \mathbb{P}(X_{t+1}=x_{t+1}|X_0=x_0,X_1=x_1,\dots ,X_{t-1}=x_{t-1},X_t=x_t)=\mathbb{P}(X_{t+1}=x_{t+1}|X_t=x_t).\end{array}$$

(1)

This property is usually paraphrased as: the conditional distribution of the future given the current state and all past events depends exclusively on the current state of the process. It is direct to show that the Markov property is equivalent to the following condition: for every increasing sequence of indices $(i_1<i_2<\dots <i_n)$ in $\mathbb{N}$, and for arbitrary states $x_{i_1},x_{i_2},\dots ,x_{i_n}$ in $\mathbb{S}$, we have:

$$\begin{array}{c}\hfill \mathbb{P}(X_{i_n}=x_{i_n}|X_{i_{n-1}}=x_{i_{n-1}},\dots ,X_{i_1}=x_{i_1})=\mathbb{P}(X_{i_n}=x_{i_n}|X_{i_{n-1}}=x_{i_{n-1}}).\end{array}$$

To characterize the transition probabilities, define a $\mathbb{S}$-indexed stochastic matrix to be a doubly indexed array of non-negative real numbers $P=\left(p\right(i,j):i,j\in \mathbb{S})$ such that ${\sum }_{j\in \mathbb{S}}p(i,j)=1$ for every $i\in \mathbb{S}$. It can be shown that a Markov chain is well-defined if the following is provided:

(i)

An initial probability distribution, encoded by a vector $\mu :=({\mu }_i:i\in \mathbb{S})$.

(ii)

A collection of $\mathbb{S}$-indexed stochastic matrices $\{P_t:={\left(p_t(i,j)\right)}_{i,j\in \mathbb{S}}:t\in \mathbb{N}\}$.

Using these two elements, one can build probability measures $P^n$ on ${\mathbb{S}}^n$ as follows,

$$\begin{array}{c}\hfill P^n(i_0,i_1,\dots ,i_{n-1})=\mu \left(i_0\right)\prod _{j=0}^{n-2}{P}_j(i_j,i_{j+1}).\end{array}$$

Furthermore, the Kolmogorov extension theorem [25] guarantees the existence of a unique probability measure ${\mathbb{P}}_{\mu }$ on ${\mathbb{S}}^{\mathbb{N}}$ such that the coordinate process satisfies:

$$\begin{array}{c}\hfill {\mathbb{P}}_{\mu }(X_0=i_0,X_1=i_1,\dots ,X_n=i_n)=P^n(i_0,i_1,\dots ,i_n),\end{array}$$

and with respect to which $(X_t:t\in T)$ is a Markov chain. In this case ${\mathbb{P}}_{\mu }$ is said to be the probability law of the Markov chain $(X_t:t\in \mathbb{N})$. This notation also remarks that ${\mathbb{P}}_{\mu }$ is the law with initial distribution $\mu$.

2.3. Homogeneity, Ergodicity and Stationarity

A Markov chain is said to be homogeneous if the transition matrices do not depend on the time parameter t, i.e., if there exists a $\mathbb{S}$-indexed stochastic matrix P such that $P_t=P$ for every $t\in T$. Note that if $(X_t:t\in T)$ is a $\mathbb{P}$–homogeneous Markov chain, then for every $t\in T$:

$$\begin{array}{c}\hfill \mathbb{P}(X_{t+1}=j|X_t=i)=p(i,j):=p_{ij}.\end{array}$$

(2)

In the rest of this paper we focus exclusively on homogeneous Markov chains, since this is the model assumed in the maximum entropy framework.

Consider now a homogeneous Markov chain $(X_t:t\in T)$ with initial distribution $\mu$ and transition matrix P. Moreover, consider $p_{ij}^{\left(m\right)}$ to be the $(i,j)$-th entry of the product matrix $P^m=P·P·\dots ·P$. These quantities correspond to the m–steps transition probabilities. Equation (2) can be generalized to

$$\begin{array}{c}\hfill \mathbb{P}(X_{t+m}=j|X_t=i)=p_{ij}^{\left(m\right)}.\end{array}$$

A stochastic matrix P is said to be ergodic if there exists $k\in \mathbb{N}$ such that all the k–step transition probabilities are positive—i.e., there is a non-zero probability to go between any two states in k steps. A homogeneous Markov chain is ergodic if it can be defined by an initial distribution $\mu$ and an ergodic matrix.

Finally, a probability distribution $\pi$ on $\mathbb{S}$ is called a stationary distribution for the Markov chain specified by P if

$$\begin{array}{c}\hfill \pi P=\pi .\end{array}$$

(3)

Equivalently, $\pi$ is stationary for P if $\pi$ is a left eigenvector of the transition matrix corresponding to the eigenvalue $\lambda =1$, and is a probability distribution on $\mathbb{S}$. While it is true that 1 is always an eigenvalue of P, it may be the case that no eigenvector associated to it can be normalized to a probability distribution. Further conditions for existence and uniqueness will be given in the next paragraph. Finally, if a $\mathbb{S}$–indexed stochastic matrix P admits a stationary probability distribution $\pi$ and $(X_t:t\in \mathbb{N})$ is a Markov chain with initial distribution $\pi$ and transition matrix P, then for every $t\in \mathbb{N}$ and $i\in \mathbb{S}$:

$$\begin{array}{c}\hfill {\mathbb{P}}_{\pi }(X_t=i)={\pi }_i.\end{array}$$

In this case $(X_t:t\in \mathbb{N})$ is said to be a stationary Markov chain, or that the Markov chain is started from stationarity.

The notion of homogeneous ergodic Markov chains is relevant in the context of spike train statistics because of the Ergodic Theorem for finite-state Markov Chains, which state that for all finite-state, homogeneous, ergodic Markov chains $(X_t:t\ge 0)$ with transition matrix P the following hold:

(a)

There exists a unique stationary distribution $\pi$ for P that satisfies that ${\pi }_i>0$ for every $i\in \mathbb{S}$.

(b)

For every $j\in \mathbb{S}$,

$$\begin{array}{c}\hfill \underset{m\to +\infty }{lim}{p}_{ij}^{\left(m\right)}={\pi }_j.\end{array}$$

Equivalently, for every distribution $\nu$, ${lim}_{t\to \infty }{\mathbb{P}}_{\nu }(X_t=j)={\pi }_j.$ This property guarantees the uniqueness of the maximum entropy Markov chain.

2.4. The Reversed Markov Chain

Given a discrete ergodic Markov chain, it is mathematically possible to define its associated time reversed Markov chain. Some Markov chains in the steady-state yield the same Markov chain (in distribution) if the time course is inverted and others do not. It has been argued multiple times that those Markov chains that are different from their time inverted version are better suited to represent biological stochastic processes [6,7,9,11,12].

Let $\overrightarrow{P}$ be a stochastic matrix, and assume that it admits a stationary probability measure $\pi$. Assume too that ${\pi }_i>0$ for every $i\in \mathbb{S}$ (according to (a) in the Ergodic Theorem from the previous section, this is the case when $\overrightarrow{P}$ is ergodic). Define the $\mathbb{S}$–indexed matrix $\overleftarrow{P}$ with entries:

$$\begin{array}{c}\hfill {\overleftarrow{P}}_{ij}={\displaystyle \frac{{\pi }_j}{{\pi }_i}}{\overrightarrow{P}}_{ji}.\end{array}$$

A direct calculation shows that $\overleftarrow{P}$ is also a stochastic matrix. Moreover, if $\pi$ is stationary for $\overrightarrow{P}$, then it is for $\overleftarrow{P}$ as well.

Using the above facts, let ${\mathbb{P}}_{\pi }^{\to }$ and ${\mathbb{P}}_{\pi }^{\leftarrow }$ be the laws of two stationary Markov chains, denoted by $X_t$ and $Y_t$, whose stationary distribution is $\pi$ and transition probabilities are $\overrightarrow{P}$ and $\overleftarrow{P}$, respectively. The following holds

$$\begin{array}{cc}\hfill {\mathbb{P}}_{\pi }^{\leftarrow }(Y_0=i_0,Y_1=i_1,\dots ,Y_n=i_n)& ={\pi }_{i_0}{\overleftarrow{P}}_{i_0{i}_1}{\overleftarrow{P}}_{i_1{i}_2}\dots {\overleftarrow{P}}_{i_{n-1}{i}_n}\hfill \\ \hfill & ={\pi }_{i_0}{\displaystyle \frac{{\pi }_{i_1}}{{\pi }_{i_0}}}{\overrightarrow{P}}_{i_1{i}_0}{\displaystyle \frac{{\pi }_{i_2}}{{\pi }_{i_1}}}{\overrightarrow{P}}_{i_2{i}_2}\dots {\displaystyle \frac{{\pi }_{i_n}}{{\pi }_{i_{n-1}}}}{\overrightarrow{P}}_{i_n{i}_{n-1}}\hfill \\ \hfill & ={\pi }_{i_n}{\overrightarrow{P}}_{i_n{i}_{n-1}}{\overrightarrow{P}}_{i_{n-1}{i}_{n-2}}\dots {\overrightarrow{P}}_{i_1{i}_0}\hfill \\ \hfill & ={\mathbb{P}}_{\pi }^{\to }(X_0=i_n,X_1=i_{n-1}\dots ,X_n=i_0).\hfill \end{array}$$

By virtue of this result, it is natural to call the chain $(Y_t:t\ge 0)$ the reversed chain associated to $(X_t:t\ge 0)$.

2.5. Reversibility and Detailed Balance

A transition matrix P is reversible with respect to $\pi$ if the associated Markov chain started from $\pi$ has the same law as the reversed chain started from the same distribution. The reversibility of P with respect to $\pi$ is equivalent to the condition of detailed balance, given by

$$\begin{array}{c}\hfill {\pi }_i{P}_{ij}={\pi }_j{P}_{ji}\forall i,j\in \mathbb{S}.\end{array}$$

(4)

Note that any probability measure $\pi$ that satisfies detailed balance with respect to P is necessarily stationary, since

$$\begin{array}{c}\hfill \sum _{i\in \mathbb{S}}{\pi }_i{P}_{ij}=\sum _{i\in \mathbb{S}}{\pi }_j{P}_{ji}={\pi }_j\sum _{i\in \mathbb{S}}{P}_{ji}={\pi }_j\mathrm{for}\mathrm{every}j\in \mathbb{S}.\end{array}$$

The converse is, however, not true in general: a stationary distribution may not satisfy Equation (4).

Intuitively, Equation (4) states that, in the stationary state, the fluxes between each pair of states balance each other. In contrast, detailed balance is broken when there is a cycle of three or more states in the state space supporting a net probability current—even in the steady state. Detailed balance is also interpreted as “time reversibility”, as one could not distinguish the steady state dynamics of the system when going forward or backward in time. Certainly, this property is not expected in stochastic processes generated by biological systems. Several disciplines use the term “equilibrium” to refer to long-term behaviour, i.e., what is not transient. In this tutorial we use the term equilibrium state exclusively to refer to probability vectors that satisfy the detailed balance conditions—given in Equation (4). Markov chains that satisfy the detailed balance condition are referred as equilibrium steady states, and conversely, steady states that do not satisfy the detailed balance conditions are called Non-Equilibrium Steady States (NESS).

How to characterize (finite state, homogeneous) reversible Markov chains? Following [26], consider any finite graph $(\mathbb{S},{(c_{ij})}_{i,j\in \mathbb{S}})$, with vertex set $\mathbb{S}$ and with the edge between vertices i and j labelled by the non-negative edge $c_{ij}=c_{ji}$. The graph can be visualized as a system of points labelled by $\mathbb{S}$, and with a line segment between points whenever the corresponding conductance is positive. Define $c_i={\sum }_{j\in \mathbb{S}}{c}_{ij}$ and the $\mathbb{S}$–indexed stochastic matrix given by

$$\begin{array}{c}\hfill p_{ij}={\displaystyle \frac{c_{ij}}{c_i}},\end{array}$$

Now define $C={\sum }_{i\in \mathbb{S}}{c}_i$. It is straightforward to prove that P is reversible with respect to the probability measure given by

$$\begin{array}{c}\hfill {\pi }_i={\displaystyle \frac{c_i}{C}},\end{array}$$

and thus it is stationary for P. The unique Markov chain started from $\pi$ and transition matrix P is called the stationary random walk on the network $(\mathbb{S},{(c_{ij})}_{i,j\in \mathbb{S}})$. Conversely, any reversible $\mathbb{S}$–valued Markov chain can be identified with the random walk on the graph with vertex set $\mathbb{S}$ and edges given by $c_{ij}=c_{ji}={\pi }_i{p}_{ij}$.

2.6. Law of Large Numbers for Ergodic Markov Chains

The Law of Large Numbers (LLN) that applies to independent and identically distributed random variables (i.i.d.) can be extended to the realm of ergodic Markov chains. In effect, for a given ergodic Markov chain $(X_t:t\ge 0)$ with stationary distribution $\pi$ and transition matrix P, define the random variables $N_i^{\left(T\right)}$ equal to the number of occurrences of the state i up to time $T-1$, i.e.,

$$\begin{array}{c}\hfill N_i^{\left(T\right)}=\sum _{t=0}^{T-1}{\mathbf{1}}_{\{X_t=i\}},\end{array}$$

where ${\mathbf{1}}_{\{·\}}$ is an indicator function. Similarly, define the random variables $N_{ij}^{\left(T\right)}$ as the number of occurrences of the consecutive pair of states $(i,j)\in {\mathbb{S}}^2$—in that order—up to time $T-1$, i.e.,

$$\begin{array}{c}\hfill N_{ij}^{\left(T\right)}=\sum _{t=1}^{T-1}{\mathbf{1}}_{\{X_{t-1}=i,X_t=j\}}.\end{array}$$

With this, the Strong Law of Large Numbers for Markov chains can be stated as follows: if $(X_t:t\ge 0)$ is ergodic and $\pi$ is its unique stationary distribution, then

$$\begin{array}{c}\hfill {\mathbb{P}}_{\mu }\left(\underset{T\to +\infty }{lim}{\displaystyle \frac{N_i^{\left(T\right)}}{T}}={\pi }_i\right)=1\mathrm{and}{\mathbb{P}}_{\mu }\left(\underset{T\to +\infty }{lim}{\displaystyle \frac{N_{ij}^{\left(T\right)}}{T}}={\pi }_i{p}_{ij}\right)=1,\end{array}$$

holds for any initial distribution $\mu$. The result, in turn, implies the Weak Law of Large Numbers for Markov chains, which state that, following the above notation, for every $\epsilon >0$ and for every starting distribution $\mu$:

$$\begin{array}{c}\hfill \underset{T\to +\infty }{lim}{\mathbb{P}}_{\mu }\left(\left|{\displaystyle \frac{N_i^{\left(T\right)}}{T}}-{\pi }_i\right|>\epsilon \right)=0\mathrm{and}\underset{T\to +\infty }{lim}{\mathbb{P}}_{\mu }\left(\left|{\displaystyle \frac{N_{ij}^{\left(T\right)}}{T}}-{\pi }_i{p}_{ij}\right|>\epsilon \right)=0.\end{array}$$

Let’s denote by $C\left(\mathbb{S}\right)$ the space of real-valued functions on $\mathbb{S}$. Clearly, any function of $C\left(\mathbb{S}\right)$ can be written as $f\left(x\right)={\sum }_{i\in \mathbb{S}}{a}_i{\mathbf{1}}_i\left(x\right)$ for certain constants $a_i$, $i\in \mathbb{S}$. Then, the above result generalizes as: for every $f\in C\left(\mathbb{S}\right)$, ergodic chain $X_t$, and probability distribution $\mu$, the following holds:

$${\mathbb{P}}_{\mu }\left(\underset{T\to +\infty }{lim}{\displaystyle \frac{1}{T}}\sum _{t=0}^{T-1}f\left(X_t\right)={\mathbb{E}}_{\pi }\left(f\left(X_0\right)\right)\right)=1.$$

This corresponds to a particular form of the Birkhoff Ergodic Theorem, which is briefly outlined in the next section and is relevant to characterize spike trains of observables as from data it is possible to accurately measure average values of firing rates and correlations. For an ergodic stationary Markov chain with a state space relatively small with respect to the sample size, this theorem guarantees that from a large sample the transition probabilities and the invariant measure can be recovered. This is not the case in spike train statistics at the population level as only a very small fraction of the state space is sampled in experimental spike trains. However, some features of the spike trains can be sampled very accurately from experimental data. In the next section, we present the basic elements to build from these characteristics a statistical model of the entire population.

3. Observables of Markov Chains and Their Properties

The notion of observable plays a central role in the study of maximum entropy spike trains. This section discusses their nature and fundamental properties.

3.1. Observables and Their Empirical Averages

Suppose a spiking neuronal network of N neurons is provided. Suppose too that measurements of spike patterns for T time bins have been performed. The observables are real-valued functions over the possible spike blocks, denoted here by $\mathbb{B}:={\mathbb{S}}^T$. Let $C\left(\mathbb{B}\right)$ be the space of such observables, i.e., the linear space of real-valued functions $f:\mathbb{B}\mapsto \mathbb{R}$. Recall the space $C\left(\mathbb{S}\right)$ of observables of range 1, discussed at the end of the above section. This space can be naturally embedded into $C\left(\mathbb{B}\right)$; thus, it can be considered as a linear subspace of the latter. More generally, the space of observables of range R for $R\le T$, denoted $C\left({\mathbb{S}}^R\right)$, is just the space of real-valued functions on ${\mathbb{S}}^R$, that we identify with its image through the natural embedding into $C\left(\mathbb{B}\right)$.

We are interested in the average of observables with respect to several probability measures. If $\mu$ is a probability measure on $\mathbb{B}$ (i.e., $\mu \left(\omega \right)\ge 0$ and ${\sum }_{\omega \in \mathbb{B}}\mu \left(\omega \right)=1$) and f an observable of range $R\le T$ i.e., $f\in C\left({\mathbb{S}}^R\right)$, we define its expectation with respect to $\mu$ as

$$\begin{array}{c}\hfill \mu \left(f\right)={\mathbb{E}}_{\mu }\left\{f\right\}:=\sum _{\omega \in \mathbb{B}}f\left(\omega \right)\mu \left(\omega \right).\end{array}$$

Since the space of blocks of length T is finite, the above sum is always finite, and thus our definition makes sense for every probability measure on $\mathbb{B}$.

In the context of spike-trains, an important class of observable is made up of $\{0,1\}$-valued functions. It can be proved that any finite-range binary observable can be written as a finite sum of finite products of functions of the form ${\mathbf{1}}_{\{X_i^{\left(j\right)}=1\}}$ that represents the event that the j-th neuron fires during the i-th bin. The average value if this observable is known as the firing rate of neuron j. This quantity has been proposed as one of the major neural coding strategies used by the brain [27].

Consider a spike block ${\mathit{x}}_{0,T-1}$, where T is the sample length. Although in most cases the probability measure $\mu$ that characterizes the spiking activity is not known, it is meaningful to use the experimental data to estimate the mean values of specific observables. The range of the validity of this procedure is usually based on prior assumptions about the nature of the source that originates the sample. For example, it can be assumed that the sample is a short piece of an infinite path that comes running from the far past, and so it can be assumed that this piece exhibits a behavior that is close to the stationary distribution. In this case, one can consider for any number $R\le T$ the quantity:

$$\begin{array}{c}\hfill Q(y_0,y_1,\dots ,y_{R-1})=\sum _{j=0}^{T-R}{\mathbf{1}}_{\{{\mathit{x}}_{j,j+R-1}=(y_0,y_1,\dots ,y_{R-1})\}},\end{array}$$

that counts the number of appearances of the sequence $(y_0,\dots ,y_{R-1})$ as a consecutive sub-sequence of ${\mathit{x}}_{0,T-1}$. Now, for any set $A\subseteq {\mathbb{S}}^R$, define:

$$\begin{array}{c}\hfill {\mu }_{{\mathit{x}}_{0,T-1}}\left(A\right):={\displaystyle \frac{1}{T-R+1}}\sum _{\mathit{y}\in A}Q\left(y\right),\end{array}$$

$${\mu }_{{\mathit{x}}_{0,T-1}}\left(f\right)=A_T\left(f\right)=\frac{1}{T-R+1}\sum _{i=0}^{T-R}f\left({\mathit{x}}_{i,R-1+i}\right).$$

When the empirical distribution is not explicitly stated, it is customary to write $\langle f\rangle$ to denote the average of the observable f with respect to this probability measure.

3.2. Moments and Cumulants

Observables are random variables whose average values can be determined from experimental data or from the explicit representation of the underlying measure characterizing the stochastic process generating the data. Important statistical properties of random variables are encoded in the cumulants. We will use the cumulants in Section 5 of this tutorial to characterize and infer properties of maximum entropy Markov chains. Let us now introduce them.

The moment of order r of a real-valued random variable X is given by $m_r=\mathbb{E}\left(X^r\right)$, for $r\in \mathbb{N}$ (here we freely use the notation $\mathbb{E}$ to denote the expectation with respect to a probability measure that should be inferred from the context). The moment generating function (or Laplace transform) of a random variable is defined by:

$$\begin{array}{c}\hfill M\left(t\right)=\mathbb{E}(e^{tX}),\end{array}$$

and provided it is a function of t with continuous derivatives of arbitrary order at 0, we have that:

$$\begin{array}{c}\hfill m_r={\left({\displaystyle \frac{d^r}{d{t}^t}}M\right)}_{t=0}.\end{array}$$

The cumulants ${\kappa }_r$ are the coefficients in the Taylor expansion of the cumulant generating function. The cumulants are defined as the logarithm of the moment generating function, namely,

$$\begin{array}{c}\hfill lnM\left(t\right)=\sum _r{\kappa }_r{t}^r/r!.\end{array}$$

The relation between the moments and cumulants is obtained extracting coefficients from the Taylor expansion, i.e.,

$$\begin{array}{c}\hfill {\kappa }_r={\left({\displaystyle \frac{d^r}{d{t}^r}}ln\left(M\left(t\right)\right)\right)}_{t=0}\end{array}$$

(5)

which yields the first values:

$$\begin{array}{ccc}{\kappa }_1\hfill & =& m_1,\hfill \\ {\kappa }_2\hfill & =& m_2-m_1^2,\hfill \\ {\kappa }_3\hfill & =& m_3-3m_2{m}_1+2m_1^3,\hfill \\ {\kappa }_4\hfill & =& m_4-4m_3{m}_1-3m_2^2+12m_2{m}_1^2-6m_1^4,\hfill \end{array}$$

and so on. In particular, the first four cumulants are the mean, the variance, the skewness and the kurtosis.

3.3. Observables and Ergodicity

Let $\theta :\mathsf{\Omega }\mapsto \mathsf{\Omega }$ be the shift operator that acts on a sequence $\omega \in \mathsf{\Omega }$ as:

$$\begin{array}{c}\hfill {\left(\theta \left(\omega \right)\right)}_i={\omega }_{i+1},\end{array}$$

i.e., $\theta$ shifts the sequence one position to the left. Now, assume that the Markov chain $(X_t:t\ge 0)$ is ergodic. Let $\pi$ be its unique stationary probability distribution. The Birkhoff Ergodic Theorem states that under the above assumptions, for every $f\in C\left(\mathbb{B}\right)$:

$$\begin{array}{c}\hfill {\mathbb{P}}_{\mu }\left(\underset{N\to +\infty }{lim}{\displaystyle \frac{1}{N}}\sum _{n=0}^{N-1}f\circ {\theta }^n={\mathbb{E}}_{\pi }\left(f\right)\right)=1,\end{array}$$

for every initial measure $\mu$. This equation means that under the ergodic hypothesis, the temporal averages converge to the spatial averages. The importance of this fundamental result should not be underestimated since this result supports the practice of regarding averages of (hopefully) large samples of experimental data as faithful approximations of the true values of the expectations of the observables.

3.4. Central Limit Theorem for Observables

Consider an arbitrarily large sequence of spike patterns of N neurons. Consider $t\in \mathbb{N}$ and let ${\mathit{x}}_{0,t-1}$ be the spike-block of length t. Also, let f be an arbitrary observable of fixed range R. The asymptotic properties of $A_t\left(f\right)$ are established in the following context: the finite sample is drawn from an ergodic Markov chain, i.e., $\mathit{x}\sim {\mathbb{P}}_{\nu }$, where ${\mathbb{P}}_{\nu }$ is the Markov probability measure of an ergodic chain $(X_t:t\ge 0)$ started from an arbitrary initial distribution. Let $\pi$ be the unique stationary measure for the Markov chain. Observe that by virtue of the ergodic assumption, the empirical averages of observables become more accurate as the sampling size grows, i.e.,

$$\begin{array}{c}\hfill {\mathbb{P}}_{\nu }\left(A_t\left(f\right)\to {\mathbb{E}}_{\pi }\left\{f\right\}\right)=1.\end{array}$$

for any starting condition $\nu$. However, the above result does not clarify the rate at which the accuracy improves. The central limit theorem (CLT) for ergodic Markov chains provides a result to approach this issue (for datails see [28]).

Theorem 1 (Central limit theorem for ergodic Markov chains).

Under the above assumptions, and keeping notation, define:

$$\begin{array}{c}\hfill \sigma =\sqrt{\left({\mathbb{E}}_{\pi }\left(\right(f(X_0,\dots ,X_{R-1})-{\mathbb{E}}_{\pi }{\left(f(X_0,\dots ,X_{R-1})\right)}^2\right)}.\end{array}$$

Let $L_t$ be the law of the random variable $\frac{\sqrt{t}}{\sigma }\left[A_t\left(f\right)-{\mathbb{E}}_{\pi }\left\{f\right\}\right]$ under the measure ${\mathbb{P}}_{\nu }$ of an ergodic Markov chain started from an arbitrary distribution. Let L be the law of a standard normal random variable. Then $L_t\to L$ in the sense of weak convergence of convergence in distribution. This is usually written as:

$$\begin{array}{c}\hfill {\mathbb{P}}_{\nu }\left\{\frac{\sqrt{t}}{\sigma }\left[A_t\left(f\right)-{\mathbb{E}}_{\pi }\left\{f\right\}\right]\le x\right\}\to \frac{1}{\sqrt{2\pi }\sigma }{\int }_{-\infty }^x{e}^{-\frac{s^2}{2\sigma }}ds.\end{array}$$

This theorem implies that “typical” fluctuations of $A_t\left(f\right)$ around its long term average ${\mathbb{E}}_{\pi }\left\{f\right\}$ are of the order of $\sigma /\sqrt{t}$. For spike trains, this theorem quantifies the expected Gaussian fluctuations of observables in terms of the sample size of the experimental data.

3.5. Large Deviations of Average Values of Observables

Although the CLT for ergodic Markov chains is precise in describing the typical fluctuations around the mean, it does not characterize the probabilities of large fluctuations. While it is clear that the probability of large fluctuations of average values vanish as the sample size increases, it is sometimes relevant to characterize the decrease rate of this probability. That is what the large deviation principle (LDP) does.

Let f be a function of finite range defined on the space of sequences. In many situations, f will be a $\{0,1\}$–valued function. Let ${\mathbb{P}}_{\pi }$ be the probability measure on the space of sequences induced by an ergodic Markov chain with stationary probability $\pi$. The empirical average $A_t\left(f\right)$ satisfies a large deviation principle (LDP) with rate function $I_f$, defined as

$$I_f\left(s\right):=-\underset{t\to \infty }{lim}\frac{1}{t}log{\mathbb{P}}_{\pi }\{A_t\left(f\right)>s\},$$

(6)

if the above limit exists. The above condition implies for large t that ${\mathbb{P}}_{\pi }\{A_t\left(f\right)>s\}\approx e^{-t{I}_f\left(s\right)}$. In particular, if $s>{\mathbb{E}}_{\pi }\left\{f\right\}$ the Law of Large Numbers (LLN) ensure that ${\mathbb{P}}_{\pi }\{A_t\left(f\right)>s\}$ goes to zero as t increases, but the rate function quantifies the speed at which this occurs.

Calculating $I_f$ using the definition (Equation (6)), is usually impractical. However, the Gärtner-Ellis theorem provides a clever alternative to circumvent this problem [29]. Let us introduce the scaled cumulant generating function (SCGF) associated to the random variable f by

$${\lambda }_f\left(k\right)=:\underset{t\to \infty }{lim}\frac{1}{t}ln{\mathbb{E}}_{\pi }\left[e^{tk{A}_t\left(f\right)}\right],k\in \mathbb{R},$$

(7)

when the limit exists (details about cumulant generating functions are found in [30]). While the empirical average $A_t\left(f\right)$ is taken over a sample (empirical measure), the expectation in (7) is computed over the probability distribution given by ${\mathbb{P}}_{\pi }\{·\}$.

Theorem 2 (Gärtner-Ellis theorem).

If ${\lambda }_f$ is differentiable, then the average $A_t\left(f\right)$ satisfies a LDP with rate function given by the Legendre transform of ${\lambda }_f$, that is

$$I_f\left(s\right)=\underset{k\in \mathbb{R}}{max}\{ks-{\lambda }_f\left(k\right)\}.$$

(8)

Therefore, the large deviations of empirical averages $A_t\left(f\right)$ can be characterized by first computing their SCGF and then finding their Legendre transform.

A useful application of the LDP is to estimate the likelihood that the empirical average $A_t\left(f\right)$ takes a value far from its expected value. Let us assume that $I_f\left(s\right)$ is a positive differentiable convex function. Then, ${\lambda }_f\left(k\right)$ is also differentiable [31] (for a comprehensive discussion about the differentiability of ${\lambda }_f\left(k\right)$ see [30].) Then, as $I_f\left(s\right)$ is convex it has a unique global minimum. Denoting this minimum by $s^{*}$, from the differentiability of $I_f\left(s\right)$ it follows that $I_f\left(s^{*}\right)=0$. Additionally, it follows from properties of the Legendre transform that $s^{*}={\lambda }_f^{\prime }\left(0\right)={\mathbb{E}}_p\left\{f\right\}$, which is the LLN that says that $A_t\left(f\right)$ concentrates around $s^{*}$. Consider $s\ne s^{*}$ and that $I_f\left(s\right)$ admits a Taylor expansion around $s^{*}$

$$I_f\left(s\right)=I_f\left(s^{*}\right)+I_f^{\prime }\left(s^{*}\right)(s-s^{*})+\frac{I_f^{″}\left(s^{*}\right){(s-s^{*})}^2}{2}+O{(s-s^{*})}^3.$$

As $s^{*}$ is zero and a minimum of $I\left(s\right)$, the first two terms of this expansion are zero, and as $I\left(s\right)$ is convex $I^{″}\left(s\right)>0$. For large t, it follows from (6) that

$$\begin{array}{cc}\hfill p\{A_t\left(f\right)>s\}& \approx e^{-t{I}_f\left(s\right)}\hfill \\ \hfill & \approx e^{-t\left(\frac{I_f^{″}\left(s^{*}\right){(s-s^{*})}^2}{2}\right)},\hfill \end{array}$$

(9)

so the “small deviations” (we are using Taylor expansion) of $A_t\left(f\right)$ around $s^{*}$ are Gaussian (in Equation (9) $1/I_f^{″}\left(s^{*}\right)={\lambda }_f^{″}\left(0\right)={\sigma }^2$). In this sense, the LDP can be considered as an extension of the CLT as it goes beyond the small deviations around $s^{*}$ (Gaussian), but additionally the large deviations (not Gaussian) of $A_t\left(f\right)$.

4. Building Maximum Entropy Temporal Models

This section presents the main concepts behind the construction of maximum entropy models for temporal data. The next Section 4.1, introduces the concept of entropy, and then Section 4.2 formulates the problem of maximizing the entropy rate. Methods for solving this problem are discussed in Section 4.3, which are then illustrated in an example presented in Section 4.4.

4.1. The Entropy Rate of a Temporal Model

4.1.1. Basic Definitions

In order to give mathematical meaning to the rather vague notion of uncertainty, a natural approach is to employ the well-established notion of Shannon entropy. For any probability measure p defined over the state space E (not necessarily $\mathbb{S}$), the Shannon entropy of p is given by

$$S\left[p\right]:=-\sum _{x\in E}p\left(x\right)logp\left(x\right).$$

Note that this definition can be used for measures on the spaces of infinite sequences $E^{\mathbb{N}}$. However, as in most cases of interest, the value saturates in infinite. A better suited notion in this context is given by the entropy rate, which plays a crucial role in the rest of this tutorial.

Definition 1 (entropy rate).

Let μ be a probability measure on the space of sequences ${\mathbb{S}}^{\mathbb{N}}$. For $n\ge 1$ let ${\mu }_n$ be the probability measure induced by μ on the initial n coordinates, i.e., ${\mu }_n$ is the probability distribution on $E^n$ given by:

$$\begin{array}{c}\hfill {\mu }_n(x_0,x_1,\dots ,x_n-1)=\mu \left(\omega \in {\mathbb{S}}^{\mathbb{N}}:X_i=x_i\mathit{for}i=0,1,\dots ,n-1\right).\end{array}$$

The entropy rate of the measure μ is defined by:

$$\mathcal{S}\left[\mu \right]=\underset{n\to \infty }{lim}{\displaystyle \frac{1}{n}}S\left[{\mu }_n\right].$$

(10)

The above definition applies to any probability distribution on the space of sequences. Intuitively, the entropy rate correspond to the entropy per time unit, and represents how much “uncertainty” is created by the process as time moves forward.

4.1.2. The Entropy Rate of I.I.D. and Markov Models

Let us consider first a null model of spike activity, where there is complete statistical independence between two consecutive spike patters. For this, first recall that $\mathbb{S}={\{0,1\}}^N$, where N is the fixed number of neurons. Without loss of generality, we can enumerate the elements of $\mathbb{S}$ as $s_1,s_2,\dots ,s_{2^N}$. Let $\nu =({\nu }_1,{\nu }_2,\dots ,{\nu }_{2^N})$ be a probability measure on $\mathbb{S}$ such that:

$$\begin{array}{c}\hfill \nu (s_k)={\nu }_k\end{array}$$

For a T–block $x=(x_0,x_1,\cdots ,x_{T-1})\in {\mathbb{S}}^T$ and for every $s\in \mathbb{S}$, we set:

$$\begin{array}{c}\hfill N_s^T\left(x\right)=\sum _{i=0}^{T-1}{\mathbf{1}}_{\{x_i=s\}}.\end{array}$$

On the space of infinite spike trains ${\mathbb{S}}^{\mathbb{N}}$ we consider the probability $\mu ={\nu }^{\otimes \mathbb{N}}$, i.e., the product measure on the space of spike trains. Observe that the induced measure is given by:

$$\begin{array}{c}\hfill {\mu }_n(x_0,x_1,\dots ,x_{T-1})=\prod _{k=1}^{2^N}{\nu }_k^{N_s^t(x_0,\dots ,x_{T-1})}.\end{array}$$

With this, a straightforward calculation shows that

$$\begin{array}{c}\hfill \mathcal{S}\left[\mu \right]=S\left[\nu \right]=-\sum _{k=1}^{2^N}{\nu }_{k}ln\left({\nu }_k\right),\end{array}$$

and in this case we observe that the entropy rate is equal to the entropy of the probability distribution induced by each coordinate map.

A reasonable next step in the hierarchy of models is to weaken the independence hypothesis and assume instead that the spike activity keeps some bounded memory of the past. For this, following the considerations of Section 2, let us consider an ergodic discrete Markov chain with transition matrix P and invariant distribution $\pi$ taking values in $\mathbb{S}$. Let $\mu =\mu (P,\pi )$ be the measure induced by this chain on the space ${\mathbb{S}}^{\mathbb{N}}$. Observe that, with the above notation:

$$\begin{array}{c}\hfill {\mu }_n(x_0,x_1,\dots ,x_{n-1})={\pi }_{x_0}\prod _{j=1}^{n-1}{P}_{x_{j-1}{x}_j}.\end{array}$$

A direct computation shows that

$$\begin{array}{cc}\hfill S\left[{\mu }_1\right]& =-\sum _{(x_0,x_1)\in {\mathbb{S}}^2}{\pi }_{x_0}{P}_{x_0{x}_1}ln\left({\pi }_{x_0}{P}_{x_0{x}_1}\right)\hfill \\ \hfill & =-\sum _{x\in \mathbb{S}}{\pi }_{x}ln\left({\pi }_x\right)-\sum _{(x_0,x_1)\in {\mathbb{S}}^2}{\pi }_{x_0}{P}_{x_0{x}_1}ln\left(P_{x_0{x}_1}\right),\hfill \end{array}$$

and induction shows that:

$$\begin{array}{c}\hfill S\left[{\mu }_n\right]=-\sum _{x\in \mathbb{S}}{\pi }_{x}ln\left({\pi }_x\right)-n\sum _{(x_0,x_1)\in {\mathbb{S}}^2}{\pi }_{x_0}{P}_{x_0{x}_1}ln\left(P_{x_0{x}_1}\right).\end{array}$$

Thus dividing by n and taking the limit in Equation (10), one finds that

$$\begin{array}{c}\hfill \mathcal{S}\left[\mu \right]=-\sum _{(x_0,x_1)\in {\mathbb{S}}^2}{\pi }_{x_0}{P}_{x_0{x}_1}ln\left(P_{x_0{x}_1}\right).\end{array}$$

4.2. Entropy Rate Maximization under Constraints

Now we introduce the central problem of this tutorial. Assume we have empirical data from spiking activity. Consider the empirical averages of K observables, $\langle f_k\rangle$, for $f_k,k=1,\dots ,K$. We need to characterize the Markov chains that are consistent with these average values. Except for trivial and uninteresting situations, there is no finite set of empirical averages that uniquely determines a distribution $\mu$ on ${\mathbb{S}}^{\mathbb{N}}$ that fits the averages, in the sense that

$$\begin{array}{c}\hfill \mu \left(f_k\right)=\langle f_k\rangle \mathrm{for}k=1,\dots ,K.\end{array}$$

Consequently, we need to impose further restrictions in order to guarantee uniqueness. A useful and meaningful approach is the so-called Maximum Entropy Markov Chain model (MEMC), which fit the unique probability measure $\mu$ among all the stationary Markov measures $\nu$ on ${\mathbb{S}}^{\mathbb{N}}$ that match the expected values of a given set of observables and that maximizes the entropy rate. Mathematically, it is written in the following form:

$$\begin{array}{cccc}\hfill & \underset{\nu \in {\mathcal{M}}_{inv}}{\max }\hfill & \hfill & \mathcal{S}\left[\nu \right]\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \nu \left(f_k\right)={\langle f_k\rangle }_e=C_k,\forall k\in \{1,\cdots ,K\},\hfill \end{array}$$

where ${\mathcal{M}}_{inv}$ is a shorthand for the sets of stationary Markov measures on ${\mathbb{S}}^{\mathbb{N}}$. Formally:

$$\begin{array}{c}\hfill {\mathcal{M}}_{inv}:=\{(\pi ,P):\pi \mathrm{is}\mathrm{a}\mathrm{probability}\mathrm{on}\mathbb{S},P\mathrm{is}\mathrm{stochastic},\pi P=\pi \}.\end{array}$$

It is to be noted that the maximum entropy principle can be derived in some scenarios from more general principles based on large deviation theory [30]. In this framework, entropy maximization corresponds to Kullback-Leiber divergence minimization. This approach can be useful for accounting additional information that is not in the form of functional constraints, but as a Bayesian prior. A major drawback of this approach to be applied to spike trains, is that it assumes stationarity in the data. While this condition is not to be naturally expected in biological systems, controlled experiments can be carried out in the context of spike train analysis in order to maintain these conditions [3,8,32,33]. The maximum entropy principle as presented here is useful only in the stationary case. However, some extensions have been proposed [34,35]. Note also that there are alternative variational principles which can be used to find distributions that extremize the value of quantities such as the maximum entropy production principle [36,37,38], or the Prigogine minimum entropy production principle [39,40]. To the best of our knowledge, these alternatives have not yet been explored in the context of spike train statistics.

4.3. Solving the Optimization Problem

We now discuss techniques for finding models that maximize the entropy rate.

4.3.1. Lagrange Multipliers and the Variational Principle

To solve the above optimization problem, let us introduce the set of Lagrange multipliers $h_k\in \mathbb{R}$ and an energy function $\mathcal{H}={\sum }_{k=1}^K{h}_k{f}_k$, which is a linear combination of observables. Consider the following unconstrained optimization problem, which can be framed in the context of the variational principle of the thermodynamic formalism [41]:

$$\mathcal{F}\left[\mathcal{H}\right]=\underset{\nu \in {\mathcal{M}}_{inv}}{sup}\left\{\mathcal{S}\left[\nu \right]+\nu \left(\mathcal{H}\right)\right\}=\mathcal{S}\left[\mu \right]+\mu \left(\mathcal{H}\right),$$

(11)

where $\mathcal{F}\left[\mathcal{H}\right]$ is called the free energy and $\nu \left(\mathcal{H}\right)={\sum }_{k=1}^K{h}_k\nu \left(f_k\right)$ is the average value of $\mathcal{H}$ with respect to the measure $\nu$. The following holds:

$$\frac{\partial \mathcal{F}\left[{\mathcal{H}}_{\mathit{h}}\right]}{\partial h_k}={\mathbb{E}}_p\left\{f_k\right\}=C_k,\forall k\in \{1,...,K\},$$

where ${\mathbb{E}}_p\left\{f\right\}$ is the average of $f_k$ with respect to p (maximum entropy measure), which is equal (by restriction) to the average value of $f_k$ with respect to the empirical measure from the data.

The maximum-entropy (ME) principle [42] has been successfully applied to spike data from the cortex and the retina [3,8,9,11,12,43]. The approach starts by fixing the set of constraints determined by the empirical average of observables measured from spiking data. Maximizing the entropy (concave functional) under constraints, gives a unique distribution. The choice of observables to measure in the empirical data (constraints) determines the statistical model. The approach of Lagrange multipliers may not be practical when trying to fit a MEMC. In the next section we introduce an alternative optimization based on spectral properties.

4.3.2. Transfer Matrix Method

In order to illustrate the transfer matrix method, we start with a classical example that allow us to introduce a fundamental definition. Let A be a adjacency matrix i.e., a $\{0,1\}$-valued square matrix with rows and columns indexed by the elements of $\mathbb{S}$. If there exists an $n\ge 0$ such that

$$\begin{array}{c}\hfill A_{ij}^n>0\end{array}$$

for every $i,j\in \mathbb{S}$, we say that A is primitive. The next well-known theorem of Linear Algebra is crucial [44] for the uniqueness of the MEMC.

Theorem 3 (Perron-Frobenius theorem).

Let A be a primitive matrix. Then,

• There is a positive maximal eigenvalue ρ > 0 such that all other eigenvalues satisfy $\mid {\rho }^{\prime }\mid <\rho$. Moreover ρ is simple;
• There are positive left- and right-eigenvectors $u=(u_1,\cdots ,u_k),v=(v_1,\cdots ,v_k)$ s.t. $uA=\rho u,Av=\rho v.$

Apply the above theorem to a primitive matrix A, and define:

$$P_{ij}=\frac{A_{ij}{v}_j}{\rho v_i};{\pi }_i=\frac{u_i{v}_i}{\langle u,v\rangle },$$

where $\langle u,v\rangle$ is the standard inner product in ${\mathbb{R}}^{2^N}$ (we refer the reader to [44] for details). The matrix P built above is stochastic. Moreover, $\pi$ is its unique stationary measure. Define the Parry measure to be the Markov measure:

$$\begin{array}{c}\hfill \mu (i_0,i_1,\cdots ,i_n)={\pi }_{i_0}{P}_{i_0{i}_1},\cdots ,P_{i_{n-1}{i}_n}.\end{array}$$

It is well known that the Parry measure is the unique measure of maximal entropy consistent with the adjacency matrix A [45,46].

Inspired by this result, we consider now the general case. Consider constraints given by a set of empirical averages of observables, as explained in the previous section. The above example certainly fits this setting: just consider binary observables associated to each pair of states $(i,j)$ that evaluates to 1 when a transition from state i to state j has been observed in the data. In our general setting, we assume that the chosen observables have a finite maximum range R. From these observables the energy function $\mathcal{H}$ of finite range R is built as a linear combination of these observables. Using this energy function we build a matrix denoted by ${\mathcal{L}}_{\mathcal{H}}$, so that for every $y,w\in {\mathbb{S}}^R$ its entries are given as follows:

$${\mathcal{L}}_{\mathcal{H}}(y,w)=\left\{\begin{array}{cc}{e}^{\mathcal{H}\left(y_1{w}_{1,R-1}\right)}\hfill & \mathrm{if} y_{1,R-1}=w_{0,R-2}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

where $y_1{w}_{R-1}$ is the concatenated block built from $y_1$ and $w_{1,R-1}$. For observables of range one, the matrix above is defined as ${\mathcal{L}}_{\mathcal{H}}(y,w)=e^{\mathcal{H}\left(y\right)}$. Assuming $\mathcal{H}>-\infty$, the elements of the matrix ${\mathcal{L}}_{\mathcal{H}}$ are non-negative. Furthermore, in every non trivial case, the matrix is primitive and satisfies the Perron-Frobenius theorem [44]. Denote by $\rho$ the unique largest eigenvalue of ${\mathcal{L}}_{\mathcal{H}}$. Just as above, we denote by $\mathbf{u}$ and $\mathbf{v}$ the left and right eigenvectors of ${\mathcal{L}}_{\mathcal{H}}$ associated to $\rho$. Notice that $u_i>0$ and $v_i>0$, for all $i\in \mathbb{S}$. The free energy associated to a transfer matrix is the logarithm of the unique maximum eigenvalue.

The matrix ${\mathcal{L}}_{\mathcal{H}}$ can be turned into a Markov matrix of maximum entropy. For a primitive matrix M with spectral radius $\rho$, and positive right eigenvector $\mathbf{v}$ associated to $\rho$, the stochastic matrix built from M is computed as follows:

$$\mathfrak{S}\left(M\right)=\frac{1}{\rho }{D}^{-1}MD,$$

where D is the diagonal matrix with entries $D_{ii}=v_i$. The MEMC transition matrix P and unique stationary probability measure $\pi$ are explicitly given by

$$P=\mathfrak{S}\left({\mathcal{L}}_{\mathcal{H}}\right);{\pi }_i:=\frac{u_i{v}_i}{\langle u,v\rangle },\forall i\in \mathbb{S}.$$

(13)

Note that when $\mathcal{H}=0$, the MEMC is characterized by the Markov transition matrix with components [47]:

$$P_{ij}=\frac{A_{ij}{v}_j}{\rho v_i},$$

where A is the adjacency matrix.

4.3.3. Finite Range Gibbs Measures

For a fixed energy function $\mathcal{H}$ of range $R\ge 2$, there is a unique stationary Markov measure $\mu$ for which there exist a constant $\gamma \ge 1$ such that [46],

$${\gamma }^{-1}\le \frac{\mu \left[{\mathit{x}}_{1,n}\right]}{exp({\sum }_{k=1}^{n-R+1}\mathcal{H}\left({\mathit{x}}_{k,k+R-1}\right)-(n+R-1)\mathcal{F}\left[\mathcal{H}\right])}\le \gamma ,$$

(14)

that attains the supremum (11). The measure $\mu$, as defined by (14), is known in the symbolic dynamics literature as Gibbs measure in the sense of Bowen [48]. All MEMCs belong to this class of measures. Moreover, the classical Gibbs measures in statistical mechanics are particular cases of (14), when $\gamma =1$, $\mathcal{F}\left[\mathcal{H}\right]=logZ$ and $\mathcal{H}$ is an energy function of range one, leading to an i.i.d stochastic process characterized by the product measure $\mu$. In this case the following holds:

$$\mu \left(\mathit{x}\right)=\frac{e^{\mathcal{H}\left(\mathit{x}\right)}}{Z}\forall \mathit{x}\in \mathbb{S};Z=\sum _{\mathit{x}\in \mathbb{S}}{e}^{\mathcal{H}\left(\mathit{x}\right)}.$$

The free energy that is defined here has a deep relationship with the free energy in thermodynamics. Consider a thermodynamic system in equilibrium. The Helmholtz free energy derived from the partition function as follows:

$$F\left(\beta \right)=-{\beta }^{-1}logZ$$

where $\beta =1/\left(kT\right)$ and k is Boltzmann’s constant and T is the temperature.

This quantity is related to the cumulant generating function for the energy. In the context of the maximum entropy principle, the physical temperature and the Boltzmann’s constant play no role, so usually both are considered equal to 1. From the free energy, all of the thermodynamic properties of the system can be obtained via its derivatives, examples are the internal energy, specific heat, and entropy. It is to be noted that the definition used in this tutorial for the free energy (11) follows from the conventions used in the field of thermodynamic formalism [41,45,46] and changes its sign with the usual convention in the field of statistical mechanics.

4.4. Example

We present here the toy example that we will use to explore statistical properties of spike trains using the non-equilibrium statistical physics approach. We present the transfer matrix technique to compute the Markov transition matrix, its invariant measure and free energy from a potential $\mathcal{H}$.

Consider a range-2 potential with two neurons ($N=2$). We use the notation introduced in Section 2.1:

$$\mathcal{H}\left({\mathbf{x}}^{0,1}\right)=h_1{x}_0^1{x}_1^2+h_2{x}_0^2{x}_1^1.$$

The state space of this problem is given by:

$$\left(\begin{array}{c}0\\ 0\end{array}\right),\left(\begin{array}{c}0\\ 1\end{array}\right),\left(\begin{array}{c}1\\ 0\end{array}\right),\left(\begin{array}{c}1\\ 1\end{array}\right).$$

The transfer matrix (12) associated to $\mathcal{H}$ is, in this case, a $4\times 4$ matrix

$${\mathcal{L}}_{{\mathbf{xx}}^{\prime }}=\left(\begin{array}{cccc}1& 1& 1& 1\\ 1& 1& e^{h_2}& e^{h_2}\\ 1& e^{h_1}& 1& e^{h_1}\\ 1& e^{h_1}& e^{h_2}& e^{h_1+h_2}\end{array}\right).$$

This matrix satisfies the hypothesis of the Perron-Frobenius theorem. The maximum eigenvalue is

$$\rho =\frac{1}{2}(3+e^{(h_1+h_2)}+\sqrt{5+4e^{h_1}+4e^{h_2}+2e^{(h_1+h_2)}+e^{(2h_1+2h_2)}}),$$

and the free energy

$$\mathcal{F}\left[\mathcal{H}\right]=log\left(\rho \right).$$

(15)

5. Statistical Properties of Markov Maximum Entropy Measures

The procedure of finding a maximum entropy model gives us a full statistical model of the system of interest. In this section we discuss the added value that having such a model can provide.

5.1. Cumulants from Free Energy

All the statistical properties of the observables and their correlations can be obtained by taking the successive derivatives of the free energy with respect to the Lagrange Multipliers. This property explains the important role played by the free energy in the framework of MEMC. In general,

$$\frac{{\partial }^n\mathcal{F}\left[\mathcal{H}\right]}{\partial h_k^n}={\kappa }_n\forall k\in \{1,...,K\},$$

where ${\kappa }_n$ is the cumulant of order n (Equation (5)). In particular, taking the first derivative:

$$\frac{\partial \mathcal{F}\left[\mathcal{H}\right]}{\partial h_k}={\mathbb{E}}_p\left\{f_k\right\}\forall k\in \{1,...,K\},$$

(16)

where ${\mathbb{E}}_p\left\{f_k\right\}$ is the average with respect to the maximum entropy distribution p, which is equal to the average value of $f_k$ with respect to the empirical measure. With Equation (16) the parameters of the MEMC can be fitted to be consistent with fixed average values of observables.

Suppose that we compute from data the average values of the following observables $\langle x_0^1{x}_1^2\rangle =0.1$ and $\langle x_0^2{x}_1^1\rangle =0.3$. We solve (16) (two equations and two unknowns) and obtain $h_1=-1.98306$ and $h_2=1.48406$. With these parameters, the following Markov transition matrix and invariant measure are obtained from (13):

$$P_{\mathbf{x}{\mathbf{x}}^{\prime }}=\left(\begin{array}{cccc}0.232971& 0.469441& 0.0987018& 0.198886\\ 0.115617& 0.232971& 0.216056& 0.435357\\ 0.549892& 0.15252& 0.232971& 0.0646176\\ 0.272896& 0.0756914& 0.509966& 0.141446\end{array}\right)\pi \left(\mathbf{x}\right)=\left(\begin{array}{c}0.29102\\ 0.248443\\ 0.248443\\ 0.212095\end{array}\right).$$

5.2. Fluctuation-Dissipation Relations

For a first-order stationary Markov chain, since each $X_n,n\ge 1$ depends on its predecessor, this induces a non-zero time-correlation between $X_n$ and $X_{n+r}$, even when the distance r is greater than 1. This correlation, and more generally, time correlations between observables can be directly derived from the free energy. This relationship is usually referred to as Fluctuation-dissipation, and is also related to the linear response function that is presented in Section 5.7.

Let P be an ergodic matrix and indexed by the states in some finite set E, and $\pi$ be its unique stationary measure. In this general context, for two real-valued functions that depend on a fixed finite number of components, we define the n–step correlation as

$$\begin{array}{c}\hfill C_{f,g}\left(n\right)={\mathbb{E}}_{\pi }\left(f\left(X_0\right)g\left(X_n\right)\right)-{\mathbb{E}}_{\pi }\left(f\left(X_0\right)\right){\mathbb{E}}_{\pi }\left(g\left(X_0\right)\right).\end{array}$$

In the particular case of MEMC with potentials of range $R>1$ there is a positive time correlation between pairs of observables $f\left(x_n\right)$ and $g\left(x_{n+r}\right)$. Suppose the correlations decay fast enough so that (at least)

$$\sum _{n=0}^{\infty }\left|C_{f,g}\left(n\right)\right|<\infty .$$

Then the following sum (known as the Green-Kubo formula [49]) converges and is non-negative:

$${\sigma }_{f_k,f_j}^2=C_{f_k,f_j}\left(0\right)+\sum _{r=1}^{\infty }{C}_{f_k,f_j}\left(r\right)+\sum _{r=1}^{\infty }{C}_{f_j,f_k}\left(r\right).$$

(17)

Additionally, it can be shown that the energy function and the free energy depends smoothly upon maximum entropy parameters. Moreover, the correlations between observables can be obtained from the free energy through:

$${\sigma }_{f_k,f_j}^2=\frac{{\partial }^2\mathcal{F}\left[\mathcal{H}\right]}{\partial h_k\partial h_j}=\frac{\partial \mu \left(f_j\right)}{\partial h_k}.$$

The relationship between a correlation and a derivative of the free energy is called the fluctuation-dissipation theorem [50]. For a MEMC characterized by $\mu (P,\pi )$, the fluctuation-dissipation relationships can be obtained explicitly:

$$\begin{array}{cc}\hfill \frac{{\partial }^2\mathcal{F}\left[\mathcal{H}\right]}{\partial h_k\partial h_j}=& {\mathbb{E}}_{\mu }\left[f_k{f}_j\right]-{\mathbb{E}}_{\mu }\left[f_k\right]{\mathbb{E}}_{\mu }\left[f_j\right]+\sum _{r=1}^{\infty }\sum _{\mathbf{x},{\mathbf{x}}^{\prime }\in \mathbb{S}}\left(f_k\left(\mathbf{x}\right)f_j\left({\mathbf{x}}^{\prime }\right){\pi }_{\mathbf{x}}{P}_{\mathbf{x}{\mathbf{x}}^{\prime }}^r-{\mathbb{E}}_{\mu }\left[f_k\right]{\mathbb{E}}_{\mu }\left[f_j\right]\right)\hfill \\ \hfill & +\sum _{r=1}^{\infty }\sum _{\mathbf{x},{\mathbf{x}}^{\prime }\in \mathbb{S}}\left(f_j\left(\mathbf{x}\right)f_k\left({\mathbf{x}}^{\prime }\right){\pi }_{\mathbf{x}}{P}_{\mathbf{x}{\mathbf{x}}^{\prime }}^r-{\mathbb{E}}_{\mu }\left[f_k\right]{\mathbb{E}}_{\mu }\left[f_j\right]\right).\hfill \end{array}$$

(18)

For MEMC built from K observables, the correlations can be conveniently arranged in a $K\times K$ symmetric matrix denoted by $\chi$ (the symmetry refers to the Onsager reciprocity relations [51]).

$${\chi }_{jk}=\frac{{\partial }^2\mathcal{F}\left[\mathcal{H}\right]}{\partial h_k\partial h_j}=\frac{\partial \mu \left(f_j\right)}{\partial h_k}=\frac{\partial \mu \left(f_k\right)}{\partial h_j}={\chi }_{kj}.$$

(19)

For the example Section 4.4, we obtain the matrix $\chi$ by taking the second derivatives of (15) and evaluate at the parameters found previously,

$${\chi }_{kj}=\left(\begin{array}{cc}0.0971481& 0.0606071\\ 0.0606071& 0.127964\end{array}\right).$$

In Figure 2, we plot the right hand side of Equation (18) for the MEMC built from the example Section 4.4 consistent with constraints considered in the example of Section 5.1, for the auto-correlation of the observable $x_0^2{x}_1^1$.

5.3. Resonances and Decay of Correlations

We now turn back to the general setting of an arbitrary ergodic matrix P with stationary measure $\pi$ associated to a Markov chain taking values on a finite state space (not necessarily the space of spike-patterns). Without loss of generality, assume that P is indexed by the states in $E=\{1,2,\dots ,M\}$. It can be proved that in this case there exists $(l_i:i=1,2,\dots ,M)$ and $(r_i:i=1,2,\dots ,M)$, sets of left and right eigenvectors respectively, associated to the eigenvalues $({\rho }_i:i=1,\dots ,M)$. We can assume that the eigenvectors and left and right eigenvalues have been sorted and normalized in such a way that ${\rho }_1=1$, $l_1$ is the unique P–stationary probability vector $\pi$, $r_1={(111\dots 1)}^T$, and

$$\begin{array}{c}\hfill \langle l_i|r_j\rangle ={\delta }_{i,j},\end{array}$$

where ${\delta }_{i,j}$ is the Kronecker delta, and $\langle uv\rangle =\langle u,v\rangle$ corresponds to the Dirac’s bra-ket, $|u\rangle \langle v|=u{v}^T$. With the same notation, the spectral decomposition of P is written:

$$\begin{array}{c}\hfill P=\sum _{i=1}^M{\rho }_i|r_i[l_i].\end{array}$$

Hence:

$$\begin{array}{c}\hfill P^n=\sum _{i=1}^M{\rho }_i^n|r_i[l_i].\end{array}$$

(20)

Given two functions $f:E\mapsto \mathbb{R}$ and $g:E\mapsto \mathbb{R}$ the following holds,

$$\begin{array}{cc}\hfill C_{f,g}\left(n\right)& :={\mathbb{E}}_{\pi }\left(f\left(X_0\right)g\left(X_n\right)\right)-{\mathbb{E}}_{\pi }\left(f\left(X_0\right)\right){\mathbb{E}}_{\pi }\left(g\left(X_0\right)\right)\hfill \\ \hfill & =\langle \pi f\circ P^{n}g\rangle -\langle \pi f\rangle \langle \pi g\rangle .\hfill \end{array}$$

(21)

Recall the discussion in previous sections regarding the reverse chain Section 2.4. Writing ${\mathbb{E}}_{\pi }^{\leftarrow }$ for the expectation operator associated to the reverse Markov measure, i.e., to the measure $\mu =\mu (\pi ,\overleftarrow{P})$, one can see that

$$\begin{array}{c}\hfill {\mathbb{E}}_{\pi }\left(f\left(X_0\right)g\left(X_n\right)\right)={\mathbb{E}}_{\pi }^{\leftarrow }\left(f\left(X_n\right)g\left(X_0\right)\right),\end{array}$$

and hence (21) becomes

$$\begin{array}{c}\hfill \langle \pi |g\circ {\overleftarrow{P}}^{n}f\rangle -\langle \pi |f\rangle \langle \pi |g\rangle .\end{array}$$

From (20)

$$\begin{array}{c}\hfill f\circ P^{n}g=\sum _{i=1}^M\langle l_i|g\rangle |f\circ r_i\rangle \end{array}$$

and thus (21) becomes

$$\begin{array}{cc}\hfill C_{f,g}\left(n\right)& =\sum _{i=1}^M{\rho }_i^n\langle l_i|g\rangle \langle \pi |f\circ r_i\rangle -\langle \pi |f\rangle \langle \pi |g\rangle \hfill \\ \hfill & =\sum _{i=2}^M{\rho }_i^n\langle l_i|g\rangle \langle \pi |f\circ r_i\rangle .\hfill \end{array}$$

(22)

In Figure 3, we show the auto-correlations of the same observable considered in Figure 1, for the same MEMC. We observe modulations in the decay of the auto-correlations due to the complex eigenvalues in Equation (22), which arise in the non-symmetric transition matrix induced by the irreversibility of the MEMC.

We have found in Equation (22) an explicit expression for the decay of correlation for observables from the set of eigenvalues and eigenvectors of the transition matrix P. This is relevant in the context of spike train statistics because as the matrix P characterizing the spike trains is not expected to be symmetric, its eigenvalues are not necessarily real and modulations in the decay of correlations are expected (resonances). When measuring correlations between observables from data, one may observe this oscillatory situation that resembles resonances. This may be a symptom of a non-equilibrium situation.

5.4. Large Deviations for Average Values of Observables in MEMC

Obtaining the probability of “rare” average values of firing rates, pairwise correlations, triplets or non-synchronous observables is relevant in spike train statistics as these observables are likely to play an important role in neuronal information processing, and rare values may convey crucial information or be a symptom that the system in not working properly.

Here, we build from a previous article [52] where it is shown that the SCGF (7) can be obtained directly from the inferred Markov transition matrix P through the Gärtner-Ellis theorem (8). Consider a MEMC with transition matrix P. Let f be an observable of finite range and $k\in \mathbb{R}$. We introduce the tilted transition matrix by f of P, parametrized by k and denoted by ${\tilde{P}}^{\left(f\right)}\left(k\right)$ [53] as follows:

$${\tilde{P}}_{ij}^{\left(f\right)}\left(k\right)=P_{ij}{e}^{kf\left(ij\right)}i,j\in \mathbb{S}.$$

(23)

The tilted transition matrix can be directly obtained from the spectral properties of the transfer matrix (12),

$$\begin{array}{ccc}\hfill {\tilde{P}}_{ij}^{\left(f\right)}\left(k\right)& =& \frac{e^{{\mathcal{H}}_{ij}}{v}_j}{v_i\rho }{e}^{kf\left(ij\right)}\hfill \\ \hfill & =& \frac{e^{\left[{\mathcal{H}}_{ij}+kf\left(ij\right)\right]}{v}_j}{v_i\rho }i,j\in \mathbb{S}.\hfill \end{array}$$

Recall that $\mathbf{v}$ is the right eigenvector associated to its maximum eigenvalue $\rho$ of the transfer matrix $\mathcal{L}$. Here we use the notation ${\mathcal{H}}_{ij}$ to specify that the energy function is built from the elements of the state space i and j. Remarkably, this result is valid not only for the observables in the energy function, i.e., from here the LDP of more general observables can be computed.

To obtain an explicit expressions for the SCGF ${\lambda }_f\left(k\right)$, it is possible to take advantage of the structure of the underlying stochastic process. For instance, for i.i.d. random process $X_t$ where $X_i\sim X$ from Definition 7, one can obtain that

$$\lambda \left(k\right)=\underset{t\to \infty }{lim}\frac{1}{t}ln\mathbb{E}{[e^{tk{A}_t\left(f\right)}]}^t=ln\mathbb{E}[e^{kf\left(X\right)}],$$

which is the case of range one observables. Using the Equation (23), we obtain that the maximum eigenvalue of the tilted matrix $\rho ({\tilde{P}}_f(k))$ is,

$$\rho ({\tilde{P}}_f(k))=\sum _j{\pi }_j{e}^{kf\left(j\right)}j\in \mathbb{S}.$$

As ${\tilde{P}}_f$ is a primitive matrix, the uniqueness of $\rho ({\tilde{P}}_f(k))$ is ensured from the Perron-Frobenius theorem.

For additive observables of ergodic Markov chains, a direct calculation (see [54]) leads us to

$${\lambda }_f(k)=ln(\rho ({\tilde{P}}^{\left(f\right)})).$$

It can also be proved that ${\lambda }_f\left(k\right)$, in this case, is differentiable [54], setting up the scene to use the Gärtner-Ellis theorem to obtain $I_f\left(s\right)$ as shown in Figure 4.

5.5. Information Entropy Production

Given a Markov chain $(X_t:t\ge 0)$ on a general finite state space E with transition matrix P started from the distribution $\nu$, denoted ${\nu }^{\left(n\right)}$ the distribution of $X_n$, namely, for $i\in E$:

$$\begin{array}{c}\hfill {\nu }^{\left(n\right)}\left(i\right)={\mathbb{P}}_{\nu }(X_n=i).\end{array}$$

Obviously, ${\nu }^{\left(0\right)}=\nu$, and

$$\begin{array}{c}\hfill {\nu }_j^{(n+1)}=\sum _{i\in E}{\nu }_i^{\left(n\right)}{P}_{ij}.\end{array}$$

The information-theoretic entropy of the probability distribution $\nu$ at time n is given by

$$\begin{array}{c}\hfill {\mathcal{S}}_n\left(\nu \right):=-\sum _{i\in E}{\nu }_i^{\left(n\right)}log{\nu }_i^{\left(n\right)},\end{array}$$

and the change of entropy over one time step is defined as

$$\begin{array}{c}\hfill \Delta {\mathcal{S}}_n:={\mathcal{S}}_{n+1}\left(\nu \right)-{\mathcal{S}}_n\left(\nu \right).\end{array}$$

A bit of algebra yields

$$\Delta {\mathcal{S}}_n=-\sum _{i,j\in E}{\nu }_j^{\left(n\right)}{P}_{ji}log\frac{{\nu }_j^{(n+1)}{P}_{ji}}{{\nu }_i^{\left(n\right)}{P}_{ij}}+\frac{1}{2}\sum _{i,j\in E}\left[{\nu }_j^{\left(n\right)}{P}_{ji}-{\nu }_i^{\left(n\right)}{P}_{ij}\right]log\frac{{\nu }_j^{\left(n\right)}{P}_{ji}}{{\nu }_i^{\left(n\right)}{P}_{ij}}.$$

The first term on the right hand side above is called information entropy flow and the second term information entropy production [12].

In the stationary case, i.e., when P admits a stationary measure $\pi$ and the chain is started from that distribution, one has that ${\nu }^{\left(n\right)}=\pi$ for every $n\ge 0$; thus, in this case, the change of entropy rate is zero, i.e., for stationary chains, the information entropy flow equals (minus) information entropy production. This case is the focus of this work. The chain is associated to spike train activity for transitions between L–blocks. Starting from stationarity the entropy production rate is explicitly given by

$$\begin{array}{c}\hfill IEP(P,\pi ):=\frac{1}{2}\sum _{\mathbf{x},{\mathbf{x}}^{\prime }\in {\mathbb{S}}^L}\left[\pi \left({\mathbf{x}}^{\prime }\right)P_{{\mathbf{x}}^{\prime }\mathbf{x}}-\pi \left(\mathbf{x}\right)P_{\mathbf{x}{\mathbf{x}}^{\prime }}\right]log\frac{\pi \left({\mathbf{x}}^{\prime }\right)P_{{\mathbf{x}}^{\prime }\mathbf{x}}}{\pi \left(\mathbf{x}\right)P_{\mathbf{x}{\mathbf{x}}^{\prime }}}\ge 0.\end{array}$$

(24)

The non-negativity implies that information entropy is positive as long as the process violates the detailed balance conditions (4). This is analogous to the second law of thermodynamics [55]. From this equation it is easy to realize that if the Markov chain satisfies the detailed balance condition, the information entropy production is zero.

In Figure 5, we compute the information entropy production from Equation (24), for the MEMC of the example (Section 4.4) for different values of the parameters $h_1,h_2$.

It may seem contradictory that in stationary state the entropy is constant, while there is a positive “production” of entropy. The information entropy production in stationary state always compensate the information entropy flow, which leaves the information entropy rate constant. In this case we refer to non-equilibrium steady states (NESS).

5.6. Gallavotti-Cohen Fluctuation Theorem

To characterize the fluctuations of the IEP, consider the MEMC $\mu (P,\pi )$ and the following observable:

$$W_n\left({\mathbf{x}}_{0,n}\right)=\frac{1}{n}ln\left(\frac{\mu \left({\mathbf{x}}_{0,n}\right)}{\mu \left({\mathbf{x}}_{n,0}\right)}\right),$$

It can be shown that $lim\frac{1}{n}{W}_n\to IEP(\pi ,P)$. The Gallavotti-Cohen fluctuation theorem is as a statement about properties of the SCGF and rate function of the IEP [14].

$${\lambda }_W\left(k\right)={\lambda }_W(-k-1),I_W\left(s\right)=I_W(-s)-s.$$

(25)

This symmetry holds for a general class of stochastic processes including NESS from Markov chains [56], and is a universal property of the IEP, i.e., it is independent of the parameters of the MEMC. To compute ${\lambda }_W\left(k\right)$ and $I_W\left(s\right)$, define $A{\left(k\right)}_{ij}=P_{ij}{\left[\frac{{\pi }_i{P}_{ij}}{{\pi }_j{P}_{ji}}\right]}^k$. If $\rho \left(k\right)$ is the largest eigenvalue of $A\left(k\right)$, then ${lim}_{n\to \infty }ln\mathbb{E}(e^{n\lambda W_n})=ln\rho \left(k\right)$.

In Figure 6, we illustrate the Gallavotti-Cohen symmetry property of the large deviation functions associated to the IEP (Equation (25)).

These properties are relevant to the large deviations of the averaged entropy production denoted $\frac{W_t}{t}$ over a trajectory ${\mathit{x}}_{0,t-1}$ of the Markov chain $p(\pi ,P)$. The following relationship holds,

$$\frac{p\left\{\frac{W_t}{t}\approx s\right\}}{p\left\{\frac{W_t}{t}\approx -s\right\}}\asymp e^{ts}.$$

This means that the positive fluctuations of $\frac{W_t}{t}$ are exponentially more probable than negative fluctuations of equal magnitude.

5.7. Linear Response

The linear response serves to quantify how a small perturbation $\delta \mathit{h}$ of a set of the maximum entropy parameters affects the average values of observables in terms of the unperturbed measure. This is relevant in the context of spike trains statistics to identify stiff and sloppy directions in the space of parameters. A small change in a sloppy parameter produces very little impact in the statistical model. In contrast, a small change in a stiff parameter produces a significant change. For a MEMC characterized by $\mu =(P,\pi )$ corresponding to an energy function with fixed parameters $\mathbf{h}$ denoted by ${\mathcal{H}}_{\mathbf{h}}$, one can obtain the average value of a given observable $f_k$ from (16).

Now, consider a perturbed energy denoted by $\tilde{\mathcal{H}}={\mathcal{H}}_{\mathit{h}+\delta \mathit{h}}$. Using a Taylor expansion, the average value of an arbitrary observable $f_k$ with respect to the MEMC can be obtained $\tilde{\mu }=(\tilde{P},\tilde{\pi })$ associated to the perturbed energy. Considering the Taylor expansion of $\mathcal{F}\left[{\mathcal{H}}_{\mathit{h}+\delta \mathit{h}}\right]$ about ${\mathcal{H}}_{\mathit{h}}$

$$\begin{array}{ccc}\hfill \frac{\partial \mathcal{F}\left[{\mathcal{H}}_{\mathit{h}+\delta \mathit{h}}\right]}{\partial h_k}& =& \frac{\partial \mathcal{F}\left[{\mathcal{H}}_{\mathit{h}}\right]}{h_k}+\sum _j\frac{{\partial }^2\mathcal{F}\left[{\mathcal{H}}_{\mathit{h}}\right]}{\partial h_k{h}_j}\delta h_j+O{\left(\delta h_j\right)}^2,\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill {\mathbb{E}}_{\tilde{\mu }}\left[f_k\right]& =& {\mathbb{E}}_{\mu }\left[f_k\right]+\sum _j\frac{{\partial }^2\mathcal{F}\left[{\mathcal{H}}_{\mathit{h}}\right]}{\partial h_k{h}_j}\delta h_j+O{\left(\delta h_j\right)}^2,\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill \Delta \mathbb{E}\left[f\right]& \approx & \chi ·\delta \mathbf{h}.\hfill \end{array}$$

(28)

We use (16) to go from (26) to (27). Observe from (27) that a small perturbation of a parameter $h_j$ influences the average value of all other observables in the energy function (as $f_k$ is arbitrary). The perturbation is modulated by the second derivatives of the free energy corresponding to the unperturbed regime $\mathcal{F}\left[{\mathcal{H}}_{\mathit{h}}\right]$ (see Figure 7).

6. Discussion and Future Work

This tutorial explores how one can use maximum entropy methods to capture asymmetric temporal aspects of spike trains from experimental data. In particular, we showed how spatio-temporal constraints can produce homogeneous irreducible Markov chains whose unique steady state is, in general, non-equilibrium (NESS)—thus, detailed balance condition is not satisfied causing strictly positive entropy production. This fact highlights that only non-synchronous maximum entropy models induce time irreversible processes, which is one of the key hallmarks of biological systems.

We have presented a survey of diverse techniques from mathematics and statistical mechanics to study these NESS, which correspond to a rich toolkit that can be employed to study unexplored aspects of spike train statistics. We emphasise that many of these concepts, including entropy production and fluctuation-dissipation relationships, have not been explored much in the context of spike train analysis. However, the fact that time irreversibility is such an important feature of living systems suggest that these notions might play an important role in neural dynamics.

Possible extensions include measuring the entropy production for different choices of spatio-temporal constraints using the maximum entropy method on biological spike train recordings. A more ambitious extension is to explore the relationship between entropy production computed from experimental data obtained from different physiological processes and relate them to features such as adaptation or learning. Concerning time-dependent neuronal network models, future studies might lead to a better understanding of the impact of particular synaptic topologies of neuronal network models on the corresponding entropy production, decay of correlations, resonances and other sophisticated statistical properties.

Other possible extensions are related to the drawbacks of current approaches. This can include limitations of the maximum entropy method related to the requirement of stationarity in the data, which is not a natural condition for some biological scenarios. However, several of the techniques presented in this tutorial naturally extend to the non-stationary case, including the information entropy production, which can still be defined along non-stationary trajectories [14]. Related to this issue is that the approach presented in this tutorial does not make any reference to the stimulus. While this issue has been addressed in the synchronous framework [34], there is still an open field to explore the Markovian extension of these ideas. Another interesting topic to explore in future studies is the inclusion of the non-stationary approach such as the state space analysis proposed in [35]. Also, another open problem is related to the efficient implementation of the transfer matrix technique, which currently requires an important computational effort in the case of large neural networks. Recently, some improvements of this approach have been proposed based in Monte Carlo methods [57].

In summary, we believe that these topics are fertile ground for multi-disciplinary exploration by teams composed of mathematicians, physicists, and neuroscientists. It is our hope that this work may foster future collaborative research among disciplines, which might bring new breakthroughs to advance our fundamental understanding of how the brain works.