Estimating Mutual Information for Spike Trains: A Bird Song Example

Abstract

Zebra finches are a model animal used in the study of audition. They are adept at recognizing zebra finch songs, and the neural pathway involved in song recognition is well studied. Here, this example is used to illustrate the estimation of mutual information between stimuli and responses using a Kozachenko–Leonenko estimator. The challenge in calculating mutual information for spike trains is that there are no obvious coordinates for the data. The Kozachenko–Leonenko estimator does not require coordinates; it relies only on the distance between data points. In the case of bird songs, estimating the mutual information demonstrates that the information content of spiking does not diminish as the song progresses.

1. Introduction

The mutual information between two random variables X and Y is often conveniently described using a diagram like this: where the whole rectangle represents the entropy $H(X,Y)$ of the joint variable $(X,Y)$. This is, in general, less than the sum of $H\left(X\right)$ and $H\left(Y\right)$ because X and Y are not independent. In this diagram, the purple and green regions together are intended to represent $H\left(X\right)$, and the green and yellow regions are intended to represent $H\left(Y\right)$. The purple region on its own represents $H\left(X\right|Y)$: the entropy remaining, on average, when the value of Y is known. In the same way, the yellow region represents $H\left(Y\right|X)$. Now, the mutual information is represented by the green section:

It is

$$I(X,Y)=H\left(X\right)-H\left(X\right|Y)=H(Y)-H(Y\left|X\right)$$

(1)

or, by substitution,

$$I(X,Y)=\mathbb{E}{log}_2\left[\frac{p_{X|Y}\left(x\right|y)}{p_X\left(x\right)}\right]=\mathbb{E}{log}_2\left[\frac{p_{Y|X}\left(y\right|x)}{p_Y\left(y\right)}\right]$$

(2)

Here, for illustrative purposes, mutual information is described relative to a specific example: the neural response of cells in the zebra finch auditory pathway to zebra finch songs. This is both an interesting neuroscientific example and an example which is typical of a broad set of neuroscience problems.

The zebra finch is a model animal used to study both auditory processing and learning; the male finch sings, and he has a single song which begins with a series of introductory notes, followed by two or three repetitions of the motif: a series of complex frequency stacks known as syllables, separated by pauses. Syllables are about 50 ms long, with songs lasting about two seconds. The songs have a very rich structure, and both male and female zebra finches can distinguish one zebra finch song from another.

Here, we use a data set consisting of spike trains recorded while the bird is listening to a set of songs, and we provide an estimate for the mutual information between the song identity and spike trains recorded from cells in the auditory pathway. This is an interesting and non-trivial problem. Generally, calculating mutual information is costly in terms of data because it requires the estimation of probabilities such as $p_Y\left(y\right)$ and $p_{Y|X}\left(y\right|x)$. For this reason, some measure of correlation is often used when quantifying the relationship between two random variables. However, not all data types have a correlation: calculating the correlation assumes algebraic properties of the data that are not universal. As an example, calculating the correlation between X and Y requires the calculation of $\mathbb{E}\left[XY\right]$, which in turn assumes that it makes sense to multiply the x and y values. This is not the case for the typical neuroscience example considered here, where the set of outcomes for X is the song identities and that for Y is the spike trains. To circumvent this, spike trains are often replaced with something else, spike counts for example. However, this involves an implicit assumption about how information is coded. This is likely to be inappropriate in many cases. Indeed, the approach taken to calculating mutual information can involve making very strong assumptions about information coding, the very thing that is being studied.

The purpose of this review paper is to demonstrate a different approach: there is a metric-space version of the Kozachenko–Leonenko estimator [1,2] introduced in [3,4,5] and inspired by [6]. This approach has been tested on simulated data, for example in [5], and this shows it to be promising. However, it is important to also test it on real data. Here, it is applied in the zebra finch example.

2. Materials and Methods

Let

$$\mathcal{D}=\{(x_1,y_1),(x_2,y_2),...,(x_n,y_n)\}$$

(3)

be a data set. In our case, the $x_i$ are the labels for songs in the set of stimuli, with each $x_i\in \{1,\dots ,n_s\}$; $n_s$ is the number of different songs. For a given trial, $y_i$ is the spiking response. This will be a point in “the space of spike trains”. What exactly is meant by the space of spike trains is less clear, but for our purposes here, the important point is that this can be regarded as a metric space, with a metric that gives a distance between any two spike trains; see [7,8], or, for a review, [9].

Given the data, the mutual information is estimated by

$$I(X,Y)\approx \frac{1}{n}\sum _{i=1}^n{log}_2\left[\frac{p_{Y|X}\left(y_i\right|x_i)}{p_Y\left(y_i\right)}\right]$$

(4)

where the particular choice of which conditional probability to use, $p_{Y|X}$ rather than $p_{X|Y}$, has been made for later convenience. Thus, the problem of estimating mutual information is one of estimating the probability mass functions $p_{Y|X}$ and $p_Y$ at the data points in $\mathcal{D}$. In our example, there is no challenge to estimating $p_X$, since each song is presented an equal number of times during the experiment $p_X\left(x_i\right)=1/n_s$ for all $x_i$ and, in general $p_X\left(x_i\right)$ is known from the experiment design. However, estimating $p_{Y|X}$ and $p_Y$ is more difficult.

In a Kozachenko–Leonenko approach, this is performed by first noting that for a small volume $R_i$ containing the point $y_i$,

$$p_Y\left(y_i\right)\approx \frac{1}{\mathrm{vol}\left(R_i\right)}{\int }_{R_i}{p}_Y\left(y\right)dy$$

(5)

with the estimate becoming more and more exact for smaller regions $R_i$. If the volume of $R_i$ were reduced towards zero, $p_Y\left(y\right)$ would be constant in the resulting tiny region. Here, $\mathrm{vol}\left(R_i\right)$ denotes the volume of $R_i$. Now, the integral ${\int }_{R_i}{p}_Y\left(y\right)dy$ is just the probability mass contained in $R_i$ and so it is approximated by the number of points in $\mathcal{D}$ that are in $R_i$:

$${\int }_{R_i}{p}_Y\left(y\right)dy\approx \frac{\left|\right\{y_j\in R_i\left\}\right|}{n}.$$

(6)

It should be noted at this point that this approximation becomes more and more exact as $R_i$ becomes bigger. Using the notation

$$k_i=\left|\{y_j\in R_i\}\right|$$

(7)

this means

$$p_Y\left(y_i\right)\approx \frac{k_i}{n\mathrm{vol}\left(R_i\right)}.$$

(8)

This formula provides an estimate for $p_Y\left(y_i\right)$ provided that a strategy is given for choosing the small regions $R_i$ around each point $y_i$. As will be seen, a similar formula can be derived for $p_{Y|X}\left(y_i\right|x_i)$, essentially by restricting the points to ${\mathcal{D}}_i=\{(x_j,y_j)\in \mathcal{D}|x_j=x_i\}$:

$$p_{Y|X}\left(y_i\right|x_i)\approx \frac{h_i}{n_c\mathrm{vol}\left(R_i\right)}$$

(9)

where $h_i$ is the number of points in $R_i$ with label $x_i$ and $n_c$ is the total number of points with label $x_i$. In the example here, $n_c=n/n_s$. Once the probability mass functions are estimated, it is easy to estimate the mutual information. However, there is a problem: the estimates also require the volume of $R_i$. In general, a metric space does not have a volume measure. Furthermore while many everyday metric spaces also have coordinates providing a volume measure, this measure it not always appropriate since the coordinates are not related to the way the data are distributed. However, the space that the $y_i$’s belong to is not simply a metric space, it is also a space with a probability density, $p_Y\left(y\right)$. This provides a measure of volume:

$$\mathrm{vol}\left(R_i\right)={\int }_{R_i}{p}_Y\left(y\right)dy$$

(10)

In short, the volume of a region can be measured as the amount of probability mass it contains. This is useful because this quantity can in turn be estimated from data, as before, by counting points:

$$\mathrm{vol}\left(R_i\right)\approx \frac{k_i}{n}.$$

(11)

The problem with this, though, is that it gives a trivial estimate of the probability. Substituting back into the estimate for $p_Y\left(y_i\right)$, Equation (8) gives $p_Y\left(y_i\right)=1$ for all points $y_i$. This is not as surprising as it might at first seem. Probability density is a volume-measure dependent quantity; that is what is meant by calling it a density and is the reason that entropy is not well defined on continuous spaces. There is always a choice of coordinate that trivializes the density.

However, it is not the entropy that is being estimated here. It is the mutual information and this is well defined: its value does not change when the volume measure is changed. The mutual information uses more than one of the probability densities on the space; in addition to $p_Y\left(y_i\right)$, it involves the conditional probabilities $p_{Y|X}\left(y\right|x)$. Using the measure defined by $p_Y\left(y\right)$ does not make these conditional probability densities trivial. The idea behind the metric space estimator is to use $p_Y\left(y\right)$ to estimate volumes. This trivializes the estimates for $p_Y\left(y_i\right)$, but it does allow us to estimate $p_{Y|X}\left(y\right|x)$ and use this to calculate an estimate of the mutual information.

In this way, the volume of $R_i$ is estimated from the probability that a data point is in $R_i$, and this, in turn, is estimated by counting points. Thus, to fix the volume $\mathrm{vol}\left(R_i\right)$, a number h of data points is specified, and for each point, the $h-1$ nearest data points are identified, giving h points in all when the “seed point” is included. This is equivalent to expanding a ball around $y_i$ until it has an estimated volume of $h/n$. This defines the small region $R_i$. The conditional probability is then estimated by counting how many points in $R_i$ are points with label $x_i$, that is, are points in ${\mathcal{D}}_i$. In fact, this just means counting how many of the h points that have been identified are in ${\mathcal{D}}_i$, or, put another way, it means counting how many of the $h-1$ nearest points to the original seed point are from the same stimulus as the seed point. In summary, the small region consists of h points. To estimate $p_{Y|X}\left(y_i\right|x_i)$, the number of points in the small region corresponding to label $x_i$ is counted; this is referred to as $h_i$ so

$$h_i=|\{y_j\in R_i|x_j=x_i\}|=|R_i\cap {\mathcal{D}}_i|.$$

(12)

This is substituted into the formula for the density estimator, Equation (6), to obtain

$$p_{Y|X}\left(y_i\right|x_i)\approx \frac{n}{n_c}\frac{h_i}{h}$$

(13)

where, as before, $n_c$ is the total number of trials for each song. It is assumed that each song is presented the same number of times. It would be easy to change this to allow for different numbers of trials for each song, but this assumption is maintained here for notational convenience. Substituting back into the formula for the estimated mutual information, Equation (4) gives

$$I_0=\frac{1}{n}\sum _{i=1}^n{log}_2\frac{n_s{h}_i}{h}$$

(14)

The calculation of $I_0$ is illustrated in Figure 1. The subscript zero has been added in order to preserve the unadorned I for the information itself and $\tilde{I}$ for the de-biased version of the estimator; this is discussed below.

This estimate is biased, and it gives a non-zero value even if X and Y are independent. This is a common problem with estimators of mutual information. One advantage of the Kozachenko–Leonenko estimator described here is that the bias at zero mutual information can be calculated exactly. Basically, for the estimator to give a value of zero $h_i=h/n_s$ would be required for every i. In fact, while this is the expected value if X and Y are independent, $h_i$ has a probability distribution which can be calculated as a sort of an urn problem. As detailed in [10], performing this calculation gives the de-biased estimator:

$$I\approx \tilde{I}=I_0-I_b$$

(15)

where $I_b$, the bias, is

$$I_b=\sum _{r=1}^h\sum _{c=1}^{n_s}\frac{n_c}{n}u(r-1;n_c-1,h-1,n-n_c){log}_2\frac{n_{c}r}{h}$$

(16)

and u is the probability for the hypergeometric distribution using the parameterization used by the distributions.jl Julia library.

$$u(k;s,h,f)=\left(\genfrac{}{}{0pt}{}{s}{k}\right)\left(\genfrac{}{}{0pt}{}{f}{m-k}\right)/\left(\genfrac{}{}{0pt}{}{s+f}{m}\right)\equiv \mathrm{Hypergeometric}(s,m,f)$$

(17)

Obviously, the estimator relies on the choice of smoothing parameter h. Recall that for a small h, the counting estimates for the number of points in the small region and for the volume of the small regions are noisy. For a large h, the assumption that the probability density is constant in the small region is poor. These two countervailing points of approximation affect $I_0$ and $I_b$ differently. It seems that a good strategy in picking h for real data is to maximize $\tilde{I}\left(h\right)$ over h. This is the approach that will be adopted here.

As an example, we will use a data set recorded from zebra finches and made available on the Collaborative Research in Computational Neuroscience data sharing website [11]. This data set contains a large number of recordings from neurons in different parts of the zebra finch auditory pathway. The original analysis of these data are described in [12,13]. The data set includes different auditory stimuli; here, though only the responses to zebra finch song are considered. There are 20 songs, so $n_s=20$, and each song is presented ten times, $n_c=10$, giving $n=200$. The zebra finch auditory pathway is complex and certainly does not follow a single track, but for our purposes, it looks like

$$\mathrm{auditory}\mathrm{nerve}\to \mathrm{CN}\to \mathrm{MLd}\to \mathrm{OV}\to \mathrm{Field}\mathrm{L}\to \mathrm{HVc}$$

(18)

where CN is the cochlear nuclei; MLd is the mesencephalicus lateralis pars dorsalis, analogous to mammalian inferior colliculus; OV is the nucleus ovoidalis; Field L is the primary auditory pallium, analogous to mammalian A1; and, finally, HVc is regarded as the locus of song recognition. The mapping of the auditory pathway and our current understanding of how to best associate features of this pathway to features of the mammalian brain is derived from, for example [13,14,15,16,17].

In the data set, there are 49 cells from each of MLd and Field L, and here, the entropy is calculated for all 98 of these cells.

3. Results

Our interest in considering the mutual information for bird songs was to check whether the early part of the spike train was more informative about the song identity. It seemed possible that the amount of information later in the spike train would be less than in the earlier portion. This does not seem to be the case.

There are a number of spike train metrics than could be used. Although these differ markedly in the mechanics of how they calculate a distance, it does appear that the more successful among them are equally good at capturing the information content. In Figure 2A, the total mutual information between song identity and spike train is plotted. Here, the Victor–Purpura (VP) metric [7], the spike count, the Earth mover distance (EMD) [18], and the van Rossum metric [8] are considered. The Victor–Purpura metric and van Rossum metric both include a parameter which can be tuned, roughly corresponding to the precision of spike timing. Here, the optimal value for each case has been used, chosen to maximize the average information. These values are $q=32.5$ Hz for the VP metric and $\tau =15$ ms for the vR metric. The mutual information estimator uses the metric to order the points, and each small region contains the $h-1$ points nearest the seed point so the estimator does not depend on the distances themselves, just the order. Indeed, the estimated mutual information is not very sensitive to the choice of q or $\tau$. This is demonstrated in Figure 2B, where the mutual information is calculated as a function of q, the parameter for the VP metric.

The Victor–Purpura metric and van Rossum metric clearly have the highest mutual information and are very similar to each other. This indicates that the estimator is not sensitive to the choice of metric, provided the metric is one that can capture features of the spike timing as well as the overall rate. The spike count does a poor job, again indicating that there is information contained in spike timing as well as the firing rate. Similar results were seen in [9,19], though a different approach to evaluating the performance of the metrics was used there.

The cells from MLd have higher mutual information, on average, than the cells from Field L. Since Field L is further removed from the auditory nerve than MLd, this is to be expected from the information processing inequality. This inequality stipulates that away from the source of information, information can only be lost, not created.

In Figure 3, the information content of the spike trains as a function of time is considered. To achieve this, the spike trains are sliced into 100 ms slices and the information is calculated for each slice. The songs have variable lengths, so the mutual information becomes harder to interpret after the end of the shortest song, marked by a dashed line. Nonetheless, it is clear that the rate of information and the information per spike are largely unchanged through the song.

4. Discussion

As well as demonstrating the use of the estimator for mutual information, we were motivated here by an interest in the nature of coding in spike trains in a sensory pathway. It is clear that the neurons in MLd and Field L are not “grandmother” neurons, responding only to a specific song and only through the overall firing rate. The firing rate contains considerably less information than was measured using the spike metrics. The spike metrics, in turn, give very similar values for the mutual information; this appears to indicate that the crucial requirement of a spike train metric is a “fuzzy” sensitivity to spike timing. This demonstrates the need for an estimator such as the Kozachenko Leonenko estimator used here.

Approaches that do not incorporate spike timings underestimate the mutual information, but histogram methods, which do include timings, are computational impractical for modest amounts of data. A pioneering paper, [19], also examines mutual information for zebra finch songs, but using a histogram approach. The substantial conclusion there was similar to the conclusion here: there was evidence that spike timings are important. However, it seems likely that this early paper was constrained in its estimates by the size of the data set. This is suggested by the way the amount of information measured increased monotonically as the bin-width in the temporal discretization was reduced, a signature of a data-constrained estimate.

Finally, it is observed that it is not the case that the precision of spiking diminishes as the song continues. Since that song can often be identified from the first few spikes of the response, it might be expected that the neuronal firing would become less precise. Precision is metabolically costly. However, although the firing rate falls slightly, the information remains constant on a per-spike basis.