Neural Estimator of Information for Time-Series Data with Dependency

Abstract

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff’s ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations.

1. Introduction

In recent decades, a tremendous effort has been done to explore capabilities of feed-forward networks and their application in various areas. Novel machine learning (ML) techniques go beyond conventional classification and regression tasks and enable revisiting well-known problems in fundamental areas such as information theory. The functional approximation power of neural networks is a compelling tool to be used for estimating information-theoretic quantities such as entropy, KL-divergence, mutual information (MI), and conditional mutual information (CMI). As an example, MI is estimated with neural networks in [1] where numerical results show notable improvements compared to the conventional methods for high-dimensional, correlated data.

Information-theoretic quantities are characterized by probability densities and most classical approaches aim at estimating the densities. These techniques may vary depending on whether the random variables are discrete or continuous. In this paper, we focus on continuous random variables. Examples of conventional non-parametric methods to estimate these quantities are histogram and partitioning techniques, where the densities are approximated and plugged-in into the definitions of the quantities, or methods based on the distance of the k-th nearest neighbor [2]. Despite vast applications of nearest neighbor methods for estimation of information-theoretic quantities, such as the proposed technique in [3], recent studies advocate using neural networks while simulations demonstrate that the accuracy of the estimations improves in several scenarios [1,4]. In particular, the results indicate that by increasing the dimension of the data, the bias of the estimation deteriorates less with neural estimators. In addition to superior performance, a neural estimator of information can be considered to be a stand-alone block and coupled in a larger network. The estimator can then be trained simultaneously with the rest of the network and measure the flow of information among variables of the network. Therefore, it facilitates the implementation of ML setups with constraints on information measures (e.g., information bottleneck [5] and representation learning [6]). These compelling features motivate exploring the benefits of neural networks to estimate other information measures and more complex data structures.

The cornerstone of neural estimators for MI is to approximate bounds on the relative entropy instead of computing it directly. These bounds are referred to as variational bounds and recently have gained attention due to their applications in ML problems. Examples are the lower bounds proposed originally in [7] by Donsker and Varadhan, and in [8] by Nguyen, Wainwright, and Jordan that are referred to as DV bound and NWJ bound, respectively. Several variants of these bounds have been reviewed in [9]. Variational bounds are tight, and the estimators proposed in [1,4,10,11] leverage this property and use neural networks to approximate the bounds and correspondingly the desired information measure. These estimators were shown to be consistent (i.e., the estimation converges asymptotically to the true value) and suitably estimate MI and CMI when the samples are independently and identically distributed (i.i.d.). However, in several applications such as time series analysis, natural language processing, or estimating information rates in communication channels with feedback, there exists a dependency among samples in the data. In this paper, we investigate analytically the convergence of our neural estimator and verify the performance of the method in estimating several information quantities.

Consider several random processes such that their realizations are dependent in time. In addition to common information-theoretic measures such as MI and CMI, more complex quantities can be studied that are paramount in representing these processes. For instance, the (temporal) causal relationship between two random processes has been expressed with quantities such as directed information (DI) [12,13] and transfer entropy (TE) [14]. Both DI and TE have a variety of applications in different areas. In communication systems, DI characterizes the capacity of a channel with feedback [15], while it has several other applications in venues including portfolio theory [16], source coding [17], and control theory [18] where DI is exploited as a measure of privacy in a cloud-based control setup. Additionally, DI was introduced as a measure of causal dependency in [19] which led to a series of works in that direction with applications in neuroscience [20,21] and social networks [22,23]. TE is also a well-celebrated measure in neuroscience [24,25], and the physics community [26,27] to quantify causality for time series. In this paper, we investigate capability of the neural estimator proposed in [11] to be used when the samples in the data are not generated independently.

Conventional approaches to estimate KL-divergence and MI such as nearest neighbor methods can be used for non-i.i.d. data; for example to estimate DI [28] and TE [29,30]. However, it is possible to leverage the benefits of neural estimators highlighted in [1] even though the data are generated from a source with dependency among its realizations. In a recent work [31], the authors estimate TE using the neural estimator for CMI introduced in [4]. Additionally, recurrent neural networks (RNN) are proposed in [32] to capture the time dependency to estimate DI. However, showing convergence of these estimators requires further theoretical investigation. Although the neural estimators are shown to be consistent in [1,4,11] for i.i.d. data, the extension of the proofs to dependent data needs to be addressed. In [32], the authors address the consistency of the estimation of DI by referring to universal approximation of RNN [33] and Breiman’s ergodic theorem [34]. Because RNNs are more complicated to be implemented and tuned, in this paper, we assume simple feed-forward neural networks, which were also proposed in [1,4,11] and in this paper. A conventional step to go beyond i.i.d. processes is to investigate stationary and ergodic Markov processes which have numerous applications in modeling real-world systems. Many convergence results for i.i.d. data such as the law of large numbers can be extended to ergodic processes; however, this generalization is not always trivial. The estimator proposed in [11] exhibits major improvements in estimating the CMI. Nevertheless, it is based on a k-nearest neighbors (k-NN) sampling technique which makes the extension of the convergence proofs to non-i.i.d. data more involved. The main contribution of this paper is to provide convergence results and consistency proofs for this neural estimator when the data are stationary and ergodic Markov.

The paper is organized as follows. Notations and basic definitions are introduced in Section 2. Then, in Section 3, the neural estimator and procedures are explained. Additionally, the convergence of the estimator is studied when the data are generated from a Markov source. Next, we provide simulation results in Section 4 for synthetic scenarios and verify the effectiveness of our technique in estimating CMI and DI. Finally, we conclude the paper in Section 5 and suggest potential future directions.

2. Preliminaries

We begin by describing the notation used throughout the paper, and the main definitions are explained afterwards. Then we review variational bounds which are the basis of our neural estimator.

2.1. Notation

Random variables and their realizations are denoted by capital and lower case letters, respectively. Given two integers i and j, a sequence of random variables $X_i,X_{i+1},\dots ,X_j$ is shown as $X_i^j$, or simply $X^j$ when $i=1$. For a stochastic processes $\mathbf{Z}$, a randomly generated sample is denoted by random variable Z. We indicate sets with calligraphic notation (e.g., $\mathcal{X}$). The space of d-dimensional real vectors is shown as ${\mathbb{R}}^d$. The probability density function (PDF) of a random variable X at $X=x$ is denoted by $p_X(x)$ or equivalently $p(x)$, and the distribution of X, by $P_X$ or simply P. The PDF of multiple random variables $X_1,\dots ,X_i$ is $p_{X_1\dots X_i}(x_1,\dots ,x_i)$ and for simplicity it is represented by $p(x_1,\dots ,x_i)$ in the paper. For the distribution P, ${\mathbb{E}}_P[·]$ denotes the expectation with respect to its density $p(·)$. All the logarithms are in base e.

The convergence of the sequence $X_n$ almost surely (or with probability one) to X is denoted by $X_n\stackrel{\mathrm{a}.\mathrm{s}.}{\to }X$ and is defined as:

$$\mathbb{P}\left(\underset{n\to \infty }{lim}{X}_n=X\right)=1.$$

2.2. Information Measures

The information-theoretic quantities of interest for this work can be written in terms of a KL-divergence, and the available neural estimators originally aim to estimate this quantity. For a random variable X with support $\mathcal{X}\subseteq {\mathbb{R}}^d$, the KL-divergence between two PDFs $p(x)$ and $q(x)$ is defined as:

$$\begin{array}{c}\hfill D(p(x)\parallel q(x)):={\mathbb{E}}_P\left[log\frac{p(X)}{q(X)}\right].\end{array}$$

(1)

Then, CMI can be defined using KL-divergence as below:

$$\begin{array}{c}I(X;Y|Z):=D(p(x,y,z)\parallel p(x|z)p(y,z)).\end{array}$$

(2)

where Y and Z are random variables with support on $\mathcal{Y}$ and $\mathcal{Z}$, which are subsets of ${\mathbb{R}}^d$. In this paper, we are focused on extending the estimators for CMI with non-i.i.d. data, where samples in time-series data might not be independently and identically distributed (e.g., generated from a Markov process); nonetheless, our method and consistency proofs are fairly general and can be applied for estimating KL-divergence as well. Consider a sequence of random samples ${\{(X_i,Y_i,Z_i)\}}_{i=1}^n$ generated from the joint process $(\mathbf{X},\mathbf{Y},\mathbf{Z})$, where the samples are not necessarily i.i.d.. A simple step toward this extension is to verify that the previous neural estimators, e.g., [11], can be used to estimate $I(X;Y|Z)$, where $(X,Y,Z)\sim p(x,y,z)$ and the processes $(\mathbf{X},\mathbf{Y},\mathbf{Z})$ are Markov, as in the following assumption.

Assumption 1.

$(\mathit{X},\mathit{Y},\mathit{Z})$ are jointly stationary and ergodic 1-st order Markov with marginal density $p(x,y,z)$. The extension of the results to d-th order Markov is straightforward.

To explore further in generalizing the neural estimators, it is possible to investigate their capability for information measures that rely on dependent random variables. Consider the pairs ${\{(X_i,Y_i)\}}_{i=1}^n$ to be samples of the processes $(\mathbf{X},\mathbf{Y})$. If the generated samples are dependent in time, it is possible to measure the causal relationship between the processes with quantities such as DI and TE, defined as below:

$$\begin{array}{c}I(X^n\to Y^n):=\sum _{i=1}^{n}I(X^i;Y_i|Y^{i-1})\end{array}$$

(3)

$$\begin{array}{c}{T}_{\mathbf{X}\to \mathbf{Y}}(i):=I(X_{i-J}^{i-1};Y_i|Y_{i-L}^{i-1}),\end{array}$$

(4)

where J and L are parameters of the TE that determine the length of memory to consider for X and $\mathbf{Y}$, respectively. Both quantities are functions of the CMI and Figure 1 visualizes the corresponding variables in each CMI term for DI and TE. In particular, each CMI term in (3) quantifies the amount of shared information between $X^i$ and $Y_i$ conditioned on $Y^{i-1}$, i.e., it excludes the effect of the causal history of $\mathbf{Y}$. In a general form, to express the causal effect of the process $\mathbf{X}$ on $\mathbf{Y}$ conditioning causally on $\mathbf{Z}$, DI is normalized with respect to n which is defined below and denoted as directed information rate (DIR):

$$\begin{array}{cc}\hfill I(\mathbf{X}\to \mathbf{Y}\parallel \mathbf{Z})& :=\underset{n\to \infty }{lim}\frac{1}{n}I(X^n\to Y^n\parallel Z^n)\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}I(X^i;Y_i|Y^{i-1},Z^i).\hfill \end{array}$$

(5)

By assuming the processes to be Markov, (5) can be simplified (see [23,35,36]). To be explicit, if both $(\mathbf{X},\mathbf{Y},\mathbf{Z})$ and $(\mathbf{Y},\mathbf{Z})$ are stationary and ergodic 1-st order Markov, from (5) the DIR can be simplified as:

$$\begin{array}{c}\hfill I(\mathbf{X}\to \mathbf{Y}\parallel \mathbf{Z})=I(X^2;Y_2|Y_1,Z^2),\end{array}$$

(6)

where the CMI is with respect to the stationary density $p(x^2,y^2,z^2)$ of the Markov model. To generalize this approach, let us define the maximum Markov order ($o_{max}$) of a set of processes to be the minimum number o such that the Markov order of the joint random variables of any subset of the processes is less than or equal to o. So if $o_{max}=l$ for $(\mathbf{X},\mathbf{Y},\mathbf{Z})$, then from (5) we can simplify the DIR term as:

$$\begin{array}{c}\hfill I(\mathbf{X}\to \mathbf{Y}\parallel \mathbf{Z})=I(X^{l+1};Y_{l+1}|Y^l,Z^{l+1}).\end{array}$$

(7)

The following example shows how DIR can be computed for a linear data model, and emphasizes on the difference when DIR is conditioned causally on another process.

Example 1.

Consider the following linear model where ${\{W_i\}}_{i=1}^{\infty }$, ${\{W_i^{\prime }\}}_{i=1}^{\infty }$, and ${\{W_i^{″}\}}_{i=1}^{\infty }$ are uncorrelated white Gaussian noises with variances ${\sigma }_x^2$,${\sigma }_y^2$, and ${\sigma }_z^2$ respectively:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{X}_i=W_i\hfill \\ Y_i=a{Y}_{i-1}+Z_{i-1}+W_i^{\prime }\hfill \\ Z_i=X_i+W_i^{″}\hfill \end{array}\right.\end{array}$$

for some $\left|a\right|<1$, and $(X_0,Y_0,Z_0)$ are distributed according to the stationary distribution of the processes $\mathit{X}$, $\mathit{Y}$, and $\mathit{Z}$. This model holds in Assumption 1 and $o_{max}=1$, so $I(\mathit{X}\to \mathit{Y})$ can be computed as:

$$\begin{array}{c}\hfill I(\mathit{X}\to \mathit{Y})=I(X_1^2;Y_2|Y_1)=\frac{1}{2}log\left(1+\frac{{\sigma }_x^2}{{\sigma }_y^2+{\sigma }_z^2}\right),\end{array}$$

while from (7):

$$\begin{array}{c}\hfill I(\mathit{X}\to \mathit{Y}\parallel \mathit{Z})=I(X_1^2;Y_2|Y_1,Z_1^2)=0.\end{array}$$

(8)

As emphasized earlier, (7) holds when $(\mathbf{X},\mathbf{Y},\mathbf{Z})$ and $(\mathbf{Y},\mathbf{Z})$ are Markov with order l. Then the CMI estimators can be used potentially to estimate the DIR. However, the consistency of the estimation still needs to be investigated since the samples are not independent. Before introducing our technique, we review the basics for estimating information measures with neural networks.

2.3. Estimating the Variational Bound

The estimators proposed in [1,4,11] are all based on tight lower bounds on the KL-divergence, such as the DV bound, introduced in [7]:

$$\begin{array}{c}\hfill D(p(x)\parallel q(x))\ge \underset{f\in \mathcal{F}}{sup}{\mathbb{E}}_P\left[f(X)\right]-log{\mathbb{E}}_Q\left[exp(f(X))\right],\end{array}$$

(9)

where p and q are two PDFs defined over $\mathcal{X}$ with corresponding distributions P and Q, respectively, and $\mathcal{F}$ is any class of functions such that $f:\mathcal{X}\to \mathbb{R}$, and the two expectations exist and are finite. Consider a neural network with parameters $\theta \in \Theta$, then $\mathcal{F}$ can be to the class of all functions constructed with this neural network by choosing different values for the parameters $\theta$. In more details, let $f(x)$ to be the end-to-end function of a neural network with parameters $\theta \in \Theta$ and the optimization in the right hand side (RHS) of (9) is equivalent to optimizing over $\Theta$ (as performed in [1]). Nevertheless, we can leverage from the fact that the DV bound is tight when the function is chosen as:

$$\begin{array}{c}\hfill f^{*}(x)=log\frac{p(x)}{q(x)}\forall x\in \mathcal{X}.\end{array}$$

(10)

Thus, the neural network can approximate $f^{*}(x)$ directly and the lower bound can be computed accordingly (as performed in [4,11]).

Definition 1.

For the PDFs $p(x,y,z)$ and $p(x|z)p(y,z)$, define the corresponding distributions on $\mathcal{X}\times \mathcal{Y}\times \mathcal{Z}$ to be $\tilde{P}$ and $\tilde{Q}$, respectively.

Since the CMI can be stated as a KL-divergence (2), the DV bound can be defined for CMI as bellow:

$$\begin{array}{c}\hfill I(X;Y|Z)\ge \underset{f\in \mathcal{F}}{sup}{\mathbb{E}}_{\tilde{P}}\left[f(X,Y,Z)\right]-log{\mathbb{E}}_{\tilde{Q}}\left[exp(f(X,Y,Z))\right],\end{array}$$

(11)

and the bound is tight by choosing

$$\begin{array}{c}\hfill f^{*}(x,y,z)=log\frac{p(x,y,z)}{p(x|z)p(y,z)}\forall x,y,z\in \mathcal{X}\times \mathcal{Y}\times \mathcal{Z}.\end{array}$$

(12)

The main barrier to compute this bound for $f^{*}(x,y,z)$ is that the densities are unknown. This challenge is addressed in [4,11] by proposing neural classifiers that can approximate $f^{*}(x,y,z)$ without knowing the densities. Below we review the steps of the estimation technique provided in [11]:

(1)

Construct the joint batch, containing samples generated according to $p(x,y,z)$.

(2)

Construct the product batch, containing samples generated according to $p(x|z)p(y,z)$.

(3)

Train the neural network with a particular loss function, which we explain later, to approximate $f^{*}(x,y,z)$, i.e., the density ratio of $\frac{p(x,y,z)}{p(x|z)p(y,z)}$.

(4)

Compute (11) using the batches and the approximated function.

To show the consistency of the estimation with this approach, it is crucial to verify if the empirical average with respect to each sample batch converges asymptotically to the corresponding expectations. Additionally, the neural network should be designed and trained to be capable of approximating the density ratio. For i.i.d. data samples, the authors in [4,11] provided the proofs in the form of concentration bounds. In this paper, we extend these proofs for non-i.i.d. data by providing convergence results for the special case of stationary and ergodic Markov processes. In the remainder of the paper, we denote the data by ${\{(X_i,Y_i,Z_i)\}}_{i=1}^n$ which are consecutive samples of the stationary Markov processes $(\mathbf{X},\mathbf{Y},\mathbf{Z})$ with marginal PDF $p(x,y,z)$.

3. Main Results

In this section, we describe our proposed neural estimator in detail. To create the batches, the estimator is equipped with a k-NN sampling block such that the empirical average over the samples converges to the expected mean. Next, we describe the roadmap to show the convergence of the estimation to the true value (i.e., consistency analysis).

3.1. Batch Construction

To create the joint batch it is sufficient to take $(X_i,Y_i,Z_i)$ randomly from the available data. Below we define the joint batch formally using an auxiliary random variable that indicates whether an instance is selected or not (see also Algorithm 1 for the implementation).

Algorithm 1: Construction of the joint batch

Definition 2 (Joint batch).

Let $W_i\sim Ber(\alpha )$ for $i=1,\dots ,n$ be independent random variables, and ${\mathcal{I}}_{\alpha ,n}(W^n):=\{i\mid i\in \{1,\dots ,n\},W_i=1\}$. Then ${\mathcal{B}}_{joint}^{\alpha }$ is defined as

$$\begin{array}{c}\hfill {\mathcal{B}}_{joint}^{\alpha }:=\{(X_i,Y_i,Z_i)\mid i\in {\mathcal{I}}_{\alpha ,n}\},\end{array}$$

(13)

where we use ${\mathcal{I}}_{\alpha ,n}$ to simplify the notation.

Please note that by the law of large numbers, the length of the joint batch is asymptotically $\alpha n$. Next, to construct the product batch we use the method based on the k-NN technique, which is introduced in [11]. Below we define our method denoted by isolated k-NN technique, and explain how the product batch is constructed (see also Algorithm 2).

Algorithm 2: Construction of the product batch

Definition 3 (Product batch).

For $s<n$, let $W_i\sim Bernoulli({\alpha }^{\prime })$ for $i=1,\dots ,s$ be independent random variables, and

$${\mathcal{I}}_{{\alpha }^{\prime },s}(W^s):=\{i\mid i\in \{1,\dots ,s\},W_i=1\}\&{\mathcal{I}}_{{\alpha }^{\prime },s}^c(W^s):=\{1,\dots ,n\}\backslash {\mathcal{I}}_{{\alpha }^{\prime },s}(W^s).$$

Then for any $\zeta \in \mathcal{Z}$ and given the data ${\{(x_i,y_i,z_i)\}}_{i=1}^n$, define ${\mathcal{A}}^{{\alpha }^{\prime },k,n,s}(\zeta ,z^n,w^s)$ as the set of indices of the k nearest neighbors of ζ (by Euclidean distance) among $\{z_i\}$ for $i\in {\mathcal{I}}_{{\alpha }^{\prime },s}^c(w^s)$. Formally, let $\pi :\{1,\dots ,n-s\}\to {\mathcal{I}}_{{\alpha }^{\prime },s}^c(W^s)$ be a bijection such that ${\parallel \zeta -z_{\pi (1)}\parallel }_2\le \dots \le {\parallel \zeta -z_{\pi (n-s)}\parallel }_2$. Then, ${\mathcal{A}}^{{\alpha }^{\prime },k,n,s}(\zeta ,z^n,w^s):=\{\pi (1),\dots ,\pi (k)\}$. So the product batch can be defined as:

$$\begin{array}{c}\hfill {\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}:=\left\{(X_{j(i)},Y_i,Z_i)\mid i\in {\mathcal{I}}_{{\alpha }^{\prime },s}(W^s),j(i)\in {\mathcal{A}}^{{\alpha }^{\prime },k,n,s}(Z_i,Z^n,W^s)\right\}.\end{array}$$

(14)

Hereafter we use ${\mathcal{I}}_{{\alpha }^{\prime },s}$, ${\mathcal{I}}_{{\alpha }^{\prime },s}^c$, and ${\mathcal{A}}^{{\alpha }^{\prime }}(\zeta )$ instead as the remaining parameters can be understood from the context. We refer to this sampling technique as isolated k-NN in the sequel. An example is also provided in Figure 2 for the case of $k=2$.

Remark 1.

Here we emphasize that the isolated indices are selected from the first s indices of samples while the neighbors can be searched among all n indices of data except the ones in ${\mathcal{I}}_{{\alpha }^{\prime },s}(w^s)$. Additionally, note that the length of the product batch is ${\alpha }^{\prime }sk$ asymptotically as $n\to \infty$ because $sk$ also tends to ∞ as we see later in the assumptions of Proposition 2.

3.2. Training the Classifier

As explained earlier, the optimal function for a tight lower bound on the CMI is obtained by the density ratio and to compute that we use the functional approximation power of neural networks. Consider a feedforward neural network with the last layer equipped with the sigmoid function. The network is parameterized with $\theta \in \Theta \subseteq {\mathbb{R}}^h$ where h is the number of parameters, and the neural network function is denoted by ${\omega }_{\theta }:\mathcal{X}\times \mathcal{Y}\times \mathcal{Z}\to [0,1]$. For an input $(X,Y,Z)$ of the network, let $C\in \{0,1\}$ denote the class of the input which determines that the tuple is generated according to $p(x,y,z)$ or $p(x|z)p(y,z)$. To be explicit, the input is either picked from the joint batch (class $C=1$) or the product batch (class $C=0$), and the goal is to learn the network parameters such that it can distinguish the class of new (unseen) queries. Let the loss function be the binary cross-entropy function. So for $\omega$ to be any function with inputs $(x,y,z)$ and ranging between $[0,1]$, the expected loss is defined as:

$$\begin{array}{c}\hfill L(\omega ):=-\mathbb{E}\left[Clog\omega (X,Y,Z)+(1-C)log(1-\omega (X,Y,Z))\right].\end{array}$$

(15)

It is well-established that by minimizing $L(\omega )$, the solution ${\omega }^{*}$ would represent the probability of classifying the input in the class $C=1$ given the input data, i.e., $\mathbb{P}\left(C=1|x,y,z\right)$. In fact, as shown in [11] (Lemma 1) if the prior distribution on the classes is unbiased, by taking the derivative in (15) we have:

$$\begin{array}{c}\hfill \Gamma (x,y,z)=\frac{p(x,y,z)}{p(x|z)p(y,z)}=\frac{{\omega }^{*}(x,y,z)}{1-{\omega }^{*}(x,y,z)}.\end{array}$$

(16)

So from (12) the optimal function can be expressed with $\Gamma (x,y,z)$ as:

$$\begin{array}{c}\hfill f^{*}(x,y,z)=log\Gamma (x,y,z)\forall x,y,z\in \mathcal{X}\times \mathcal{Y}\times \mathcal{Z}.\end{array}$$

(17)

Therefore, by training the neural network, we can approximate the optimal function $f^{*}(x,y,z)$ and estimate the lower bound for CMI.

Consider the neural network ${\omega }_{\theta }$, then the empirical loss function is defined as:

$$\begin{array}{cc}\hfill & L_{emp}({\omega }_{\theta }):=-\frac{1}{2|{\mathcal{B}}_{joint}^{\alpha }|}\sum _{(X,Y,Z)\in {\mathcal{B}}_{joint}^{\alpha }}log{\omega }_{\theta }(X,Y,Z)\hfill \\ \hfill & -\frac{1}{2|{\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}|}\sum _{(X,Y,Z)\in {\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}}log(1-{\omega }_{\theta }(X,Y,Z)),\hfill \end{array}$$

(18)

and the optimal parameters are obtained by solving the following problem:

$$\begin{array}{c}\hfill \widehat{\theta }:=arg\underset{\theta }{min}{L}_{emp}({\omega }_{\theta }).\end{array}$$

(19)

Consequently, we can approximate the density ratio $\Gamma (x,y,z)$ from (16):

$$\begin{array}{c}\hfill \widehat{\Gamma }(x,y,z)=\frac{{\omega }_{\widehat{\theta }}(x,y,z)}{1-{\omega }_{\widehat{\theta }}(x,y,z)}.\end{array}$$

(20)

To avoid having boundary values (i.e., ${\omega }_{\widehat{\theta }}(x,y,z)$ close to zero or 1), the output of the neural network is clipped between $[\tau ,1-\tau ]$ for some small $\tau >0$.

Remark 2.

Please note that $\widehat{\Gamma }(x,y,z)$ approximates the density ratio, if the batch sizes $|{\mathcal{B}}_{joint}^{\alpha }|$ and $|{\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}|$ are balanced. Otherwise, (20) requires a correction coefficient (see [11]). To fulfill this, given the number of samples n, one can choose the parameters such that $\alpha n={\alpha }^{\prime }sk$. Then, by the law of large numbers, the batches will asymptotically be balanced.

3.3. Estimation of the DV Bound

The final step in the estimation of CMI is to compute the lower bound (11) empirically using $\widehat{\Gamma }(x,y,z)$. So by substituting the expectations with empirical averages with respect to samples in the joint and the product batch, the CMI estimator is defined as:

$$\begin{array}{c}\hfill {\widehat{I}}_{DV}^n(X;Y|Z):=\frac{1}{|{\mathcal{B}}_{joint}^{\alpha }|}\sum _{(x,y,z)\in {\mathcal{B}}_{joint}^{\alpha }}log\widehat{\Gamma }(x,y,z)+log\frac{1}{|{\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}|}\sum _{(x,y,z)\in {\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}}\widehat{\Gamma }(x,y,z).\end{array}$$

(21)

In practice, to mitigate the induced inaccuracy due to sampling from the original data, the training and estimation is repeated for several sampling trials. The steps for implementing the estimator are described in Algorithm 3. In the next part, we provide the convergence results for our estimator to validate substitution of the expectations in (11) with empirical averages with respect to the joint and the product batch. Then we show the convergence of the overall estimation to the true CMI value.

Algorithm 3: Estimation of CMI

3.4. Consistency Analysis

The consistency of our neural estimator (i.e., showing that the estimator converges to its true value) is based on the universal functional approximation power of neural networks and concentration results for the samples collected in the joint batch and in the product batch using the isolated k-NN. Informally, Hornik’s functional approximation theorem [37] guarantees that feedforward neural networks are capable of fitting any continuous function. So depending on the true density of the data, there exists a choice of parameters $\tilde{\theta }$ that enables approximating the desired function with any arbitrary accuracy. Next, we show that the empirical loss function $L_{emp}({\omega }_{\theta })$ is concentrated around its mean $L({\omega }_{\theta })$ for any $\theta$. Combining these tools, we are able to minimize the empirical loss function as in (19) and we expect $\widehat{\theta }$ to be close to $\tilde{\theta }$ asymptotically; thus, eventually $\widehat{\Gamma }(x,y,z)$ properly approximates $\Gamma (x,y,z)$. Additionally, the empirical computation of the DV bound is concentrated around the expected value which concludes the consistency of the end-to-end estimation of the CMI.

In this paper, we put the main focus on extending the concentration results provided in [11] (Proposition 1) with Markov assumption on data. Although conventionally many asymptotic results for i.i.d. data are assumed to hold for Markov data as well, the required extensions here are more involved due to the additional complexity of the k-NN method. In the following, we first show the convergence of the empirical average for the joint batch,

$${|{\mathcal{B}}_{joint}^{\alpha }|}^{-1}{\sum }_{(X,Y,Z)\in {\mathcal{B}}_{joint}^{\alpha }}g(X,Y,Z)\to {\mathbb{E}}_{\tilde{P}}[g(X,Y,Z)],$$

where $g(·)$ is any measurable function such that the expectation exists and is finite. As the product batch collects samples corresponding to the k nearest neighbors, convergence results for nearest neighbor regression are invoked to show that the empirical average for the product batch converges to the expectation with respect to the product distribution $\tilde{Q}$,

$${|{\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}|}^{-1}{\sum }_{(X,Y,Z)\in {\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}}g(X,Y,Z)\to {\mathbb{E}}_{\tilde{Q}}[g(X,Y,Z)].$$

Then, we conclude the consistency of the overall estimation.

3.4.1. Convergence for the Joint Batch

One well-known extension to the law of large numbers for non-i.i.d. processes is Birkhoff’s ergodic theorem, and is the basis of our proof to show the following proposition on the convergence of the sample average over the joint batch.

Proposition 1.

Consider the sequence of random variables ${\{(X_i,Y_i,Z_i)\}}_{i=1}^n$ generated under Assumption 1. Consider the distribution $\tilde{P}$ in Definition 1, for any measurable function $g(·)$ such that ${\mathbb{E}}_{\tilde{P}}\left[g(X,Y,Z)\right]$ exists and is finite,

$$\begin{array}{c}\hfill \frac{1}{|{\mathcal{B}}_{joint}^{\alpha }|}\sum _{(X,Y,Z)\in {\mathcal{B}}_{joint}^{\alpha }}g(X,Y,Z)\stackrel{\mathrm{a}.\mathrm{s}.}{\to }{\mathbb{E}}_{\tilde{P}}\left[g(X,Y,Z)\right].\end{array}$$

(22)

Proof. 

See Appendix A. ☐

3.4.2. Convergence for the Product Batch

From Definition 3, the empirical summation over all samples in the product batch is equivalent to averaging $|{\mathcal{I}}_{{\alpha }^{\prime },s}|$ times k-NN regressions. Considering a sequence of pairs ${\{(U_i,V_i)\}}_{i=1}^n$ generated from stationary ergodic processes $(\mathbf{U},\mathbf{V})$, the k-NN regression denotes the problem of estimating $m(u):=\mathbb{E}[V|U=u]$ with $m_n(u):=\frac{1}{k(n)}{\sum }_{j=1}^{k(n)}{V}_{r_j}$ where $r_j$ refers to the j-th nearest neighbor of u among $U_1,\dots ,U_n$. This problem has been well studied when the pairs $(U_i,V_i)$ are generated i.i.d.. For example in [38], the authors show the convergence of $m_n(u)$ as:

$$\begin{array}{c}\hfill \mathbb{P}\left(\int |m_n(u)-m(u)|p(u)du\ge ϵ\right)\le exp(-na{ϵ}^2),\end{array}$$

(23)

for some positive constant a, when $k(n)\to \infty$ and $\frac{k(n)}{n}\to 0$. However, if the pairs are not independent, convergence results require a more advanced condition denoted geometric $\varphi$-mixing condition or geometric ergodicity condition [39,40]. As argued in [39], the geometric ergodicity is not a restrictive statement and holds for a wide range of processes (see also [41]). For instance, linear autoregressive processes are geometrically ergodic [41] (Ch. 15.5.2). Below we review the $\varphi$-mixing condition.

Definition 4

($\varphi$-mixing condition). A process $\mathit{U}$ is ϕ-mixing if for a sequence ${\{{\varphi }_n\}}_{n\in \mathbb{N}}$ of positive numbers satisfying ${\varphi }_n\to 0$ as $n\to \infty$, for any integer $i>0$ we have:

$$\begin{array}{c}\hfill \left|P(A\cap B)-P(A)P(B)\right|\le {\varphi }_{i}P(A),\end{array}$$

(24)

for all $n>0$ and all sets A and B which are members of $\sigma (U_1,\dots ,U_n)$ and $\sigma (U_{n+i},U_{n+i+1},\dots )$, respectively. If $\{{\varphi }_n\}$ is a geometric sequence, $\mathit{U}$ is called geometrically ϕ-mixing.

To show the convergence of the empirical average over the product batch, we make the following assumptions.

Assumption 2.

The sequence ${\{(X_i,Y_i,Z_i)\}}_{i=1}^n$ is geometrically ϕ-mixing.

Assumption 3.

We assume that $\mathcal{Y}$ and $\mathcal{Z}$ are compact.

Proposition 2.

Let the sequence of random variables ${\{(X_i,Y_i,Z_i)\}}_{i=1}^n$ be generated under Assumptions 1–3, and we choose $k(n)$ and $s(n)$ such that:

$$\begin{array}{cc}\hfill & s(n)k(n)=n\hfill \\ \hfill & k(n)\to \infty \hfill \\ \hfill & s(n)\to \infty \hfill \\ \hfill & k(n)/{(logn)}^2\to \infty .\hfill \end{array}$$

(25)

Consider $\tilde{Q}$ defined in Definition 1. Then, for any function $g(·)$ such that ${\mathbb{E}}_{\tilde{Q}}\left[g(X,Y,Z)\right]$ exists and is finite, and additionally,

$$\begin{array}{c}\hfill |g(x,y_1,z)-g(x,y_2,z)|<L_g\left|y_1-y_2\right|\forall x\in \mathcal{X},z\in \mathcal{Z},y_1,y_2\in \mathcal{Y},\end{array}$$

(26)

where $L_g>0$ is the Lipschitz constant, we have that:

$$\begin{array}{c}\hfill \frac{1}{|{\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}|}\sum _{(X,Y,Z)\in {\mathcal{B}}_{prod}^{{\alpha }^{\prime },s}}g(X,Y,Z)\stackrel{\mathrm{a}.\mathrm{s}.}{\to }{\mathbb{E}}_{\tilde{Q}}\left[g(X,Y,Z)\right].\end{array}$$

(27)

Proof. 

See Appendix B. ☐

Remark 3.

Examples of choices for $k(n)$ and $s(n)$ satisfying (25) are for instance $k(n)=n^{\frac{1}{2}}$ and $k(n)={(logn)}^{2+ϵ}$ for some $ϵ>0$. Please note that in [11], the consistencies are shown when $k(n)=\Theta (n^{\frac{1}{2}})$. However, the convergence result in [11] (Theorem 1) is an explicit bound, so the condition on $k(n)$ can be relaxed (choosing a smaller $k(n)$) when we are only interested in the asymptotic behavior.

3.4.3. Convergence of the Overall Estimation

To complete our analysis on the consistency of the neural estimator, it is required to show that the loss function is properly approximated and it converges to the optimal loss as n increase. The following assumptions on the neural network and the densities enable us to show this convergence.

Assumption 4.

For a network ${\omega }_{\theta }$ parameterized with $\theta \in \Theta$, the assumption holds if Θ is closed, $\Theta \subseteq \{\theta |\parallel \theta {\parallel }_2\le K\}$ for some constant $K>0$ and ${\omega }_{\theta }$ is B-Lipschitz, for some constant $B>0$, regarding θ, for all $(x,y,z)$, i.e.,

$$|{\omega }_{{\theta }_1}(x,y,z)-{\omega }_{{\theta }_2}(x,y,z)|\le B{\parallel {\theta }_1-{\theta }_2\parallel }_2,\forall {\theta }_1,{\theta }_2\in \Theta ,(x,y,z)\in \mathcal{X}\times \mathcal{Y}\times \mathcal{Z}.$$

Assumption 5.

There exist $0<p_{min}<p_{max}<\infty$ such that for all $x,y,z\in \mathcal{X}\times \mathcal{Y}\times \mathcal{Z}$, the values of $p(x,y,z)$ and $p(x|z)p(y,z)$ are both in the interval $[p_{min},p_{max}]$, and it holds that

$$\begin{array}{c}\hfill \frac{p_{min}}{p_{max}+p_{min}}\ge \tau ,\end{array}$$

(28)

to guarantee that $\tau \le {\omega }^{*}\le 1-\tau$.

The following theorem concludes the consistency of the end-to-end estimator.

Theorem 1.

Let Assumptions 1, 2, 3, 4, and 5 hold and $k(n)$ and $s(n)$ satisfy (25). Then the CMI estimator ${\widehat{I}}_{DV}^n(X;Y|Z)$ (defined in (21)), converges strongly to $I(X;Y|Z)$, i.e.,

$$\begin{array}{c}\hfill {\widehat{I}}_{DV}^n(X;Y|Z)\stackrel{\mathrm{a}.\mathrm{s}.}{\to }I(X;Y|Z).\end{array}$$

(29)

Proof. 

See Appendix D. ☐

In the next section, we apply our estimator in several synthetic scenarios to verify its capability in estimating CMI and DI.

4. Simulation Results

In this section, we experiment with our proposed estimator of CMI and DI in the following auto-regressive model which is widely used in different applications, including wireless communications [42], defining causal notions in econometrics [43], and modeling traffic flow [44], among others:

$$\left[\begin{array}{c}{X}_i\\ Y_i\\ Z_i\end{array}\right]=A\left[\begin{array}{c}{X}_i\\ Y_i\\ Z_i\end{array}\right]+B\left[\begin{array}{c}{X}_{i-1}\\ Y_{i-1}\\ Z_{i-1}\end{array}\right]+\left[\begin{array}{c}{N}_i^x\\ N_i^y\\ N_i^z\end{array}\right],$$

(30)

where A and B are $3\times 3$ matrices and the rest of variables are d-dimensional row vectors. A models the instantaneous effect of $X_i$, $Y_i$, and $Z_i$ on each other and its diagonal elements are zero, while B models the effect of previous time instance. $N_i^x$, $N_i^y$, and $N_i^z$ (denoted as noise in some contexts) are independent and generated i.i.d. according to zero-mean Gaussian distributions with covariance matrices ${\sigma }_x^2{I}_d$, ${\sigma }_y^2{I}_d$, and ${\sigma }_z^2{I}_d$, respectively (i.e., the dimensions are d and components are uncorrelated). Please note that this model fulfills Assumptions 1 and 2 by setting appropriate initial random variables. Although the Gaussian random variables do not range in a compact set and thus, Assumption 3 does not hold, we could use truncated Gaussian distributions. Such adjustment does not significantly change the statistics of the generated dataset since the probability of finding a value far away from the mean is negligible.

In the following section, we test the capability of our estimator in estimating both conditional mutual information (CMI) and directed information (DI). In both cases, n samples are generated from the model and the estimations are performed according to Algorithms 1 and 2. Then according to Algorithm 3, the joint and product batches are split randomly in half to construct train and evaluation sets. Then the parameters of the classifier are trained with the train set and the final estimation is computed with the evaluation set (Codes are available at https://github.com/smolavipour/Neural-Estimator-of-Information-non-i.i.d, accessed on 20 May 2021).

To verify the performance of our technique, we also compared it with the approach taken in [4,31] which is as follows. Conditional mutual information can be computed by subtracting two mutual information terms, i.e.,

$$\begin{array}{c}\hfill I(X;Y|Z)=I(X;Y,Z)-I(X;Z).\end{array}$$

(31)

So instead of estimating the CMI term directly, one can use a neural estimator such as the classifier based estimator in [4] or the MINE estimator [1], and estimate each MI term in (31) to estimate the CMI. In what follows, we refer to this technique as MI-diff since it computes the difference between two MI terms.

4.1. Estimating Conditional Mutual Information

In this scenario, we estimate $I(X_1;Y_1|Z_1)$ when A and B are chosen to be:

$$A=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 0\\ 0& 0& 0\end{array}\right],B=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 1\\ 0& 0& 0\end{array}\right].$$

Then from (30), the CMI can be computed as below:

$$\begin{array}{cc}\hfill I(X_1;Y_1|Z_1)& =h(X_1|Z_1)-h(X_1|Y_1,Z_1)\hfill \\ \hfill & =h(Y_0+Y_1+N_1^x|Z_1)-h(Y_0+N_1^x|Y_1,Z_1)\hfill \\ \hfill & =h(Y_0+Y_1+N_1^x)-h(Y_0+N_1^x)\hfill \\ \hfill & =\frac{d}{2}log\left(1+\frac{{\sigma }_y^2+{\sigma }_z^2}{{\sigma }_x^2+{\sigma }_y^2+{\sigma }_z^2}\right).\hfill \end{array}$$

(32)

Each estimated value is an average of $T=20$ estimations, where in each round the batches are re-selected while having a fixed dataset. This procedure is repeated for 10 Monte Carlo trials and the data are re-generated for each trial. The hyper-parameters and settings of the experiment are provided in

Table 1. Hyper-parameters.

Hidden units	64
Hidden layers	2 (64 × 64)
Activation	ReLU
$\tau$	${10}^{-3}$
Optimizer	Adam
Learning rate	${10}^{-3}$
Epochs	200

. In Figure 3, the CMI is estimated (as ${\widehat{I}}_{DV}^{n,T}(X_1;Y_1|Z_1)$ in Algorithm 3) with $n=2\times {10}^4$ samples with dimension $d=1$ when ${\sigma }_y=2$, ${\sigma }_z=2$ and by varying ${\sigma }_x$. It can be observed that the estimator can properly estimate the CMI while the variance of the estimation is also small. The latter can be inferred from the shaded region, which indicates the range of estimated CMI for a particular ${\sigma }_x$ over all Monte Carlo trials. Next, the experiment is repeated for $d=10$ and the results are depicted in Figure 4, where we compare our estimation of CMI with the MI-diff approach, which is explained in (31) and each MI term is estimated with the classifier-based estimator proposed in [4]. It can be observed that the means of both estimators are similar; nonetheless, estimating the CMI directly is more accurate and has less variation compared to the MI-diff approach. Additionally, our method is faster since it computes the information term only once, while in the MI-diff approach, two different classifiers are trained to estimate each MI term.

4.2. Estimating Directed Information

DI can explain the underlying causal relationship among processes. This notion has wide applications in various areas. For example, consider a social network where the activities of users are monitored (e.g., the messages times as studied in [23]). The DI between these time-series data expresses how the activity of one user can affect the activity of the others. In addition, to such data analytic applications, DI characterizes the capacity of communication channels with feedback and by estimating the capacity, rates and powers of transmission can be adjusted in radio communications (see for example [32]). Now in this experiment, consider a network of three processes $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{Z}$, such that the time-series data are modeled with (30) with $d=1$ where

$$\begin{array}{c}\hfill A=0,B=\left[\begin{array}{ccc}0& 0& 0\\ b_1& 0& 0\\ 0& b_2& 0\end{array}\right].\end{array}$$

(33)

In this model, where the relations are depicted in Figure 5, the process $\mathbf{X}$ is affecting $\mathbf{Y}$ with a delay and similarly the signal of $\mathbf{Y}$ appears on $\mathbf{Z}$ in the next time instance while an independent noise is accumulated on both steps. The DIR from $\mathbf{X}\to \mathbf{Y}$ in this network can be computed as follows:

$$\begin{array}{cc}\hfill I(\mathbf{X}\to \mathbf{Y})& =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}I(X^i;Y_i|Y^{i-1})\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}H(Y_i|Y^{i-1})-H(Y_i|X^i,Y^{i-1})\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}H(Y_i)-H(Y_i|X_{i-1})\hfill \\ \hfill & =\frac{1}{2}log\left(1+\frac{b_1^2{\sigma }_x^2}{{\sigma }_y^2}\right).\hfill \end{array}$$

(34)

Similarly, for the link $\mathbf{Y}\to \mathbf{Z}$, we have:

$$\begin{array}{cc}\hfill I(\mathbf{Y}\to \mathbf{Z})& =\frac{1}{n}\sum _{i=1}^{n}I(Y^i;Z_i|Z^{i-1})\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^{n}H(Z_i|Z^{i-1})-H(Z_i|Y^i,Z^{i-1})\hfill \\ \hfill & =\frac{1}{n}\sum _{i=1}^{n}H(Z_i)-H(Z_i|Y_{i-1})\hfill \\ \hfill & =\frac{1}{2}log\left(1+\frac{b_1^2{b}_2^2{\sigma }_x^2+b_2^2{\sigma }_y^2}{{\sigma }_z^2}\right).\hfill \end{array}$$

(35)

Next we can compute the true DIR for the link $\mathbf{X}\to \mathbf{Z}$ as:

$$\begin{array}{cc}\hfill I(\mathbf{X}\to \mathbf{Z})& =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}I(X^i;Z_i|Z^{i-1})\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}H(Z_i|Z^{i-1})-H(Z_i|X^i,Z^{i-1})\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}H(Z_i)-H(Z_i|X_{i-2})\hfill \\ \hfill & =\frac{1}{2}log\left(1+\frac{b_1^2{b}_2^2{\sigma }_x^2}{b_2^2{\sigma }_y^2+{\sigma }_z^2}\right).\hfill \end{array}$$

(36)

Please note that the DIR corresponding to other links (i.e., the above links in the reverse direction) is zero by similar computations. Suppose we represent the causal relationships with a directed graph, where a link between two nodes exists if the corresponding DIR is non-zero. Then according to (34)–(36), the causal relationships are described with the graph of Figure 6a.

To estimate the DIR, note that the processes are Markov and the maximum Markov order ($o_{max}$) for the set of all processes is $o_{max}=2$ according to (30) and (33). Hence by (7), we can estimate DIR with the CMI estimator. For instance the DIR for processes $(\mathbf{X},\mathbf{Y})$ can be obtained by:

$${\widehat{I}}_{DV}^n(\mathbf{X}\to \mathbf{Y}):={\widehat{I}}_{DV}^n(X^3;Y_3|Y^2),$$

where the right hand side is computed similar to (21). We performed the experiment with $n=2\times {10}^5$ samples of dimension $d=1$ generated according to the model (30) and (33) with $b_1=1$, $b_2=2$, ${\sigma }_x=3$, ${\sigma }_y=2$, and ${\sigma }_z=1$, while the settings of the neural network were chosen as in Table 1. The estimated values are stated in

Table 2. True and estimated DIR.

	True DIR	Estimation with Our Method (Mean ± Std)
$I(\mathbf{X}\to \mathbf{Y})$	$0.59$	$0.57\pm 0.00$
$I(\mathbf{X}\to \mathbf{Z})$	$0.57$	$0.55\pm 0.00$
$I(\mathbf{Y}\to \mathbf{Z})$	$1.99$	$1.92\pm 0.01$
$I(\mathbf{Y}\to \mathbf{X})$	0	$0.00\pm 0.00$
$I(\mathbf{Z}\to \mathbf{X})$	0	$0.00\pm 0.00$
$I(\mathbf{Z}\to \mathbf{Y})$	0	$0.00\pm 0.00$

. It can be seen that the bias of the estimator is fairly small while the variance of the estimations is negligible. This is inline with the observations in [11] when estimating CMI for i.i.d. case.

Although $I(\mathbf{X}\to \mathbf{Z})>0$, intuitively $\mathbf{X}$ is only affecting $\mathbf{Z}$ causally through $\mathbf{Y}$, which suggests that $I(\mathbf{X}\to \mathbf{Z}\parallel \mathbf{Y})=0$. This event is referred to as proxy effect when studying directed information graph (see [45]). In fact the graphical representation of causal relationships can be simplified using the notion of causally conditioned DIR as depicted in Figure 6b. To see this formally, note that from (30) it yields that:

$$\begin{array}{cc}\hfill I(\mathbf{X}\to \mathbf{Z}\parallel \mathbf{Y})& =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}I(X^i;Z_i|Y^i,Z^{i-1})\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}H(Z_i|Y^i,Z^{i-1})-H(Z_i|X^i,Y^i,Z^{i-1})\hfill \\ \hfill & =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}H(Z_i|Y_{i-1})-H(Z_i|Y_{i-1})\hfill \\ \hfill & =0.\hfill \end{array}$$

(37)

Considering $o_{max}=2$, the causally conditioned DIR terms can be estimated with our CMI estimator according to (7); for instance,

$$\begin{array}{c}\hfill {\widehat{I}}_{DV}^n(\mathbf{X}\to \mathbf{Y}\parallel \mathbf{Z}):={\widehat{I}}_{DV}^n(X^3;Y_3|Y^2,Z^3).\end{array}$$

The estimation results are provided in

Table 3. True and estimated DIR.

	True DIR	Estimation with Our Method (Mean ± Std)
$I(\mathbf{X}\to \mathbf{Y}\parallel \mathbf{Z})$	$0.59$	$0.57\pm 0.00$
$I(\mathbf{X}\to \mathbf{Z}\parallel \mathbf{Y})$	0	$0.00\pm 0.00$
$I(\mathbf{Y}\to \mathbf{Z}\parallel \mathbf{X})$	$1.42$	$1.52\pm 0.01$
$I(\mathbf{Y}\to \mathbf{X}\parallel \mathbf{Z})$	0	$0.01\pm 0.00$
$I(\mathbf{Z}\to \mathbf{X}\parallel \mathbf{Y})$	0	$0.01\pm 0.00$
$I(\mathbf{Z}\to \mathbf{Y}\parallel \mathbf{X})$	0	$0.01\pm 0.00$

for all the links, where for each link we averaged over $T=20$ estimations (as in Algorithm 3); then the procedure is repeated for 10 Monte Carlo trials in which we generate a new dataset according to the model.

In this experiment, we did not explore the effect of higher dimensions for data, although one should note that for the causally conditioned DIR estimation, with $d=1$ the neural network is fed with data of size 9. Nevertheless, the performance of higher dimensions for this estimator with i.i.d. data has been studied in [11] and the challenges of dealing with high dimensions when data has dependency can be considered to be a future direction of this work. Additionally, although the information about $o_{max}$ may not always be available in practice, it can be approximated by data-driven approaches similar to the method described in [45].

5. Conclusions and Future Directions

In this paper, we explored the potentials of a neural estimator for information measures when there exist time dependencies among the samples. We extended the analysis on the convergence of the estimation and provided experimental results to show the performance of the estimator in practice. Furthermore, we compared our estimation method with a similar approach taken in [4,31] (which we denoted as MI-diff), and demonstrations on synthetic scenarios show that the variances of our estimations are smaller. However, the main contribution is the derivation of proofs of convergence when the data are generated from a Markov source. Our estimator is based on a k-NN method to re-sample the dataset such that the empirical average over the samples converges to the expectation with certain density. The convergence result derived for the re-sampling technique is stand-alone and can be adopted in other sampling application.

Our proposed estimator can be used potentially in the areas of information theory, communication systems, and machine learning. For instance, the capacity of channels with feedback can be characterized with directed information and estimated with our estimator and can be investigated as a future direction. Furthermore, in machine learning applications where the data has some form of dependency (either spatial of temporal), regularizing the training with information flow requires the estimator of information to capture causality which is considered in our technique. Finally, information measures can be used in modeling and controlling a complex system and the results in this work can provide meaningful measures such as conditional dependence and causal influence.