Transfer Entropy Expressions for a Class of Non-Gaussian Distributions

1. Introduction

Granger causality is a well-known concept based on dynamic co-dependence [1]. In the framework of Granger causality, the cause precedes and contains unique information about the effect. The concept of Granger causality has been applied in a wide array of scientific disciplines from econometrics to neurophysiology, from sociology to climate research (see [2,3] and references therein), and most recently in cell biology [4].

Information theory has increasingly become a useful complement to the existing repertoire of methodologies in mathematical statistics [5,6]. Particularly, in the area of Granger causality, transfer entropy [7], an information theoretical measure of co-dependence based on Shannon entropy, has been applied extensively in non-parametric analysis of time-resolved causal relationships. It has been shown that (conditional) mutual information measured in nats and transfer entropy coincide in definition [8–10]. Moreover, for Gaussian-distributed variables, there is a tractable equivalence by a factor of two between transfer entropy and a linear test statistic for Granger causality [11]. Although similar equivalences for non-Gaussian variables have been given in [8], it should be remarked that such equivalences cannot be generalized to non-Gaussian distributions as the linear models underlying the construction of linear test statistics for Granger causality are rendered invalid under assumptions of non-Gaussianity.

The aim of this paper is to present closed-form expressions for transfer entropy for a number of non-Gaussian, unimodal, skewed distributions used in the modeling of occurrence rates, rare events and ‘fat-tailed’ phenomena in biological, physical and actuarial sciences [12]. More specifically, we will derive expressions for transfer entropy for the multivariate exponential, logistic, Pareto (type $ℐ$ – $ℐ$ ) and Burr distributions. As for real-world applications, the exponential distribution is the naturally occurring distribution for describing inter-arrival times in a homogeneous Poisson process. In a similar manner, the exponential distribution can be used to model many other change of state scenarios in continuous settings, e.g., time until the occurrence of an accident given certain specifications. The logistic distribution is of great utility given its morphological similarity to the Gaussian distribution and is frequently used to model Gaussian-like phenomena in the presence of thicker distribution tails. The Pareto distribution (in either of its forms) is used in modeling of size related phenomena such as size of incurred casualties in non-life insurance, size of meteorites, and size of trafficked files over the Internet. The Burr distribution is another distribution used in non-life insurance to model incurred casualties, as well as in econometrics where it is used to model income distribution.

The specific choice of these distributions is contingent upon the existence of unique expressions for the corresponding probability density functions and Shannon entropy expressions. A counter-example is given by the multivariate gamma distribution, which although derived in a number of tractable formats under certain preconditions [12,13], lacks a unique and unequivocal multivariate density function and hence a unique Shannon entropy expression.

Another remark shall be dedicated to stable distributions. Such distributions are limits of appropriately scaled sums of independent and identically distributed variables. The general tractability of distributions with this property lies in their “attractor” behavior and their ability to accommodate skewness and heavy tails. Other than the Gaussian distribution (stable by the Central Limit Theorem), the Cauchy-Lorentz distribution and the Lévy distribution are considered to be the only stable distributions that can be expressed analytically. However, the latter lacks analytical expressions for Shannon entropy in the multivariate case. Expressions for Shannon entropy and transfer entropy for the multivariate Gaussian distribution have been derived in [14] and [11], respectively. Expressions for Shannon entropy and transfer entropy for the multivariate Cauchy-Lorentz distribution can be found in the Appendix.

As a brief methodological introduction, we will go through a conceptual sketch of Granger causality, the formulation of the linear models underlying the above-mentioned test statistic, and the definition of transfer entropy before deriving the expressions for our target distributions.

2. Methods

Employment of Granger causality is common practice within cause-effect analysis of dynamic phenomena where the cause temporally precedes the effect and where the information embedded in the cause about the effect is unique. Formulated using probability theory, under H₀, given k lags and the random variables A and B and the set of all other random variables C in any arbitrary system, B is said to not Granger-cause A at observation index t, if

$$H_0:A_t⫫\{B_{t-1},\dots ,B_{t-k}\}\mid \{A_{t-1},\dots ,A_{t-k},C_{t-1},\dots ,C_{t-k}\}$$

(1)

where ⫫ denotes probabilistic independence. Henceforth, for the sake of convenience, we implement the following substitutions: X = A_t, $Y={\left\{B\right\}}_{t-1}^{t-k}$ and $Z={\{A,C\}}_{t-1}^{t-k}$. It is understood that all formulations in what follows are compatible with any multivariate setting. Thus, one can parsimoniously reformulate the hypothesis in Equation (1) as:

$$H_0:X⫫Y\mid Z$$

(2)

The statement above can be tested by comparing the two conditional probability densities: f_X|Z and f_{X|Y Z} [15].

2.1. Linear Test Statistics

In parametric analysis of Granger causality, techniques of linear regression have been the dominant choice. Under fulfilled assumptions of ordinary least squares’ regression and stationarity, the hypothesis in Equation (2), can be tested using the following models:

$$H_0:X={\beta }_1+Z{\beta }_2+\varepsilon$$

(3)

$$H_1:X={\gamma }_1+Z{\gamma }_2+Y{\gamma }_3+\eta$$

(4)

where the β and γ terms are the regression coefficients, and the residuals ε and η are independent and identically distributed following a centered Gaussian N(0, σ²). Traditionally, the F-distributed Granger-Sargent test [1], equivalent to the structural Chow test [16], has been used to examine the statistical significance of the reduction in residual sum of squares in the latter model compared to the former. In this study however, we will focus on the statistic G(X, Y |Z) = ln(Var_ε/Var_η) [11,17]. This statistic is χ²-distributed under the null hypothesis, and non-central χ²-distributed under the alternate hypothesis. There are two types of multivariate generalizations of G(X, Y |Z); one by means of total variance, using the trace of covariance matrices [18], and one by generalized variance, using the determinant of covariance matrices [11,17]. For a thorough discussion on the advantages of either measure we refer the reader to [18,19]. Choosing the latter extension, the test statistic in G(X, Y |Z) can be reformulated as:

$$\begin{array}{l}G(X,Y\mid Z)=\text{ln\hspace{0.17em}}\left(\frac{\mid {\mathrm{\Sigma }}_{\varepsilon }\mid }{\mid {\mathrm{\Sigma }}_{\eta }\mid }\right)\\ =\text{ln\hspace{0.17em}}\left(\frac{\mid {\mathrm{\Sigma }}_{XZ}\mid ·\mid {\mathrm{\Sigma }}_{YZ}\mid }{\mid {\mathrm{\Sigma }}_Z\mid ·\mid {\mathrm{\Sigma }}_{XYZ}\mid }\right)\end{array}$$

(5)

where the last equality follows the scheme presented in [11].

2.2. Transfer Entropy

Transfer entropy, a non-parametric measure of co-dependence is identical to (conditional) mutual information measured in nats (using the natural logarithm). Mutual information is a basic concept, based on the most fundamental measure in information theory, the Shannon entropy, or, more specifically, the differential Shannon entropy in the case of continuous distributions. The differential Shannon entropy of a random variable S with a continuous probability density f_S with support on is

$$H(S)\equiv -\text{E}[{\text{log}}_b\hspace{0.17em}{f}_S]=-{\int }_S{f}_S\hspace{0.17em}{\text{log}}_b\hspace{0.17em}{f}_{S}ds$$

(6)

where b is the base of the logarithm determining the terms in which the entropy is measured; b = 2 for bits and b = e for nats [14,20]. The transfer entropy for the hypothesis in Equation (2) is defined as [7]:

$$\begin{array}{l}T(Y\to X\mid Z)=H(X\mid Z)-H(X\mid Y,Z)\\ =H(X,Z)-H(Z)+H(Y,Z)-H(X,Y,Z)\end{array}$$

(7)

Interestingly, for Gaussian variables one can show that G(X, Y |Z) = 2 · T(Y → X|Z) [11]. Naturally, such equivalences fail when using other types of distributions that do not meet the requirements of linear models used to construct G(X, Y |Z).

In the following, we shall look at closed-form expressions for transfer entropy for the multivariate exponential, logistic, Pareto (type $ℐ$ – $ℐ$ ) and Burr distributions. Before deriving the results, it should be noted that all marginal densities of the multivariate density functions in this study are distributed according to the same distribution; i.e., the marginal densities of a multivariate exponential density are themselves exponential densities.

3. Results

In this section we will derive the expression for transfer entropy for the multivariate exponential distribution. The remaining derivations follow an identical scheme and are presented in the Appendix. The differential Shannon entropy expressions employed in this study can be found in [21].

The multivariate exponential density function for a d-dimensional random vector S is:

$$f_{\mathbf{S}}=\prod _{i=1}^d\frac{\alpha +i-1}{{\theta }_i}\text{exp\hspace{0.17em}}\left(\frac{s_i-{\lambda }_i}{{\theta }_i}\right)\hspace{0.17em}{\left[\sum _{i=1}^d\text{exp\hspace{0.17em}}\left(\frac{s_i-{\lambda }_i}{{\theta }_i}\right)-d+1\right]}^{-(\alpha +d)}$$

(8)

where S ∈ ℝ^d, s_i > λ_i, θ_i > 0 for i = 1, ..., d and α > 0. For the multivariate exponential distribution the differential Shannon entropy of S is:

$$H(\mathbf{S})=-\sum _{i=1}^d\text{log\hspace{0.17em}}\left(\frac{\alpha +i-1}{{\theta }_i}\right)+(\alpha +d)\sum _{i=1}^d\frac{1}{\alpha +i-1}-\frac{d}{\alpha }$$

(9)

Thus, transfer entropy for a set of multivariate exponential variables can be formulated as:

$$\begin{array}{l}T(Y\to X\mid Z)=H(X,Z)-H(Z)+H(Y,Z)-H(X,Y,Z)\\ =-\sum _{i=1}^{d_X}\text{log\hspace{0.17em}}\left(\frac{\alpha +i-1}{{\theta }_i^{(X)}}\right)-\sum _{i=1}^{d_Z}\text{log\hspace{0.17em}}\left(\frac{\alpha +d_X+i-1}{{\theta }_i^{(Z)}}\right)\\ +(\alpha +d_X+d_Z)\hspace{0.17em}\left(\sum _{i=1}^{d_X}\frac{1}{\alpha +i-1}+\sum _{i=1}^{d_Z}\frac{1}{\alpha +d_X+i-1}\right)\\ -\frac{d_X+d_Z}{\alpha }+\sum _{i=1}^{d_Z}\text{log\hspace{0.17em}}\left(\frac{\alpha +i-1}{{\theta }_i^{(Z)}}\right)-(\alpha +d_Z)\hspace{0.17em}\left(\sum _{i=1}^{d_Z}\frac{1}{\alpha +i-1}\right)\\ +\frac{d_Z}{\alpha }-\sum _{i=1}^{d_Z}\text{log\hspace{0.17em}}\left(\frac{\alpha +i-1}{{\theta }_i^{(Z)}}\right)-\sum _{i=1}^{d_Y}\text{log\hspace{0.17em}}\left(\frac{\alpha +d_Z+i-1}{{\theta }_i^{(Y)}}\right)\\ +(\alpha +d_Z+d_Y)\hspace{0.17em}\left(\sum _{i=1}^{d_Z}\frac{1}{\alpha +i-1}+\sum _{i=1}^{d_Y}\frac{1}{\alpha +d_Z+i-1}\right)\\ -\frac{d_Z+d_Y}{\alpha }+\sum _{i=1}^{d_X}\text{log\hspace{0.17em}}\left(\frac{\alpha +i-1}{{\theta }_i^{(X)}}\right)+\sum _{i=1}^{d_Z}\text{log}\hspace{0.17em}\left(\frac{\alpha +d_X+i-1}{{\theta }_i^{(Z)}}\right)\\ -\sum _{i=1}^{d_Y}\text{log\hspace{0.17em}}\left(\frac{\alpha +d_X+d_Z+i-1}{{\theta }_i^{(Y)}}\right)-(\alpha +d_X+d_Z+d_Y)\\ \left(\sum _{i=1}^{d_X}\frac{1}{\alpha +i-1}+\sum _{i=1}^{d_Z}\frac{1}{\alpha +d_X+i-1}+\sum _{i=1}^{d_Y}\frac{1}{\alpha +d_X+d_Z+i-1}\right)\\ +\frac{d_X+d_Z+d_Y}{\alpha }\end{array}$$

(10)

which, after simplifications, reduces to

$$\begin{array}{l}T(Y\to X\mid Z)=\sum _{i=1}^{d_Y}\text{log\hspace{0.17em}}\left(1+\frac{d_X}{\alpha +d_Z+i-1}\right)\\ -d_Y\hspace{0.17em}\left[\sum _{i=1}^{d_X}\frac{1}{\alpha +i-1}+\sum _{i=1}^{d_Z}\left(\frac{1}{\alpha +d_X+i-1}-\frac{1}{\alpha +i-1}\right)\right]\\ +(\alpha +d_Z+d_Y)\sum _{i=1}^{d_Y}\frac{1}{\alpha +d_Z+i-1}\\ -(\alpha +d_X+d_Z+d_Y)\sum _{i=1}^{d_Y}\frac{1}{\alpha +d_X+d_Z+i-1}\end{array}$$

(11)

where d_X represents the number of dimensions in X, and where α is the multivariate distribution parameter. As stated previously, the expression in Equation (11) holds for the multivariate logistic, Pareto (type $ℐ$ – $ℐ$ ) and Burr distributions as proven in the Appendix. For the specific case of d_X = d_Y = d_Z = 1, the transfer entropy expression reduces to:

$$T(Y\to X\mid Z)=\text{log\hspace{0.17em}}\left(\frac{\alpha +2}{\alpha +1}\right)-\frac{1}{\alpha +2}$$

(12)

In any regard, T(Y → X|Z) depends only on the number of involved dimensions and the parameter α. The latter parameter, α, operates as a multivariate distribution feature and does not have a univariate counterpart. This result indicates that the value assigned to the conditional transfer of information from the cause to the effect decreases with increasing values of α. However, the impact of the multivariate distribution parameter α in this decrease, shrinks rather rapidly as the numbers of dimensions increase.

4. Conclusions

The distributions discussed in this paper are frequently subject to the modeling of natural phenomena, and utilized frequently within biological, physical and actuarial engineering. Events distributed according to any of the discussed distributions are not suitable for analysis using linear models and require non-parametric models of analysis or transformations where feasible.

The focus of this paper has been on non-parametric modeling of Granger causality using transfer entropy. Our results show that the expressions for transfer entropy for the multivariate exponential, logistic, Pareto (type $ℐ$ – $ℐ$ ) and Burr distributions coincide in definition and are dependent on the multivariate distribution parameter α, and the number of dimensions. In other words, the transfer entropy expressions are independent of other parameters of the multivariate distributions.

As underlined by our result, the value of transfer entropy depends in a declining manner on the multivariate distribution parameter α as the number of dimensions increase.