Information Theory and Dynamical System Predictability

Abstract

Predicting the future state of a turbulent dynamical system such as the atmosphere has been recognized for several decades to be an essentially statistical undertaking. Uncertainties from a variety of sources are magnified by dynamical mechanisms and given sufficient time, compromise any prediction. In the last decade or so this process of uncertainty evolution has been studied using a variety of tools from information theory. These provide both a conceptually general view of the problem as well as a way of probing its non-linearity. Here we review these advances from both a theoretical and practical perspective. Connections with other theoretical areas such as statistical mechanics are emphasized. The importance of obtaining practical results for prediction also guides the development presented.

1. Introduction

Prediction within dynamical systems originated within the modern era in the study of the solar system. The regularity of such a system on time scales of centuries meant that very precise predictions of phenomena such as eclipses are possible at such lead times. On longer times scales of order million or more years, chaotic behavior due to the multi-body gravitational interaction becomes evident (see e.g., [1]) and effects such as planetary collisions or system ejection can occur. Naturally at such leads where chaotic behavior dominates predictions become far less precise. Of course, predictions over such a timescale are only theoretical and not subject to observational verification.

In the last century prediction within a greater range of practical dynamical systems has been attempted. Perhaps the best known of these have been turbulent fluids such as the atmosphere and ocean as well as earthquake prediction for which the system can be considered even more non-linear. It is fair to say that considerable progress has been made in the first area; the second area has potential currently limited by a lack of observations while in the third area more limited success has been achieved (see e.g., [2]).

Predictions made using a dynamical model typically suffer primarily from two deficiencies: Firstly the model used may have certain inadequacies as a representation of reality and secondly initial conditions for a prediction may not be known exactly. Such problems are known as model errors and initial condition errors respectively. Uncertainty in boundary conditions can sometimes be of importance as well. We omit a discussion of this effect in the present review.

Progress can be made in improving predictions either by improving physical depictions within models or by improving the observational network and thereby reducing errors in the initial conditions. There is considerable evidence for the atmosphere however that no practical observational network will ever eliminate significant prediction errors due to initial condition errors. Even if one was able to define such conditions to the round-off accuracy of the computing device deployed, at some practical prediction time even these minute errors would grow sufficiently large as to overwhelm the forecast made. Such behavior is of course characteristic of dynamical systems classified loosely as chaotic or turbulent.

In general model errors are almost by definition quite hard to study in a systematic fashion since they are caused by quite diverse factors which are not very well understood. In fact if they were better understood they would be removed by improving the dynamical model using this knowledge. Thus the issue of model error tends primarily to be an engineering rather than a theoretical study. We therefore focus our attention here on initial condition errors but it should be borne in mind that model errors can often cause substantial problems for forecasters. It is interesting that the general public often confuses the two issues and attributes errors due to initial condition uncertainty to the inadequacy of meteorologists in reducing model errors. Of course meteorologists are not always averse to using the reverse strategy in response.

Inevitable deficiencies in observing networks imply uncertainty about the initial conditions used in predictions which can therefore be considered random variables. The study of the evolution of such variables thus in a general sense can be considered to be the study of predictability. Naturally functionals connected with uncertainty defined on such variables play an important role in such studies. In the context of atmospheric prediction, most attention has focused on the first two moments of the associated probability functions since often such functions are quasi-normal. Study of the problems from the viewpoint of entropic functionals is the natural generalization of this which allows for a more general treatment and this has received considerable attention in the last decade and is the subject of this review.

The dynamical systems of practical interest are generally of rather high dimensionality and this has posed particular problems for predictability studies. It means that study of the evolution of the multivariate probability distribution is generally impossible and studies are often confined to Monte Carlo (ensemble) predictions. The size of such ensembles are usually far less than the system dimensionality which has led to a wide variety of reduced state space techniques for identifying important error growth directions. Related reduced state space methods can also be used to truncate the dynamical system to a stochastically forced low order system. Such reductions have proved very useful in illuminating dominant dynamical processes and may also be amenable to complete analytical solution.

With regard to predictability studies, the functionals which have been studied most are differential entropy, information divergence and mutual information. This second is also commonly called the Kullback-Leibler divergence and the relative entropy. We use the terminology information divergence or just divergence to distinguish it clearly from the usual entropy. The dynamical evolutions of the first two are of particular interest with the second being defined with respect to the time evolving probability distribution and the equilibrium (climatological) probability distribution of the system. In general these two distributions converge with time and when they are statistically indistinguishable in some sense (to be made precise below) all predictability has been lost. Indeed an illuminating way to view predictability is as a measure of the disequilibrium of a statistical system. All this is discussed further in Section 2, Section 3 and Section 5 below.

The review is structured as follows: The next section is a brief review of the mostly well known properties of the relevant entropic functionals to be discussed. Section 3 discusses general entropic evolution equations in dynamical systems for which the generic Chapman Kolmogorov equation is relevant. Several of these results are not new but all of the material is not widely known outside statistical physics. Section 4 discusses the concept of information (uncertainty) flow within a dynamical system and its applications. Section 5 discusses various approaches to the study of predictability using the previous tools and outlines a number of new results obtained in the last decade. Finally Section 6 draws some general conclusions on work to date and potential future directions.

The approach taken in this review is slanted toward the authors and co-workers work in this area but other perspectives are also outlined and referenced. In general the nature of this work is rather eclectic given the attempt to balance the theoretical with the practical. It is hoped there will be materials here of interest to a rather broad audience. The intention is to connect information theoretic approaches to predictability with the broad area of statistical physics. A different approach to this subject which emphasizes more geophysical applications may be found in the excellent review of [3]. Other more theoretical approaches may be found in [4] and [5]. We have also chosen in the interests of brevity to omit the treatment of data assimilation and data filters from an information theoretic perspective but remind the reader that there is significant effort in that area as well (see, for example, [6] and [7]).

2. Relevant Information Theoretic Functionals and Their Properties

Results quoted without proof may be found in [8]. Consider two vector random variables $\mathit{X}$ and $\mathit{Y}$ with associated countable outcome alphabet A and associated probability functions $p\left(x\right)$ and $q\left(x\right)$ with $x\in A$. The entropy or uncertainty is defined by

$$H\left(\mathit{X}\right)\equiv \sum _{x\in A}{\displaystyle p\left(x\right)log\left(\frac{1}{p\left(x\right)}\right)}$$

which is obviously non-negative. The (information) divergence between $\mathit{X}$ and $\mathit{Y}$ is defined by

$$D\left(p\right|\left|q\right)\equiv \sum _{x\in A}p\left(x\right)log\left(\frac{p\left(x\right)}{q\left(x\right)}\right)$$

This is also non-negative and only vanishes when $p=q$ so can be considered a “distance” function between functions although it is neither symmetric nor satisfies the triangle identity of Euclidean distance. In the limit that the two probability functions are “close” then indeed it does satisfy these identities to arbitrary accuracy. This is an indication that the divergence induces a metric structure in the sense of differential geometry on the space of probability densities. Further discussion can be found in [9].

If we concatenate $\mathit{X}$ and $\mathit{Y}$ then we can define a joint probability function $p(x,y)$ with associated marginal distributions $p\left(x\right)$ and $p\left(y\right)$. The mutual information then measures the “distance” between this joint distribution and one for which $\mathit{X}$ and $\mathit{Y}$ are independent i.e., $p(x,y)=p\left(x\right)p\left(y\right)$. Thus we have

$$I(\mathit{X};\mathit{Y})\equiv D\left(p(x,y)\right|\left|p\left(x\right)p\left(y\right)\right)=\sum _{x,y\in A}p(x,y)log\left(\frac{p(x,y)}{p\left(x\right)p\left(y\right)}\right)$$

Given its definition, the mutual information represents the degree of dependency between different random variables. Finally we can use the joint distribution $p(x,y)$ to define the conditional entropy $H\left(\mathit{X}\right|\mathit{Y})$ as the expected uncertainty in $\mathit{X}$ given that $\mathit{Y}$ is known precisely.

The entropic functionals defined above can be generalized to so called differential entropic functionals defined on random variables with continuous alphabets. It is interesting however to attempt to interpret them as limits of their countable analogs. Thus for example the differential entropy $h\left(\mathit{X}\right)$ is defined as:

$$h\left(\mathit{X}\right)\equiv -{\displaystyle {\int }_{S}p\left(x\right)log\left(p\left(x\right)\right)dx}$$

(1)

where S is the continuous outcome set for $\mathit{X}$. If we convert this to a Riemann sum we obtain

$$h\left(\mathit{X}\right)\sim -{\displaystyle \sum _{i\in \mathsf{\Lambda }}p\left(x_i^{*}\right)log\left(p\left(x_i^{*}\right)\right)\Delta =-{\displaystyle \sum _{i\in \mathsf{\Lambda }}{P}_{i}log{P}_i}}+log\Delta =H(\tilde{\mathit{X}})+log\Delta$$

(2)

where Δ is the (assumed constant) volume element for the Riemann sum partitioning Λ chosen. Clearly as this approaches zero, the second term approaches $-\infty$ and the differential entropy is finite only because $H(\tilde{\mathit{X}})$ diverges to $+\infty$. This latter divergence occurs because the larger the size of the index/alphabet set Λ the larger the entropy since there is increased choice in outcomes. One can overcome this rather awkward limiting process by restricting attention to entropy differences of different random variables in which case the $log\Delta$ term cancels.

By contrast the “differential” divergence is a straightforward limit of the ordinary divergence:

$$\begin{array}{ccc}D\left(p\right|\left|q\right)& \equiv {\int }_{S}p\left(x\right)log(p\left(x\right)/q\left(x\right))dx& \sim \sum _{i\in \mathsf{\Lambda }}p\left(x_i^{*}\right)log(p\left(x_i^{*}\right)/q\left(x_i^{*}\right))\Delta \\ & & =-{\displaystyle \sum _{i\in \mathsf{\Lambda }}{P}_{i}log(P_i/Q_i)}=D\left(P\right|\left|Q\right)\end{array}$$

(3)

Note the cancellation of Δ in the third expression here.

This cancellation effect is also important to the transformational properties of the differential functionals. In particular suppose we have the following general non-linear change of variables:

$$\mathbf{y}=\mathbf{r}\left(\mathbf{x}\right)$$

The transformed probability density $p^{{}^{\prime }}\left(\mathbf{y}\right)$ is well known to be given by

$$p^{{}^{\prime }}\left(\mathbf{y}\right)=p\left({\mathbf{r}}^{-\mathbf{1}}\left(\mathbf{y}\right)\right)\left|det\left\{\frac{\partial {\mathbf{r}}^{-\mathbf{1}}\left(\mathbf{y}\right)}{\partial \mathbf{y}}\right\}\right|$$

(4)

and the change of variables formula for integration has it that

$$d\mathbf{x}=\mathbf{d}\mathbf{y}\left|det\left\{\frac{\partial {\mathbf{r}}^{-\mathbf{1}}\left(\mathbf{y}\right)}{\partial \mathbf{y}}\right\}\right|\equiv d\mathbf{y}\left|detJ\right|$$

(5)

where J is the so-called Jacobian of the transformation. So we have

$$\begin{array}{cc}D(p^{{}^{\prime }}\left|\right|q^{{}^{\prime }})={\int }_{S^{{}^{\prime }}}{p}^{{}^{\prime }}\left(\mathbf{y}\right)log(p^{{}^{\prime }}\left(\mathbf{y}\right)/q^{{}^{\prime }}\left(\mathbf{y}\right))d\mathbf{y}\hfill & ={\int }_{S}p\left(\mathbf{x}\right)log(\left(p\left(\mathbf{x}\right)\left|detJ\right|\right)/\left(q\left(\mathbf{x}\right)\left|detJ\right|\right))d\mathbf{x}=D\left(p\right|\left|q\right)\hfill \end{array}$$

providing the determinant of the transformation does not vanish which is a condition for the non-singularity of the transformation. Notice that this proof does not work for the differential entropy because of the lack of cancellation. The difference of the differential entropy of two random variables will be invariant under affine transformations because then $detJ$ is constant and the integral of the probability density is also constant (unity). For an affine transformation of the form

$${\mathit{x}}^{\prime }=A\mathit{x}+\mathit{c}$$

where the matrix A and vector $\mathit{c}$ are constant one can easily establish in using the similar arguments to above that

$$h\left(A\mathit{X}\right)=h\left(\mathbf{X}\right)+log\left|detA\right|$$

(6)

Notice that this equation also implies that new distributions that are given by

$$p^{\prime }\left(\mathit{x}\right)=p\left({\mathit{x}}^{\prime }\right)$$

have the same differential entropy providing that the affine transformation is volume preserving i.e., $detA=\pm 1$. Note however that in general $D\left(p^{\prime }\right|\left|p\right)$ will be non-zero and positive unless the probability distribution has a symmetry under the transformation.

In addition to the above general transformational invariance and straightforward limiting property, the divergence also satisfies an intuitive fine graining relationship. Thus if one subdivides a particular partitioning Λ into a finer partitioning ${\mathsf{\Lambda }}^{\prime }$ then one can easily establish (using the log sum inequality) that the divergence defined on the new partition is at least as large as the original coarse grained functional. Thus if the limit to the continuous functional is taken in this fine graining fashion then the divergence converges monotonically to the continuous limit. This has the intuitive interpretation that as finer and finer scales are considered, that “differences” in probability functions become larger as finer structure is used to make the assessment of this “difference”.

3. Time Evolution of Entropic Functionals

We begin our discussion within the context of a general stochastic process. A well known result from information theory called the generalized second law of thermodynamics states that under certain conditions the divergence is a non-increasing function of time. We formulate this in terms of causality.

Definition 1. 

Suppose we have two stochastic processes $\mathit{X}\left(t\right)$ and $\mathit{Y}\left(t\right)$ with $t\in R$ denoting time and associated probability functions p and q the same outcome sets $\left\{\mathit{x}\left(t\right)\right\}$. We say that the two processes are causally similar if

$$p\left(\mathit{x}\left(t\right)\left|\mathit{x}\right(t-a)\right)=q\left(\mathit{x}\left(t\right)\left|\mathit{x}\right(t-a)\right)\forall \mathit{x}\forall t\in Rand\forall a>0$$

(7)

This condition expresses intuitively the notion that the physical system giving rise to the two processes is identical and that precise knowledge of the set of values for outcomes at a particular time are sufficient to determine the future evolution of probabilities. This is intuitively the case for a closed physical system however not the case for an open system since in this second case other apparently external variables may influence the dynamical evolution. Note that the condition of probabilistic causality (7) introduces an arrow in time (via the restriction $a>0$). Also note that a time-homogeneous Markov process satisfies this condition.

It is now easy to show that the divergence of two causally similar processes is non-increasing (the proof can be found in [8]). In some systems referred to as reversible (see below) this functional will actually be conserved while in others termed irreversible, it will exhibit a strict decline. Note that this result applies to all causally similar processes but often one is taken to be the equilibrium process and in irreversible processes the divergence then measures the degree of equilibration of the system.

Consider now a continuous time continuous state Markov process governed by the well known (see e.g., [10]) Fokker Planck equation (FPE):

$$\begin{array}{ccc}\hfill {\partial }_{t}p& =& -{\displaystyle \sum _{i=1}^N{\partial }_i\left[A_i(\mathbf{x},t)p\right]}+\frac{1}{2}{\displaystyle \sum _{i,j=1}^N{\partial }_i{\partial }_j\left\{{\left[\mathbf{B}(\mathbf{x},t){\mathbf{B}}^t(\mathbf{x},t)\right]}_{ij}p\right\}}\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill C_{ij}& \equiv & {\left[\mathbf{B}(\mathbf{x},t){\mathbf{B}}^t(\mathbf{x},t)\right]}_{ij}non-negativedefinite\hfill \end{array}$$

(9)

where $A_i$ and $C_{ij}$ are respectively referred to as the drift vector and diffusion tensor. We can derive a series of interesting results regarding the time evolution of both differential entropy and divergence. Detailed proofs may be found in the Appendix. The first result is well known in statistical physics and dynamical systems studies (see, for example, [11] and [12]).

Theorem 2. 

Suppose we have a stochastic process obeying Equation (8) with $\mathbf{B}=0$ and associated probability function f then the ordinary (differential) entropy satisfies the evolution equation

$$h_t=\int f\nabla \cdot \mathbf{A}dx={〈\nabla \cdot \mathbf{A}〉}_f$$

(10)

Notice the importance of $\nabla \cdot \mathbf{A}$ to the entropy evolution. This also measures the rate at which an infinitesimal volume element expands or contracts in the dynamical system. When it vanishes the system is sometimes said to satisfy a Liouville condition. Hamiltonian systems which includes many inviscid (frictionless) fluids satisfy such a condition. We shall use Equation (10) in a central way in the next section when we consider the concept of information flow.

A particular instructive case when $\nabla \cdot \mathbf{A}<0$ occurs in the case of dissipative systems. A damped linear system has this quantity as a negative constant. In such a case the entropy declines since all trajectories end in the same fixed point. In the case that the system has some stochastic forcing and so the diffusion term in the Fokker Planck equation does not vanish then the stochastic forcing generally acts to increase entropy as the following extension shows:

Theorem 3. 

Suppose we have a general stochastic process with probability function p governed by a Fokker Planck equation which is restricted by the stochastic forcing condition

$${\partial }_i{C}_{ij}=0$$

(11)

where we are assuming the summation convention. Then the evolution of the differential entropy is given by the equation

$$h_t={〈\nabla •\mathbf{A}〉}_p+\frac{1}{2}{〈C_{ij}{\partial }_i\left(logp\right){\partial }_j\left(logp\right)〉}_p$$

(12)

The second term here clearly results in no decline in entropy given that $C_{ij}$ is non-negative definite.

In the case of a stochastically forced dissipative system it is possible for the system to reach an equilibrium where the entropy creation due to the stochastic forcing is balanced by its destruction via the dissipative deterministic dynamics. This balance is an example of a fluctuation dissipation result. A particular case of this theorem with $C_{ij}$ constant was stated in [11] and the two terms on the right hand side of (12) were referred to as entropy production and entropy flux respectively. Also within this reference there is a discussion of the relationship of this type of entropy evolution equation to others proposed in statistical physics using Lyapunov exponents (e.g., [13]).

In contrast to the evolution of differential entropy, the divergence is conserved in all systems even dissipative ones providing $\mathbf{B}=0$:

Theorem 4. 

Suppose we have two stochastic processes obeying Equation (8) which have the additional condition that $\mathbf{B}(\mathbf{x},t)=0$ then the divergence of the two processes (if defined) is time invariant.

In many dynamical systems with $\mathbf{B}=0$ if one calculates the divergence with respect to a particular finite partitioning of state space rather than in the limit of infinitesimal partitioning then the conservation property no longer holds and in many interesting cases it declines with time instead and the system equilibrates. This reflects the fact that as time increases the difference in the distributions tends to occur on the unresolved scales which are not measured by the second divergence calculation. Such a coarse graining effect is of course the origin of irreversibility and often the “unresolved” scales are modeled via a stochastic forcing i.e., we set $\mathbf{B}\ne 0$. In such a case we get a strict decline with time of divergence. The following result was due originally to [14], the proof here follows [15]

Theorem 5. 

Suppose we have two distinct (i.e. differing on a set of measure greater than zero) stochastic processes obeying (8) with $C=\mathbf{B}(\mathbf{x},t){\mathbf{B}}^{\mathbf{t}}(\mathbf{x},\mathbf{t})$ positive definite almost everywhere and with associated probability densities f and g then the divergence strictly declines and satisfies the evolution equation

$$D{\left(f\right|\left|g\right)}_t=-{〈C_{ij}{\partial }_i\left(log(f/g)\right){\partial }_j\left(log(f/g)\right)〉}_f$$

(13)

This final theorem shows the central role of stochastic forcing in causing divergence to decline and hence to the modeling of irreversibility. In the context of statistical physics, the non-positive term on the right hand side of (13) with g set to the equilibrium distribution (invariant measure) forms a part of discussions on non-equilibrium entropy production (see [11]).

In stochastic modeling of dynamical systems it is common to separate the state space into fast and slow components and model the former with noise terms and dissipation of the slow modes. Presumably in this case if such a model works well as a coarse grained model for the total unforced and undissipated system then the last two theorems imply that there is a “leakage” of divergence from the slow to the fast components of the system.

It is an empirical fact that in many physical systems of interest, we find that if a slow subspace is chosen which represents a large fraction of the variability of the system then the subspace divergence will always show a monotonic decline which is suggestive that stochastic models of this system may work well. In the more general dynamical system context however it is an interesting question under what circumstances the fine scales can cause an increase in the divergence with respect to the coarse scales. After all the coarse grained system is not closed and so the monotonicity result quoted at the beginning of this section need not necessarily apply. Related questions have rather naturally generated considerable interest in the statistical physics literature since they relate to the fundamental issue of temporal direction. There is a vast literature and interested readers are referred to, for example, [16,17,18] and [19] Chapter 5.

It is possible to extend the Fokker Planck equation to include discontinuous jump processes and then this equation becomes the more general (differential) Chapman-Kolmogorov equation. The additional terms are often referred to (on their own) as the master equation. It is then possible by similar arguments to those given above to conclude that the jump processes result in an additional strict monotonic decline in divergence. The interested reader is referred to Chapter 3 of [15] for a sketch proof and more information and references.

There is also a well known connection between these results and the classical kinetic theory of Boltzmann which applies to dilute gases. In the latter case the probability distribution of molecules in the absence of collisions is controlled by an equation identical to the Fokker Planck equation with $\mathbf{B}=0$. The molecules themselves are controlled by a Hamiltonian formulation which means that the probability equation can be shown to also satisfy $\nabla •\mathbf{A}=0$. Thus both the entropy and divergence are conserved. Collisions between molecules are modeled probabilistically using a hypothesis known as the Stosszahlansatz or molecular chaos hypothesis (see [20] p. 375). This amounts to the insertion of master equation terms and so the resulting Boltzmann equation can be viewed as a Chapman Kolmogorov equation. The particular form of the master equation ensures that both divergence and differential entropy satisfy monotonic declines and increases respectively. This result is known as Boltzmann’s H-theorem and is the traditional origin of irreversibility in statistical mechanics. More detail can be found in standard treatments of statistical mechanics.

In the context of inviscid fluid turbulence there has been considerable interest in the establishment of H-theorems (see e.g., [21,22,23] and references cited therein). A revealing and conceptually powerful way in which such turbulence can be effectively coarse-grained (and hence simplified) is through moment closures such as eddy-damped Markovian methods (see [24]). This is basically equivalent to the stochastic modeling of the fluid (see [25] Chapter 5) in the manner considered above. Interestingly when this particular kind of coarse-graining is performed one is able to establish rigorously that entropy monotonically increases as a fluid approaches statistical equilibrium (see [21]). It is worth re-emphasizing that this state of affairs is not necessarily guaranteed in more general systems for which Equation (12) applies.

4. Information Flow

4.1. Theory

If we partition a closed dynamical system then an interesting question arises regarding the evolution of uncertainty within the subsets chosen. How does it depend on the evolution of the other partitions? In general as we saw in the last section, entropy is not conserved globally unless the Liouville property holds, so the flow of uncertainty within a system does not usually follow the standard continuity equation satisfied by quantities such as energy, charge or mass. This concept of uncertainty flow has some important practical applications since it is clear that reduction of uncertainty in one partition member at a particular time may depend importantly on the reduction of uncertainty in another partition member at an earlier time. Clearly then understanding the flow of uncertainty is potentially important to optimizing predictions.

This issue was first addressed by [26] who studied the propagation of perturbations in simple non-linear dynamical systems using a “moving frame” or co-moving Lyupanov exponent defined by

$$\lambda (v;x_1,x_2)\equiv \underset{t\to \infty }{lim}\frac{1}{t}ln\left[\frac{\Delta (v,x_1,x_2,t)}{\Delta (v,x_1,x_2,0)}\right]$$

where perturbation amplitudes Δ are defined using an $L_2$ norm on a moving interval:

$$\Delta (v,x_1,x_2,t)\equiv {\left[{\int }_{x_1+vt}^{x_2+vt}{\left|\delta \psi (x,t)\right|}^{2}dx\right]}^{\frac{1}{2}}$$

with respect to small deviations in the system variables $\psi (x,t)$. Maximization of this with respect to the moving frame velocity v showed the preferred velocity of growing perturbations. Since regular Lyupanov exponents are often related to entropy production it was natural to try to find an information theoretic counterpart for the co-moving exponents. This turned out empirically and in the systems studied, to be the time lagged mutual information (TLMI) of random variables $I(\mathit{X}\left(t_1\right);\mathit{Y}\left(t_2\right))$ where the random variables are located at different spatial locations. Associated with the TLMI is the natural velocity scale

$$v^{\prime }\equiv \frac{d(\mathit{X},\mathit{Y})}{\left|t_2-t_1\right|}$$

where d is the spatial distance between the random variables. The TLMI turned out to be maximized when this velocity matched that which maximized the co-moving Lyupanov exponent.

In addition to this match of physical propagation scales, mutual information has an appealing interpretation as the reduction in uncertainty of $\mathit{X}\left(t_1\right)$ due to perfect knowledge of $\mathit{Y}\left(t_2\right)$ i.e., roughly speaking, the contribution of uncertainty in the former due to uncertainty in the latter. This follows from the identity

$$I(\mathit{X}\left(t_1\right);\mathit{Y}\left(t_2\right))=h\left(\mathit{X}\left(t_1\right)\right)-h\left(\mathit{X}\left(t_1\right)\right|\mathit{Y}\left(t_2\right))$$

(14)

This measure of information flow was further verified as physically plausible in more complex and realistic dynamical systems by [27]. It was however shown to give misleading results in certain pathological situations by [28]. In particular when both $\mathit{X}$ and $\mathit{Y}$ are subjected to a synchronized source of uncertainty then unphysical transfers are possibly indicated by the lagged mutual information. This is somewhat analogous to over interpreting a correlation as causative. Note that the mutual information reduces to a simple function of correlation in the case that the distributions are Gaussian.

In [29] a new information theoretic measure of flow was suggested which overcame the problem of lack of causality identified in the earlier study. The situation was considered where each spatial location was a Markov process of order q with time being discretized with an interval of $\Delta t$. Thus the probability function at any particular spatial point depends only on the values at this particular location for the previous q times. In such a situation there can, by construction, be no information flow between spatial points. He then tested the deviation from this null hypothesis using a conditional divergence functional. For an order $q=1$ Markov process this transfer entropy is defined as (see [29])

$$T(\mathit{Y}\to \mathit{X},t)\equiv \int \int \int p(x(t+\Delta t),x\left(t\right),y\left(t\right))log\frac{p\left(x\right(t+\Delta t\left)\right|x\left(t\right),y\left(t\right))}{p\left(x\right(t+\Delta t\left)\right|x\left(t\right))}dx(t+\Delta t)dx\left(t\right)dy\left(t\right)$$

Conceptually this represents the expected change in the probability function at a particular spatial location due to perfect knowledge of a random variable at another location and earlier time beyond which that would result from the perfect knowledge of the first variable at the earlier time. This rather long winded description is perhaps better expressed by writing the transfer entropy (TE) in terms of conditional entropies:

$$T(\mathit{Y}\to \mathit{X},t)=h\left(\mathit{X}\right(t+\Delta t\left)\right|\mathit{X}\left(t\right))-h(\mathit{X}(t+\Delta t)\left|\mathit{X}\right(t),\mathit{Y}(t\left)\right)$$

(15)

These results can be easily generalized to the case $q>1$ with the additional penalty of further complexity of form.

An important aspect of the above two measures is their practical computability. The TLMI is a bivariate functional while the TE is a functional of order $q+1$. Computation of entropic functionals in a practical context generally requires Monte Carlo methods (in the predictability context, the so-called ensemble prediction) and often the choice of a coarse graining or “binning” with respect to random variable outcomes. The coarse graining is required in conjunction with Monte Carlo samples to estimate required probability densities. This as usual creates problems in defining functionals of order greater than perhaps five or so since there are then for most practical cases, insufficient ensemble members to reliably sample the required dimensions of the problem. This issue is commonly referred to as the “curse of dimensionality”. The functionals mentioned above can sometimes avoid this problem because of their low order.

A different approach to this problem was taken in [30] who took as their starting point the basic entropy evolution Equation (10) in the absence of stochastic forcing. To simplify things they considered a two random variable system. In this case it is possible to calculate the entropy evolution equation for one of the random variables alone by integrating out the other variable in the (Liouville) evolution equation:

$$\frac{d{H}_1}{dt}=-\int \int p(x_1,x_2)\left[\frac{A_1}{p\left(x_1\right)}\frac{\partial p\left(x_1\right)}{\partial x_1}\right]d{x}_{1}d{x}_2$$

where $A_i$ are the deterministic time tendency components (see Equations (8) and (10)). Now if the random variable $X_1$ was not influenced by the random variable $X_2$ then we would expect the entropy of this first variable to evolve according to the one dimensional version of (10) i.e.,

$$\frac{\partial H_1^{*}}{\partial t}={〈\frac{\partial A_1}{\partial x_1}〉}_p$$

The difference between the actual $X_1$ entropy $H_1$ and the idealized isolated entropy $H_1^{*}$ can be interpreted as the flow of uncertainty from $X_2$ to $X_1$ or expressed differently as the “information flow”

$$T_{2\to 1}\equiv H_1-H_1^{*}$$

In a number of simple dynamical systems this information flow was computed and was found to behave qualitatively but not quantitatively like the transfer entropy of Schreiber discussed above. In particular it was observed that in general flow is not symmetric in both cases i.e.,

$$T_{2\to 1}\ne T_{1\to 2}$$

One would, of course, expect symmetry in the case of the flow of a conserved quantity which is an indication of the peculiarity of uncertainty/information flow.

The approach above was generalized to the case of a finite number of random variables in the two papers [31] and [32]. Much work remains to be done in exploring the implications of this approach in dynamical systems of practical interest. A different generalization of the formalism was proposed in [33] who considered the information flow between two finite dimensional subspaces of random variables. Denoting the two outcome vectors by x and y, these authors considered the (Ito) stochastic system

$$\begin{array}{ccc}\hfill dx& =& (F_1\left(x\right)+F_{12}(x,y))dt\hfill \\ \hfill dy& =& F_2(x,y)dt+BdW\hfill \end{array}$$

(16)

where B is a constant matrix governing the additive stochastic forcing (see (8)). The application envisaged by this system is where the x are slow coarse grained variables while the y are fast fine grained variables. The information flow $T_{y\to x}$ then plays the role of the second entropy production term of Equation (12) since it represents uncertainty generation in the coarse grained variables due to the interaction with fine grained variables. Associated with this system is evidently a full Fokker Planck rather than the simpler Liouville equation. However the general idea of [30] of flow between y and x still goes through in a similar manner. In the important case that the distribution for x is exponential

$$p_1(x,t)=exp(-\varphi (x,t))$$

it is easily derived that the information flow has the simple form

$$T_{y\to x}=-{\langle {\nabla }_x·F_{12}\rangle }_p+{\langle F_{12}·{\nabla }_x\varphi \rangle }_p$$

and in the case that ϕ and $F_{12}$ are polynomials in x this reduces to a matrix equation in moments of the appropriate exponential distribution family. Such distributions are, of course, widely seen in equilibrium statistical mechanics (Gibbs ensembles) or in quasi-equilibrium formulations of non-equilibrium statistical mechanics [34].

In the case that the $F_1$, $F_{12}$ and $F_2$ satisfy certain natural divergence free conditions common in fluid systems, the authors are able to derive an interesting relation between the divergence rate of change and the information and energy flows within the system:

$$D(p_1\left|\right|p_{1eq}{)}_t=-T_{y\to x}+\frac{1}{{\sigma }^2}{E}_{y\to x}$$

where we are assuming B in (16) is diagonal with the white noise forcing components having constant variance ${\sigma }^2$ and $p_{1eq}$ is the coarse grained equilibrium distribution. This equation is to be compared with (13) where the direct stochastic representation of the fine grained variables causes the information divergence to decline.

In a strict formal sense the methods detailed in [30,31,32,33] are (in the view of this author) to be preferred to the more empirical methods associated with TLMI and TE since they are derived from the fundamental entropy evolution equation in the identical manner that flows of conserved quantities are derived between subsystems. One might criticize even these approaches since there is no insistence on a monotonic increase in entropy in the governing equations. In the view of this author this is less of an issue since many stochastic dynamical systems of importance to geophysical applications do not obey such a restriction and yet the flow of uncertainty in such systems is a very natural subject for investigation. Note however that the information divergence on the other hand does obey such a monotonic principle at least in the stochastic systems of Section 3. In many of the simple models investigated to date it is reassuring that the different types of measures proposed do not show significant qualitative difference. These connections on the other hand clearly deserve further much broader investigation. The empirical measures have strong practical advantages as we shall see below and in addition have clear and important practical interpretations in terms of conditional uncertainty reduction so are likely to play an important role in the future.

4.2. Applications

One important practical application of the preceding formal development occurs in the problem of forecasting within dynamical systems. In high dimensional systems, errors in the initial conditions inevitably occur because the observing system typically is only able to partially resolve state space. More specifically for geophysical prediction, the high dimensionality is caused by the requirement of adequately resolving the spatial domain of interest and any observing network used to define initial conditions is typically not comprehensive with regard to model resolution.

These errors propagate and magnify as the prediction time increases (see next section). One approach to improving predictions is therefore obviously to reduce initial condition errors. Unfortunately however the very large improvements in observing platforms required to achieve this for all initial conditions are very expensive. Frequently however predictions in specific areas are regarded as having a high utility (for example, a storm forecast over a large city) and the initial conditions errors affecting such specific predictions are believed to be confined to very particular geographical locations. Given such a scenario, a targeted observation platform improvement strategy may be a very effective and inexpensive method of improving important predictions.

For the above strategy to work however the sensitivity of prediction errors to initial condition errors must be well known. In atmospheric prediction, common methods of identifying this sensitivity are via linearization of the atmospheric dynamical system (see e.g., [35]) or by the use of a Kalman filter (see e.g., [36]). Such methods however make rather restrictive assumptions namely that linear dynamics are appropriate or that prediction distributions are Gaussian. These are questionable particularly for long range, low skill predictions since the dynamical systems involved are highly non-linear and the typical states therefore turbulent.

The measures of uncertainty/information flow discussed above make none of these restrictive assumptions so are obvious candidates for improving this sensitivity identification. This can be seen directly in the simplest measure, the TLMI, by consideration of Equation (14) with $t_1$ set to the prediction time and $t_2$ to the initial time. The TLMI then tells us how much the prediction random variable $X\left(t_1\right)$ would have its uncertainty reduced if we could reduce the uncertainty of the initial condition random variable $Y\left(t_2\right)$ to zero. This is precisely the sensitivity we require for targeted initial condition improvement. The fact that the TLMI is a bivariate functional also means that Monte Carlo techniques are sufficient for its calculation.

The feasibility of using this functional in a practical setting was investigated in the atmospheric case by [37]. The author used a fairly realistic global model of the atmosphere. Initial conditions for the model were drawn as sample members from a simple multivariate Gaussian distribution intended to represent errors due to observing network sparseness. Each sample/ensemble member was integrated for around a week using the dynamical model and the TLMI calculated for a prediction at a specific location and dynamical variable of interest. In general the functional tended (in mid-latitudes at least) to peak strongly in small regions to the west of the prediction location. This geographically confined sensitivity lasted for the duration of the ensemble prediction. Two examples at 6 days are shown in Figure 1a and Figure 1b which are reproduced from [37]. Calculated here are the TLMI with prediction and initial condition variables both the same and respectively temperature and zonal wind. Both random variables are at a near surface location as there is little vertical propagation of uncertainty. The sensitivity is well confined geographically and it is worth observing that by 6 days an ensemble prediction for the regions displayed typically exhibits marked non-linear effects. The TE of order 1 between the prediction time and the initial conditions was also calculated and found to give similar results except in the vicinity of the location of the particular prediction variable under consideration. Consideration of the right hand side of Equation (15) shows that this result is not surprising.

The application of TE to non-linear time series such as those useful in biological situations has been considered by [28] who studied the interaction of heart and breathing rates. He found that unlike the TLMI, the TE exhibited an asymmetry with the transfer entropy from heart to breathing being generally greater than that in the reverse direction. This suggested the dominance causally of the heart which accords with basic biological understanding. Their results were however somewhat sensitive to the coarse graining chosen to evaluate the various probability densities. This sensitivity was not apparent qualitatively in the predictability applications discussed earlier given a sufficiently large ensemble.

5. Predictability

5.1. Introduction

In seeking to make predictions in any dynamical system one always needs to deal with uncertainty. This is particularly so if such a system is large and complex. In general predictions within such systems are usually made by solving an initial value problem using a set of partial differential equations. Uncertainty is introduced since the initial value vector can never be precisely defined and indeed observational limitations mean that this error can be a small but appreciable fraction of the desired accuracy of any practical prediction.

As mentioned earlier, when predictions of the real world are attempted one also faces an uncertainty associated with its representation by a set of differential equations. In general, this second kind of uncertainty which is often called model error, can only be measured rather indirectly via the careful comparison of model predictions to reality and an attempted disentanglement of the two kinds of error. This is a complex undertaking and so here we usually restrict our attention only to errors of the first kind and deal with what is sometimes referred to as perfect model predictions.

5.2. Error Growth Approaches

We briefly review the traditional approach to the predictability problem in geophysical applications. More detail can be found in the review [38], Chapters 2 and 3 of [19] and the book [39]. This approach studies the separation of initially very close dynamical trajectories and is concerned with the (initially exponential) rate of growth of such errors as well as the associated vectors or state space patterns of maximal growth. The approach was pioneered in the 1960s by Edward Lorenz of MIT and a review may be found in [40]. It has the great virtue of simplicity and hence transparency.

In dynamical systems theory this study has led naturally to the concepts of Lyapunov exponents and vectors. One may project an arbitrary infinitesimal error onto these vectors and each component grows exponentially at the rate controlled by the corresponding exponent. The exponents (unlike the vectors) are global quantities i.e., do not depend on the particular initial state (see [41]). They measure the (global) divergence of trajectories within the system and so are related naturally to the rate of uncertainty growth associated with infinitesimally small errors. The largest exponent clearly gives an approximate global e-folding time for small errors within the system so can be used to roughly deduce a predictability limit since the time at which errors have grown to be comparable with typical system deviations is the time at which predictability is essentially lost. It is worth emphasizing the limitations of this estimate since the growth rate analysis applies only to infinitesimally small errors and not to the later and finite errors (see below).

The interpretation of the Lyapunov exponents as measures of uncertainty growth finds formal expression in the Kolmogorov-Sinai entropy $h_{KS}$ which measures the global time rate of change of the entropy of system trajectories evaluated using a continuum limit of dynamical system partitioning. Since the exponents are measuring the exponential rate of growth of infinitesimal state space volumes in various directions, the discussion of Section 3 on entropy evolution indicates intuitively at least that this concept appears to imply that $h_{KS}$ is the sum of all positive Lyupanov exponents. Only the positive exponents are important since at the infinitesimal limit of errors (or partitioning) the negative exponents do not contribute to uncertainty change. Rigorously [42] has established that in a general class of systems $h_{KS}$ is bounded above by this positive sum. For more (mathematical) discussion on when this upper bound is achieved and more detail in general see the review [43].

In a practical context, ideas of error growth similar to those above have proven highly influential since they often imply a very large reduction in the degrees of freedom of the high dimensional dynamical system underlying realistic turbulent systems. This naturally aids in predictability analysis. Since the calculation of Lyapunov vectors is often technically difficult in practical situations, instead either singular vectors and values are derived using a linearization of the system (see [38]) or “bred” vectors (and values) are derived by iterating a system over a fixed time interval of relevance (see [44]). Vectors of this type form the basis for operational probabilistic prediction since they are assumed to span the important directions of error growth (see below). Again note however that this argument is really only valid over rather short prediction times where the linearization is valid or the bred vector iteration time relevant.

The restriction of the previous methods to small errors and short prediction times has led to finite size generalizations of the Lyapunov vectors and exponents (see [45,46] and for a related normal mode concept [47]). Associated with these finite concepts has been a coarse-grained version of $h_{KS}$ (see, for example, [48]) which is evaluated with respect to a coarse partition of state space rather than the continuum limit assumed for $h_{KS}$. Idealized dynamical systems with many degrees of freedom have been analyzed with these coarse-graining tools and it has been discovered that the usual (infinitesimal) Lyapunov exponent analysis is often irrelevant to practical predictability and the analysis shows that much smaller “hydrodynamical” Lyupanov exponents and coarse grained vectors are more important (see [19] p. 50 and p. 65). This remains a very important and active area of research and the interested reader is referred to the excellent recent monograph [19].

5.3. Statistical Prediction

For the remainder of this review we consider approaches that consider the probability distribution associated with the full state space. As we shall see these approaches are not restricted to small errors or short prediction times however the issue of coarse graining discussed above does play a central role in obtaining approximate results in realistic systems.

Probabilistic approaches to prediction have a long history most particularly in the context of the atmosphere. An excellent survey of this may be found in the review [38]. In the practical context, probabilistic prediction is implemented by sampling an initial condition distribution and this Monte Carlo methodology is called “ensemble prediction”. This procedure is discussed in more detail below. Historically this sampling is often restricted to subspaces spanned by the error vectors discussed in the last subsection (see e.g., [49] for Lyapunov vectors; [35] for singular vectors and [44] for bred vectors) but other approaches involving only the initial condition probability distribution have been proposed (see e.g., [50]). These subspace approaches are motivated in part by the practical consideration that ensemble sample sizes are, for complex and expensive models, quite small compared to the dimension of the state space under consideration.

The present review is focussed on using information theoretic functionals to analyze the evolution of prediction probability distributions. As a consequence little attention is paid to how the initial distribution is defined. The philosophy adopted is typically to assume that this task has been adequately addressed using a good data assimilation methodology. Ensemble prediction sampling when utilized is assumed to be done in a straightforward unbiased fashion from the assumed initial condition distribution. No restriction of the initial condition distribution to a particular subspace is assumed.

We also restrict our focus here to dynamical systems associated with turbulent fluids such as the atmosphere and ocean. Here routine predictions are common and in many situations demonstrably skillful. Additionally the degradation of this skill with prediction time is particularly evident. Predictions within the solar system are generally much more skillful but the degradation of this skill is not as pronounced on practical time scales (see e.g., [1]). As we shall argue below, the degradation time scale is closely related to the statistical equilibration time of the dynamical system. For the atmosphere this is of the order of a month or so. In the ocean it can be two to three orders of magnitude longer than this. In the solar system it is several orders of magnitude longer still [51] and so cannot be observed.

Given the uncertainty in the initial value vector, a powerful perspective regarding prediction is to regard it fundamentally as a statistical rather than deterministic undertaking. One is interested not just in the prediction itself but also in the associated uncertainty. Thus from a mathematical viewpoint, prediction is best considered as the study of the temporal evolution of probability distributions associated with variables of practical interest. Such a probability distribution will be referred to here as the prediction distribution.

Statistical predictions within such systems are often not considered in isolation but are evaluated against some reference distribution associated with the known long term behavior of the system. This might be considered from a Bayesian perspective as representing a prior distribution of the system since it gives the statistical knowledge of the system prior to any attempted prediction. This reference or prior distribution will be referred to here as the equilibrium distribution since under an ergodic hypothesis, the longer term statistical behavior of the system can also be considered to be the equilibrium behavior. In Bayesian terminology the prediction distribution is referred to as the posterior since it is a revised description of the desired prediction variables once initial condition knowledge is taken into account.

Different aspects of the prediction and equilibrium distribution figure in the assessment of prediction skill. Indeed a large variety of different statistical functionals has been developed which are tailored to specific needs in assessing such predictions. It is, of course, a subjective decision as to what is and is not important in such an evaluation and naturally the choice made reflects which particular practical application is required. An excellent discussion of all these statistical issues is to be found in [52] which is written from the practical perspective.

Given that statistical prediction is essentially concerned with the evolution of probability distributions and the assessment of such predictions is made using measures of uncertainty, it is natural to attempt to use information theoretic functionals in an attempt to develop greater theoretical understanding. In some sense such an approach seeks to avoid the multitude of statistical measures discussed above and instead focus on a limited set of measures known to have universal importance in the study of dynamical systems from a statistical viewpoint. The particular functionals chosen to do this often reflect the individual perspective of the theoreticians and it is important to carefully understand their significance and differences.

5.4. Proposed Functionals

To facilitate the discussion we fix notation. Suppose we make a statistical prediction in a particular dynamical system. Associated with every such prediction is a time dependent random variable $\mathit{Y}\left(t\right)$ with $t\ge 0$ and this variable has a probability density $p(\mathit{y},t)$ where the state vector for the dynamical system is $\mathit{y}$. Naturally there is also a large collection of such statistical predictions possible each with their own particular initial condition random variable. We denote the random variable associated with the total set of predictions at a particular time by $\mathit{X}\left(t\right)$ with associated density $q(\mathit{y},x,t)$ where x is some labeling of the different predictions. For consistency we have

$$p_x(\mathit{y},t)=q(\mathit{y},x,t)$$

(17)

Note that the particular manner in which a collection of statistical predictions are assembled remains to be specified. In addition another important task is to specify what the initial condition distributions are exactly. Clearly that issue is connected with an assessment of the observational network used to define initial conditions.

To the knowledge of this reviewer, the first authors to discuss information theoretic measures of predictability in the atmospheric or oceanic context were [53] who considered a particular mutual information functional of the random variables $\mathit{X}\left(0\right)$ and $\mathit{X}\left(t\right)$. More specifically, they considered a set of statistical predictions with deterministic initial conditions which had the equilibrium distribution. This is a reasonable assumption because it ensures a representative sample of initial conditions. The framework was considered in the specific context of a stochastic climate model introduced earlier by the authors. In a stochastic model deterministic initial conditions are meant to reflect a situation where errors in the unresolved fine grained variables of the system are the most important source of initial condition errors. In the model considered, the equilibrium distribution is time invariant which implies that this is also the distribution of $\mathit{X}\left(t\right)$ for all t. As we saw in the previous section we have the relation

$$I\left(\mathit{X}\right(t);\mathit{X}(0\left)\right)=h\left(\mathit{X}\right(t\left)\right)-h\left(\mathit{X}\right(t\left)\right|\mathit{X}\left(0\right))$$

(18)

so the functional proposed measures the expected reduction in the uncertainty of a random variable given perfect knowledge of it at the initial time [54]. This represents therefore a reasonable assessment of the expected significance of initial conditions to any prediction. As $t\to \infty$ the second conditional entropy approaches the first unconditioned entropy since knowledge of the initial conditions does not affect uncertainty of $\mathit{X}\left(t\right)$. Thus as the system equilibrates this particular functional approaches zero. Note that it represents an average over a representative set of initial conditions so does not refer to any one statistical prediction. We shall refer to it in this review as the generic utility. The authors referred to it as the transinformation. We adopt the different terminology in order to contrast it with other proposed functionals (see below).

A few years later a somewhat different approach was suggested by [55] who rather than considering a representative set of statistical predictions, considered instead just one particular statistical prediction. They then defined a predictability measure which was the difference of the differential entropy of the prediction and equilibrium distributions:

$$R\left(t\right)=h\left(\mathit{Y}\right(\infty \left)\right)-h\left(\mathit{Y}\right(t\left)\right)$$

(19)

which they call the predictive information [56]. This has the straightforward interpretation as representing the reduction in uncertainty of a prediction variable $\mathit{Y}\left(t\right)$ compared to the equilibrium or prior variable $\mathit{Y}\left(\infty \right)$. This appears to be similar to Equation (18) conceptually since if the former author’s framework is considered, then $h\left(\mathit{Y}\right(\infty \left)\right)=h\left(\mathit{X}\right(t\left)\right)$ but remember that (19) refers to just one particular statistical prediction. In the special case that both the prediction and equilibrium distributions are Gaussian one can derive the equation [57]

$$R\left(t\right)=-\frac{1}{2}log\left(det\left(C\left(t\right)C^{-1}\left(\infty \right)\right)\right)$$

(20)

where the matrix $C\left(t\right)$ is the covariance matrix for the Gaussian distribution at time t. The matrix $C\left(t\right)C^{-1}(t=\infty )$ is easily seen to be non-negative definite assuming reasonable behavior by the equilibrium covariance. The eigenvectors of this matrix ordered by their non-negative eigenvalues (from greatest to least) then contribute in order to the total predictive information. Thus these vectors are referred to as predictability patterns with the first few such patterns often dominating the total predictive information in practical applications. In a loose sense then these are the predictability analogs of the principal components whose eigenvalues explain variance contributions rather than predictive information. Note that the principal components are the eigenvectors of $C\left(\infty \right)$.

Another approach to the single statistical prediction case was put forward by the present author in [58] (see also [59]). They proposed that rather than considering the difference of uncertainty of the prior and posterior distributions that the total discrepancy of these distributions be measured using the divergence functional. The motivation for this comes from Bayesian learning theory (see [60] Chapter 2). Here a prior is replaced after further observational evidence by a posterior and the discrepancy between the two, as measured by the divergence, is taken as the utility of the learning experiment. Information theory shows that the divergence is the coding error made in assuming that the prior holds when instead the posterior describes the random variable of interest. Given this background, the divergence can be regarded as a natural measure of the utility of a perfect prediction. We shall refer to this functional as the prediction utility.

These two functionals differ conceptually since one uses the reduction in uncertainty as a measure while the other measures the overall distribution change between the prior and posterior. This difference can be seen abstractly as follows: Consider a translation and a rotation of a distribution in state-space then as we saw in section 2, the difference of the entropy of the original and transformed distributions is zero but the divergence of the two distributions is generally positive. These issues can be seen explicitly by calculating the divergence for the Gaussian case and comparing it to (20):

$$\begin{array}{c}D\left(p\left(t\right)\right|\left|p\left(\infty \right)\right)=-\frac{1}{2}log(det\left(C\left(t\right)C^{-1}\left(\infty \right)\right))+\frac{1}{2}tr\left([C\left(t\right)-C\left(\infty \right)]C^{-1}\left(\infty \right)\right)+\frac{1}{2}\overline{\mathbf{y}}\left(t\right){}^t{C}^{-1}\left(\infty \right)\overline{\mathbf{y}}\left(t\right)\end{array}$$

(21)

where we are assuming for clarity that the mean vector $\overline{\mathbf{y}}\left(\infty \right)=0$. The first term on the right here is identical to (20) so the difference in the measures comes from the second and third terms. The third non-negative definite term arises from the distributions having different means which is analogous to the translation effect discussed above. The second term is analogous to the rotation effect discussed above: Suppose hypothetically that the equilibrium Gaussian distribution is in two dimensions with a diagonal covariance matrix but with variances in the two directions unequal. Consider hypothetically that the prediction distribution is a rotation of this distribution through $\pi /2$. Evidently the divergence of the two distributions will not occur through the first or third term in (21) since the mean vector is unchanged as is the entropy. Thus the second term must be the only contributor to the positive divergence.

A very simple concrete example illustrates the differences discussed. Suppose that temperature at a location is being predicted. Further suppose that (realistically) the historical and prediction distributions for temperature are approximately normal. Next suppose that the historical mean temperature is $20{}^{\circ }$C with a standard deviation of $5{}^{\circ }$C. Finally suppose that the statistical prediction for a week in advance is $30{}^{\circ }$C with a standard deviation of $5{}^{\circ }$C. It is clear that the first measure discussed above would be zero whereas the second would be $2.0$ and due only to the third term in (21). Clearly no reduction in uncertainty has occurred as a result of the prediction but the utility is evident. Obviously this example is somewhat unrealistic in that one would usually expect the prediction standard deviation to be less than $5{}^{\circ }$C but nevertheless the utility of a shift in mean is clear and is intuitively of a different type to that of reduced uncertainty. Prediction utility of this kind in a practical context is partially measured by the widely used anomaly correlation skill score (see [61]).

The transformational properties for the three functionals above were discussed in Section 2: The divergence is invariant under a general non-degenerate non-linear transformation of state space. The mutual information, since it is the divergence of two particular distributions, can also be shown to have this property. The difference of differential entropies is invariant under the smaller class of affine transformations. Note that this does not cover all invariant transformations for this case since there are non-affine transformations with $detJ$ constant.

There is an interesting connection between the three proposed functionals that was first observed by [3]: Suppose we take the general stochastic framework assumed above (not just the specific model but any stochastic model) and calculate the second and third functionals and form their expectation with respect to the set of statistical predictions chosen. It is easy to show then that they both result in the first functional. The proof is found in the appendix. Thus for the particular case of a stochastic system with deterministic initial conditions, the predictability functionals averaged over a representative set of predictions both give the same answer as the functional originally suggested by [53].

Finally it is conceptually interesting to view predictability as a measure of the statistical disequilibrium of a dynamical system. Asymptotically as the prediction distribution approaches the equilibrium distribution (often called the climatology) all measures of predictability proposed above approach zero. The initial conditions become less and less important and the prior distribution is approached meaning that the act of prediction has resulted in no new knowledge regarding the random variables of interest in the system. As we saw in Section 3, in a variety of important stochastic dynamical systems, this disequilibrium is measured conveniently by the divergence functional since it is non-negative and exhibits a strict monotonic decline with time. These properties thus allow it to be used to discuss the general convergence properties of stochastic systems as discussed for example in [15] Section 3.7. As we also saw in Section 3 if the system is fully resolved i.e., all fine grained variables are retained, then the divergence is conserved. Thus it is fundamentally the coarse graining of the system that is responsible for the irreversible equilibration and hence loss of predictability. In practical fluid dynamical systems this occurs in two different ways: Firstly it is obvious that all spatial scales cannot be resolved in any model and the effects of the fine scales on the resolved scales need to be included in some way. This is often done using dissipative and stochastic formulations. Secondly in the sense of probability densities, even the retained scales can usually never be completely resolved. The corresponding Fokker Planck equation is usually not exactly solvable in non-linear systems and numerical simulations face severe practical issues in high dimensions (the well known curse of dimensionality). All of this means that in nearly all realistic cases, Monte Carlo methods are the only practical choice and these mean an effective coarse graining since there is a limit to the scale at which the probability density is observable using an ensemble. We discuss these issues further below.

5.5. Applications

A number of dynamical models of increasing complexity have been studied using the functionals discussed in the previous subsection. Some of these have direct physical relevance but others are included to illustrate various dynamical effects.

In considering predictability of a dynamical system one is usually interested from a practical perspective in two things:

(1)

In a generic sense how does predictability decay in time?

(2)

How does predictability vary from one prediction to the next?

Phrased more abstractly, what is the generic character of the equilibration and secondly how does the predictability vary with respect to the initial condition label x in Equation (17)?

These issues have been studied from the perspective of information theory in the following types of models.

5.5.1. Multivariate Ornstein Uhlenbeck (OU) Stochastic Processes

Models of this type have seen application as simplified representations of general climate variability (e.g., [62]); of the El Nino climate oscillation (e.g., [63,64,65]); of mid-latitude decadal climate variability (e.g., [66]) and of mid-latitude atmospheric turbulence (e.g., [67]). In the first three cases there is a very natural (and large) timescale separation between internal atmospheric turbulent motions and variability that is usually thought of as climatic which inherits a slow timescale from the ocean. In the fourth case the separation is associated with the various parts of the atmospheric turbulence spectrum. In general the first kind of separation is more natural and of a greater magnitude. Such strong separations are of course the physical justification for the various stochastic models. Models of this type have been used to carry out skillful real time predictions most particularly in the El Nino case but also in the mid-latitude atmospheric case.

A general time dependent OU stochastic process can be shown to have the property that Gaussian initial condition random variables remain Gaussian as time evolves. This follows from the fact that Gaussian random variables subjected to a linear transformation remain Gaussian and the general solution for these processes (see Equation (4.4.83) from [15]) is a linear combination of the initial condition random variables and Gaussian Wiener random variables. In the time independent case (the usual climate model) calculation of the covariance matrix (Equation (4.4.45) from [15]) shows that it depends only on the initial condition covariance matrix and the prediction time. This implies that different initial conditions with identical covariance matrices have identical covariance matrices at other times also. Consideration of Equations (20) and (21) then shows that such a set of statistical predictions have identical predictive information but may have varying prediction utility depending on the initial condition mean vector. Additionally if the initial conditions have zero covariance i.e., are deterministic and are sampled in a representative fashion from the equilibrium distribution then the results of 5.2 show that this initial condition invariant predictive information is just the generic utility of the set of predictions. In what follows we shall refer to the third term from (21) as the Signal since it depends on the first moment of the prediction distribution while the first two terms will be referred to as the Dispersion since they depend on the covariance of the same distribution. Notice that if the signal is averaged over a representative set of deterministic initial conditions then it simply becomes minus the second term because of the discussion at the end of the last subsection. Thus minus the second term is a measure of the average contribution of the signal to prediction utility.

A simple example, considered previously by the present author as a minimal model of El Nino (see [58,65]), serves to illustrate the kinds of behavior encountered. Here a stochastically forced damped oscillator is modeled with two variables. In Ito form

$$\begin{array}{ccc}\hfill d\mathit{x}& =& -A\mathit{x}dt+\mathit{u}dW\hfill \\ \hfill A& \equiv & \left(\begin{array}{cc}0& 1\\ \beta & \gamma \end{array}\right)\mathit{u}=\left(\begin{array}{c}0\\ F\end{array}\right)\hfill \\ \hfill \gamma & \equiv & \frac{2}{\tau }\beta \equiv \left(\frac{4{\pi }^2}{T^2}+\frac{1}{{\tau }^2}\right)\hfill \end{array}$$

where τ is the oscillation decay time and T is the period. Asymptotically the covariance matrix of this system can be shown to be diagonal so the basis we are using is a principal component basis. Figure 2, which is reproduced from [58], shows the prediction utility of a representative sample of predictions with deterministic initial conditions with time being the vertical coordinate and sample member the horizontal coordinate. The relaxation process is evident for all predictions and one can show analytically that it is controlled by τ. Additionally it is clear that variations in prediction utility are a significant fraction of overall utility and that the character of this variation tends to persist strongly throughout any particular prediction. In other words predictions tend to have high/low utility at all times. We refer to this effect as predictability durability. Recall that the variation in utility is driven entirely in this example by the signal. Calculation of this term shows it to be

$$Signal={\left(\frac{{\overline{x}}_1}{{\sigma }_1^2}\right)}^2+{\left(\frac{{\overline{x}}_2}{{\sigma }_2^2}\right)}^2$$

where the equilibrium variances are ${\sigma }_i^2$. Thus the signal is simply the rescaled $L_2$ norm squared of the prediction means (called anomalies in the climate context). In this model it appears that it takes a significant amount of time for the stochastic forcing to erode this signal which is the cause of the durability.

The simple model above was generalized in the first reference above to the time dependent case on the basis that one would expect the instability of the system to follow certain cyclic features such as the seasons. The basic qualitative results shown above however remain unaltered and in particular the variation of prediction utility was still controlled mainly by the signal even when rather large variations in the decay time τ throughout the time cycle were assumed.

5.5.2. Low Order Chaotic Systems

Such systems were originally proposed as very simple analogs for atmospheric turbulence (e.g., [68,69]). They have been extensively studied from the viewpoint of atmospheric predictability and were the original motivation for the now extensive mathematical field of chaotic dynamics. Here, unlike the previous case, there is no great separation of timescales and the growth of small perturbations occurs through non-linear interaction of the degrees of freedom rather than from random forcing from fast neglected modes. These systems are often characterized by an invariant measure that is fractal in character. This implies that the dimension of the space used to calculate probability densities is non-integral which means some care needs to be taken in defining appropriate entropic functionals. As noted earlier the non-linearity of the system but also the fractal equilibrium strange attractor mean that generally the best practical approach is Monte Carlo sampling with an appropriate coarse graining of state space. One can for example define the entropy of the equilibrium strange attractor as

$$\begin{array}{ccc}\hfill E& =& \underset{M\to \infty }{lim}\underset{r\to 0}{lim}E(M,r)\hfill \\ \hfill E(M,r)& \equiv & -\frac{1}{M}\sum _{i=1}^{M}ln\left(\frac{N_i\left(r\right)}{M{r}^d}\right)\hfill \\ & =& -\frac{1}{M}\sum _{i=1}^{M}ln\left(\frac{N_i\left(r\right)}{M}\right)+dlnr\equiv S(M,r)+dlnr\hfill \end{array}$$

(22)

where $N_i\left(r\right)$ is the number of attractor sample members within a radius r of the i’th sample member; d is the so-called fractional information dimension of the attractor and M is the number of sample points. Careful comparison with (1) and (2) shows this agrees with the usual definition up to a constant which multiplies $r^d$ to give the volume element in a d dimensional space. The information dimension can be estimated by considering a sequence of decreasing r and calculating $S(M,r)$ (see [70] Section 7.9). If one considers the calculation of the divergence of samples drawn from particular subregions of the attractor then a simpler expression is relevant which does not involve the information dimension:

$$\begin{array}{ccc}\hfill D& =& \underset{M\to \infty }{lim}\underset{r\to 0}{lim}D(M,r)\hfill \\ \hfill D(M,r)& \equiv & \frac{1}{M}\sum _{i=1}^{M}ln\left(\frac{N_i^1\left(r\right)}{N_i^2\left(r\right)}\right)\hfill \end{array}$$

(23)

where $N_i^1\left(r\right)$ is the number of the first sample members within r of the i’th first sample member while $N_i^2\left(r\right)$ is the number of second sample members within the same radius of the same i’th first sample member. Naturally only a particular coarse graining $D(M,r)$ is usually practically available since M is finite.

The present author (see [58]) considered the behavior of coarse grained prediction utility (23) in the classical Lorenz chaotic system [68]. Here we also compare that with the corresponding coarse grained predictive information derived from (22). A representative set of initial conditions with means drawn from equilibrium strange attractor were considered. For simplicity a (homogeneous) Gaussian distribution of two dimensions tangential to the attractor was considered with a standard deviation approximately three orders of magnitude smaller than the attractor dimensions. Thus the initial condition uncertainty was far less then the equilibrium or climatological uncertainty. The evolution of the divergence is shown in Figure 3 for a representative set of initial conditions (format is the same as Figure 2). This is reproduced from [58]. We observe again a characteristic relaxation toward equilibrium. The temporal decline in divergence is almost always monotonic although as noted in the previous subsection, this is not compulsory for the coarse grained measure used.

In stark contrast to the stochastic models considered previously, variations in the prediction utility from one set of initial conditions to another is typically strongly related to variations in predictive information. This is shown for a large sample of initial conditions in Figure 4a which shows the relationship at around halfway through the relaxation displayed in Figure 3. It is worth noting that the prediction distributions although initially chosen to be Gaussian rapidly lose this property and indeed by the time of Figure 4a it can be shown that although the variations in the predictive information correlates well with the prediction utility, variations in the Gaussian predictive information of (20) show very little correlation. This suggests that it is the higher order cumulant contributions to the entropy which are driving variations in utility. As in the stochastic case, the predictions show considerable durability. This is illustrated in Figure 4b which shows the correlation of the utility of early and late predictions.

5.5.3. Idealized Models of Turbulence

These systems attempt to simulate the realistic spectrum of turbulence with between ${10}^2$ through ${10}^4$ degrees of freedom and in an environment which is often homogeneous. They represent a bridge between the models of the previous subsections and realistic models of both atmosphere and ocean. The nature of the turbulence exhibited means that there is a large range of timescales present and there are large non-linear transfers of energy between modes often in the form of a scale cascade. Thus both the effects of the previous two types of models are typically present.

Calculation of information theoretic functionals for models of this dimension involves essential difficulties. As we saw in the simple chaotic model above, the most straightforward method involves a Monte Carlo method known as a prediction ensemble and some choice of coarse graining to estimate probability densities via a partitioning of state space. When we confront models with significantly higher dimensions however this problem becomes very acute since the required number of partitions increases with the power of the dimension. In order to obtain a statistically reliable estimate of density for each partition we require that the ensemble size be at least the same order as the number of partitions chosen. It is fairly clear that only approximations to the functionals will be possible in such a situation.

Two approaches have been taken to this approximation problem. The first assumes that the ensemble is sufficiently large so that low order statistical moments can be reliably calculated and then implicitly discards sample information regarding higher order moments as not known. One then uses the moment information as a hard constraint on the hypothetical distribution and derives that distribution using a maximum entropy hypothesis by deciding that no knowledge is available regarding the higher order moments. In the case that only second order or less moments are available this implies, of course, that the hypothetical distribution is Gaussian. Typically however some higher order moments can also be reliably estimated from prediction ensembles so a more general methodology is desirable. Under the appropriate circumstances, it is possible to solve for more general distributions using a convex optimization methodology. This was originally suggested by [71] and was developed extensively with the present application in mind by [72,73,74,75,76].

In the latter references moments up to order four were typically retained and so generalized skewness and kurtosis statistics were assumed to be reliably estimated from prediction ensembles. An interesting feature of these maximum entropy methods is that as more moments are retained the divergence increases (see [72]). Thus the Gaussian divergence is bounded above by that derived from the more general distributions exhibiting kurtosis and skewness effects. This is analogous with the coarse to fine graining hierarchy effect noted in the final paragraph of Section 2. One feature of many models of turbulence makes this moment maximum entropy approximation approach attractive: It is frequently observed that the low order marginal distributions are quasi Gaussian and so can usually be very well approximated by retaining only the first four moments. Such a situation contrasts strongly with that noted above in simple chaotic systems where highly non-Gaussian behavior is usual.

Another approach to the approximation issue (developed by the present author in [77]) is to accept that only marginal distributions up to a certain low order are definable from a prediction ensemble. One then calculates the average divergence with respect to all possible marginal distributions and calls the result the marginal divergence. As in the maximum entropy case, the marginal divergence of a certain order defined in this way strictly bounds from above the marginal divergences of lower order. This again is consistent with greater retention of information as higher order marginal distributions are used to discriminate between distributions. Note also that at the top of this chain sits the full divergence.

An interesting aspect of these approximations is the role of the prediction ensemble in defining either the moment constraints or the marginal distributions. Some reflection on the matter shows that in fact these objects are subject to sample error and this becomes larger as higher order approximations are considered. This suggests that in the first case the constraints assumed in deriving maximum entropy distributions should not be imposed as hard constraints but instead be weak constraints as they should reflect the presence of the sample error. This rather subtle issue has received little attention to date in the literature in the context of maximum entropy (see however [74]). See [77] for an information theoretic analysis for the simpler marginal distribution estimation case.

Four different turbulence models of varying physical complexity and dimension have been analyzed in detail using the above approximation methods:

(1)

A truncated version of Burgers equation detailed in [78]. This system is a one dimensional inviscid turbulence model with a set of conserved variables which enable a conventional Gibbs Gaussian equilibrium distribution. Predictability issues relevant to the present discussion can be found in [73,79].

(2)

The one dimensional Lorenz 1996 model of mid-latitude atmospheric turbulence detailed in [40]. This model exhibits a variety of different behaviors depending on the parameters chosen. For some settings strongly regular wave-like behavior is observed while for others a more irregular pattern occurs with some resemblance to atmospheric turbulence. The most unstable linear modes of the system tend to have most weight at a particular wavenumber which is consistent with the observed atmosphere. Predictability issues from the present perspective can be found in [73].

(3)

Two dimensional barotropic quasi-geostrophic turbulence detailed in, for example, [80]. Barotropic models refer to systems with no vertical degrees of freedom. Quasi-geostrophic models are rotating fluids which filter out fast waves (both gravity and sound) in order to focus on low frequency variability. The barotropic versions aim at simulating very low frequency variability and exclude by design variability associated with mid-latitude storms/eddies which have a typically shorter timescale. Predictability issues are discussed in [75] in the context of a global model with an inhomogeneous background state.

(4)

Baroclinic quasi-geostrophic turbulence as discussed in, for example, [81]. This system is similar to the last except simple vertical structure is incorporated which allows the fluid to draw energy from a vertical mean shear (baroclinic instability). These systems have therefore a representation of higher frequency mid-latitude storms/eddies. This latter variability is sometimes considered to be approximately a stochastic forcing of the low frequency barotropic variability although undoubtedly the turbulence spectrum is more complicated. Predictability issues from the present perspective were discussed in [82] for a model with an homogeneous background shear. They are discussed from a more conventional predictability viewpoint in [83].

Generally speaking for all systems the predictability time scale (as measured by the prediction utility) is related to the equilibration time scale of the turbulence being considered. Variations in prediction utility with initial conditions are typically similar to those seen in stochastic or simple chaotic systems. The origin of these variations is a rather complex subject and the answers obtained also vary significantly according to the system examined. In the first subsection above we considered the case where both the prediction and equilibrium distributions were Gaussian. This meant that an analytical formula involving the lowest two moments was possible and we found it convenient conceptually to separate terms into those dependent on the first and second moments of prediction distribution which we referred to as the signal and dispersion. It is important to emphasize however that this separation does not reflect a decomposition of the utility into the predictive information (the uncertainty reduction of the prediction) and a remainder term. Indeed as we saw above, the second term of Equation (21) which is included in the dispersion, is not at all related to uncertainty reduction but reflects differences in the two distributions unrelated to entropy/uncertainty or their means.

When we move to the maximum entropy approximation for prediction utility and consider moments of higher order it is possible to generalize this separation of prediction moment contributions to the divergence into those dependent on the prediction mean and those dependent on higher moments [84]. A comprehensive and rigorous description of this generalized signal/dispersion decomposition is to be found in [73]. The contribution of these two terms was examined in this reference for the first two turbulence systems listed above. The dominant contributor to overall utility variations was found to depend on the parameters chosen for the model as well as the prediction time chosen. There is also within this reference a discussion of when moments beyond the first two are important to a maximum entropy approximation retaining the first four moments.

When the higher order geostrophic turbulence models are considered (models 3 and 4 in the list above), it is usual that a considerable fraction of the equilibrium variability occurs due only to a relatively small number of modes. Predictability studies have usually focused on the equilibration behavior of such modes.

In the study of a barotropic turbulence model [75] the focus was on the first four principal components of the equilibrium distribution. The maximum entropy framework retaining four moments was used and the variability of prediction utility in terms of the generalized signal and dispersion examined with the conclusion that dispersion was overwhelmingly responsible for variations.

A baroclinic turbulence model was studied in [82]. The equilibrium turbulence spectrum in such models is well known to be dominated energetically by the large scale vertically uniform (barotropic) modes (see [25]). The predictability study therefore restricted attention to barotropic modes and the gravest four horizontal Fourier modes. The marginal distributions for both prediction and equilibrium distributions were observed to be quite close to Gaussian so only the first two moments were retained for calculation of entropic functionals by the maximum entropy method. In addition the marginal distributions with respect to these four dimensions was also calculated directly by a rather coarse partitioning. In contrast to the barotropic model above, variations in prediction utility were overwhelmingly related to variations in the (Gaussian) signal. This strong difference in predictability behavior has yet to be explained. It may be due to the importance in the latter case of the stochastic like forcing of the low frequency barotropic modes by the higher frequency baroclinic modes. These are of course absent in the barotropic turbulence case. Thus it may be that barotropic model is more like the low order chaos case while the baroclinic model is more like the simple stochastic case. This is speculation however and deeper investigation is required.

5.5.4. Realistic Global Primitive Equation Models

Such models are simplified versions of models used in real time to predict the daily weather. In contrast to most models of the previous subsection, the background for the turbulence is very inhomogeneous as it depends on geographical features such as topography and the jet stream. The latter is driven by the temperature gradient between the equator and pole and so peaks strongly in mid-latitudes. Compared with the quasi-geostrophic turbulence systems, more modes are generally required to explain the equilibrium variance: Of order around 100 compared with less than 10 for the homogeneous baroclinic quasi-geostrophic case. This dynamical situation has tended to limit the application of the approximation approaches discussed earlier. The only study conducted to date with such models from an information theoretic perspective was by the present author in [85]. This used marginal divergences on the dominant 100 equilibrium modes as well as the Gaussian maximum entropy approach discussed above. Practical prediction ensemble sizes restrict marginal divergences to order four or less. Generally speaking, marginal distributions of dynamical variables tend to be close to Gaussian however this issue remains to be systematically investigated. In the present context it was found that as initial conditions were varied, there was a correlation of order 0.8 between Gaussian estimates of divergence and those derived from marginal divergences. The latter tended to have a higher correlation among themselves.

In this system, the equilibration time scale implied by the above estimates of prediction utility is a strong inverse function of the amount of mean vertical shear present in the atmosphere. When this shear is strong as it is during winter it is between 1 and 2 months while for the summer weak shear case it is around 3 months. These estimates of predictability are significantly longer than those derived from practical weather prediction where two weeks is generally accepted to be the current predictability limit. This difference may be due to model error of practical predictions and/or due to simplifications present in the model studied in [85] which excludes important random processes such as moist convection which can provide a significant mechanism to reduce predictability.

The variation in prediction utility with respect to a representative set of initial conditions was also studied and the results found were somewhat similar to those noted in the baroclinic turbulence model discussed earlier. Overall divergence fluctuations (either Gaussian or marginal) were most strongly related to signal rather than dispersion variations. This effect was more pronounced for short range rather than long range predictions. The dispersion was however also related to utility but at a lower level than signal. Like both the stochastic and simple chaotic models discussed earlier, prediction utility variations tended to show durability i.e., high utility predictions tended to remain high throughout a prediction and vice versa.

5.5.5. Other Recent Applications

There have been interesting applications of the functionals discussed here to various climate prediction problems. This has been aided by the fact that to an even greater extent than the turbulence models discussed above, the marginal distributions observed tend to be very close to Gaussian. Indeed it is almost as if a central limit theorem operates in many complex geophysical models in such a way that many distributions of dynamical variables of interest tend to be approximately normal.

The El Nino coupled ocean/atmosphere system was considered in depth with realistic models in [86] and conclusions were broadly in line with the simple stochastic models considered earlier. In general the signal rather than dispersion played a larger role in explaining predictability variation.

The problem of global warming projection was considered from the current perspective by [87]. They used the divergence between the equilibrium distribution of various models and the present observed climate/equilibrium distribution as a measure of the veracity of a particular model. They then found a relationship between this veracity and the amplitude of projected global average warming over the next century. Models with higher veracity tended to project greater warming. In related work, global warming model errors were analyzed using an array of empirical information theoretic tools in [88]. This allowed the authors to identify global warming patterns of highest sensitivity.

Part of the issue of long term climate projection concerns its separation into that due to ocean initial conditions (natural decadal variability) and that due to changes in forcing due to the radiative effects of greenhouse gases. This issue was addressed using divergence measures of predictability in [89] where it was found that the predictability switches from the first kind to the second kind at leads of roughly 5–10 years.

The predictability of the natural decadal predictability mentioned above was analyzed in detail using divergence measures in [90]. A focus was on the behavior of the first few principal components and the authors used the Gaussian signal/dispersion decomposition to carefully analyze the nature of this predictability and the way it transfers between principal components as predictions evolve.

Finally the Gaussian signal/dispersion decomposition of divergence has been used to study the nature of the predictability of polar sea ice in a very recent unpublished study [91].

6. Conclusions and Outlook

Predictability issues in dynamical models are studied very naturally using information theoretic functionals. This follows because such variables are best considered to be random variables and entropic functionals have a very intuitive interpretation in terms of uncertainty and as measures of the differences between distributions. Adding to this attraction is the fact that these functionals exhibit invariance under general transformations of the variables describing the dynamical system and that they have deep connections with concepts from statistical physics. Indeed it is argued in this review that the utility of a prediction can be measured by the degree of statistical disequilibrium within the system. Thus the study of the predictability of a system is equivalent to the study of the nature of its statistical equilibration.

There have been several other advances in physical understanding with practical consequences which have resulted from the use of the information functionals discussed in this review:

(1)

The mechanisms responsible for variations in statistical predictability have been clarified. Traditionally it has been assumed that these are associated with variations in the instability of the initial conditions. The information theoretical analysis reveals that other mechanisms are possible. One, which has been referred to as signal here, results from variations in the “distance” with which initial conditions are from the equilibrium mean state of the system. Such excursions often take considerable time to “erode” and so are responsible for considerable practical predictability. Preliminary calculations show that this second effect may indeed be more important than the first in certain weather and climate applications. Note that this is not simply the effect of persistence of anomalies since this signal can transfer between the components of the state vector of a system. From the numerical studies conducted it appears that this second effect may be important when there is a strong separation of time scales within the system. These results are deserving of further theoretical and numerical investigation.

(2)

It has been shown that ideas from information transfer allow us to address in generality the question of improving predictions using targeted additional observations. The methods discussed above apply in the long range, non-linear regime for which previous methods are likely inapplicable.

(3)

The physical mechanism causing the decline in mid-latitude weather predictability has been studied over the full temporal range of predictions. Numerical experiments have shown the central role of vertical mean shear in controlling the equilibration/loss of predictability.

Application of the concepts considered here has often relied on the fact that the random variables under study are quasi-Gaussian since this allows application of analytical expressions. It should be emphasized however (and we saw this in Section 4) that a major attraction of information theory is the ability to go beyond this situation and consider highly non-linear and non-Gaussian distributions. As we saw in detail in Section 5, considerable progress has been made using maximum entropy methods to improve approximate calculation of functionals in high dimensional system. No doubt however more work remains in this challenging area where the curse of dimensionality always lurks.

The fundamental mechanism that drives equilibration and hence loss of predictability is the dynamical process which leads to statistical irreversibility. This, as we saw in Section 3, is intimately related to the manner in which a system is coarse grained and fundamental progress in that area will drive a deeper understanding of system predictability. Thus the expectation of the author is that further progress in the area of non-equilibrium statistical mechanics will be of great benefit to the study of the basic nature of prediction.