Properties of the Statistical Complexity Functional and Partially Deterministic HMMs

Abstract

Statistical complexity is a measure of complexity of discrete-time stationary stochastic processes, which has many applications. We investigate its more abstract properties as a non-linear function of the space of processes and show its close relation to the Knight’s prediction process. We prove lower semi-continuity, concavity, and a formula for the ergodic decomposition of statistical complexity. On the way, we show that the discrete version of the prediction process has a continuous Markov transition. We also prove that, given the past output of a partially deterministic hidden Markov model (HMM), the uncertainty of the internal state is constant over time and knowledge of the internal state gives no additional information on the future output. Using this fact, we show that the causal state distribution is the unique stationary representation on prediction space that may have finite entropy.

1. Introduction

An important task of complex systems sciences is to define “complexity”. Measures that quantify complexity are of both theoretical (e.g., [1]) and practical interest. In applications, they are widely used to identify “interesting” parts of simulations and real-world data (e.g., [2]). There exist various measures of different kinds of complexity. In particular, statistical complexity constitutes a complexity measure for stationary stochastic processes in doubly infinite discrete time and discrete state space. It was introduced by Jim Crutchfield and co-workers within a theory called computational mechanics [3,4,5]. Note that here “computational mechanics” is unrelated to computer simulations of mechanical systems. Statistical complexity is applied to a variety of real-world data, e.g., in [6]. An important, closely related concept of computational mechanics is the so-called $\epsilon$-machine. It is a particular partially deterministic HMM that encodes the mechanisms of prediction. Partially deterministic HMMs are often called deterministic stochastic automata to emphasize their close connection to a key concept of theoretical computer science, namely deterministic finite state automata [7].

In this paper, we look at more abstract features of statistical complexity as well as partially deterministic HMMs. We consider statistical complexity to be a non-linear functional from the space of Δ-valued stationary processes (Δ countable) to the set ${\overline{\mathbb{R}}}_{+}={\mathbb{R}}_{+}\cup \{\infty \}$ of non-negative extended real numbers. Here, we identify stationary processes with their law, i.e., with shift-invariant probability measures on the sequence space ${\Delta }^{\mathbb{Z}}$, and equip the space of measures with the usual weak-* topology (often called “weak topology”). Because Δ is discrete, this topology is equal to the topology of finite-dimensional convergence. In ergodic theory, Kolmogorov-Sinai entropy is studied as a function of the (invariant) measure, and the questions of continuity properties, affinity, and behaviour under ergodic decomposition arise naturally (e.g., [8]). We believe that these questions are worth considering also for complexity measures. A formula for the ergodic decomposition of excess entropy, which is another complexity measure for stochastic processes, was obtained in [9,10]. Our results presented here include the corresponding formula for statistical complexity, and this formula directly implies concavity. The most important result is lower semi-continuity of statistical complexity. We consider this a desirable property for a complexity measure, as it means that a process cannot be complex if it can be approximated by non-complex ones.

In Section 2., we define statistical complexity and show its relations to a discrete version of Frank Knight’s prediction process [11,12]. The prediction process is the measure-valued process of conditional probabilities of the future given the past. It takes values in the space $\mathcal{P}({\Delta }^{\mathbb{N}})$ of probability measures on Δ^N, called prediction space. In our formulation, statistical complexity is the marginal entropy of the prediction process. This is equivalent to the classical definition as entropy of a certain partition of the past. We only replace equivalence classes with the respective induced probabilities on the future. In this section, we also show that the discrete (and thus technically vastly simplified) version of the prediction process has a continuous Markov transition kernel (Proposition 5).

In Section 3., we investigate properties of partially deterministic HMMs. Here, we use a general notion of HMM (sometimes called edge-emitting HMM), where new internal state and output symbol are jointly determined and may have dependencies conditioned on the last internal state. Partial determinism means that this dependence is extreme in the sense that the last internal state and the output altogether uniquely determine the following internal state. We show that, if one knows the past output trajectory, the remaining uncertainty (measured by entropy) of the internal state is constant over time, although it may depend on the ergodic component (Proposition 18). Furthermore, the distribution of future output is the same for any internal state that is compatible with the past output (Corollary 20). In Section 3.3., we construct a canonical Markov kernel, such that taking any measure $\nu$ on prediction space $\mathcal{P}({\Delta }^{\mathbb{N}})$ (i.e., $\nu$ is a measure on measures) as initial distribution, we obtain a partially deterministic HMM of a process $P\in \mathcal{P}({\Delta }^{\mathbb{N}})$. This process P coincides with the measure $r(\nu )$ represented by $\nu$ in the sense of integral representation theory, and if $\nu$ is appropriately chosen, we obtain the $\epsilon$-machine of computational mechanics (or something isomorphic) as special case. Using the properties of partially deterministic HMMs, we obtain that there is no invariant representation on prediction space with finite entropy other than, possibly, the causal state distribution, which may have finite or infinite entropy (Proposition 23).

Section 4. contains our results about statistical complexity. We show that the complexity of a process is the average complexity of its ergodic components plus the entropy of the mixture (Proposition 26). As a direct consequence, statistical complexity is concave (Corollary 27) and non-continuous (even w.r.t. variational topology). But it does have a continuity property. Namely, using the results of the previous sections, we show in Theorem 32 that it is weak-* lower semi-continuous.

2. Prediction Dynamic and Statistical Complexity

For the whole article, fix a countable set Δ with at least two elements and discrete topology. We identify Δ-valued stochastic processes $X_{\mathbb{Z}}:={(X_k)}_{k\in \mathbb{Z}}$, defined on some probability space $(\Omega ,\mathfrak{A},\mathbb{P})$, with their respective laws $P:=\mathbb{P}\circ X_{\mathbb{Z}}^{-1}\in \mathcal{P}({\Delta }^{\mathbb{Z}})$. Here, $\mathcal{P}$ denotes the set of probability measures. If $X_{\mathbb{Z}}$ is stationary, P is in the set ${\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ of shift-invariant probability measures. Let ${\xi }_k:{\Delta }^{\mathbb{Z}}\to \Delta$ be the canonical projections. Then ${\zeta }^{\mathbb{Z}}$ is a process on (${\Delta }^{\mathbb{Z}},\mathcal{B}({\Delta }^{\mathbb{Z}}),P$) with the same distribution as $X_{\mathbb{Z}}$. Here, $\mathcal{B}$ denotes the Borel σ-algebra. We often decompose the time set $\mathbb{Z}$ into the “future” $\mathbb{N}$ and the “past” $\mathbb{Z}\backslash \mathbb{N}=-{\mathbb{N}}_0$, where ${\mathbb{N}}_0=\mathbb{N}\cup \left\{0\right\}$. For simplicity of notation, we denote the canonical projections on ${\Delta }^{\mathbb{N}}$ with the same symbols, ζ_k, as the projections on ${\Delta }^{\mathbb{Z}}$. If not stated otherwise, product spaces are equipped with product and spaces of probability measures are equipped with weak-$\ast$-topology. We use the arrow $\stackrel{*}{\rightharpoonup }$ to denote weak-$\ast$ convergence.

2.1. Discrete Version of Knight’s Prediction Process

For every measurable stochastic process with time set ${\mathbb{R}}_{+}$ on some Lusin space, Frank Knight defines the corresponding prediction process as a process of conditional probabilities of the future given the past. This theory originated in [11] and was further developed in [12,13,14]. The most important properties of the prediction process are that its paths are right continuous with left limits (cadlag), it has the strong Markov property and determines the original process. The continuity of the time set and the generality of the state space lead to a lot of technical difficulties. In our simpler, discrete setting, these difficulties mostly disappear, and useful properties of the prediction process, such as having cadlag paths, become meaningless. A new aspect, however, is added by considering infinite pasts of stationary processes via the time-set $\mathbb{Z}$. The marginal distribution (unique because of stationarity) of the prediction process is an important characteristic, which is used to define statistical complexity. For this subsection, fix a stationary process $X_{\mathbb{Z}}$ with distribution $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$.

We use the following notation concerning Markov kernels and conditional probabilities. If K is a kernel from Ω to a measurable space M, we consider K as measurable function from Ω to $\mathcal{P}(M)$ and write $K(\omega ;A):=K(\omega )(A)$ for the probability of a measurable set A w.r.t. the measure $K(\omega )$. Given random variables $X,Y$ on Ω, we write $K=\mathbb{P}(X\mid Y)$ if K is the conditional probability kernel of X given Y, i.e., $K(\omega ;A)=\mathbb{P}(\{X\in A\}|Y)(\omega )$.

Definition 1. 

Let $Z_{\mathbb{Z}}=Z_{\mathbb{Z}}^P$ be the $\mathcal{P}({\Delta }^{\mathbb{N}})$-valued stochastic process of conditional probabilities defined by $Z_k:=P({\xi }_{[k+1,\infty [}\mid {\xi }_{]-\infty ,k]})$ for $k\in \mathbb{Z}$. Then $Z_{\mathbb{Z}}$ is called prediction process of $X_{\mathbb{Z}}$. $\mathcal{P}({\Delta }^{\mathbb{N}})$ is called prediction space.

It is evident that the Markov property of the prediction process in continuous time also holds in discrete time. Nevertheless, we give a proof, because it is elementary in our discrete setting. The corresponding transition kernel works as follows. Assume the prediction process is in state $z\in \mathcal{P}({\Delta }^{\mathbb{N}})$. The transition kernel maps z to a measure on measures, namely $P(Z_1\mid Z_0=z)\in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$. Note that z is a state of the prediction process but at the same time a probability measure. Thus it makes sense to consider the conditional probability given ${\xi }_1=d$ w.r.t. the measure z. It is intuitively plausible that the next state will be one of those conditional probabilities with d distributed according to the marginal of z. The resulting measure has to be shifted by one as time proceeds. With $\varsigma :{\Delta }^{\mathbb{N}}\to {\Delta }^{\mathbb{N}}$, we denote the left shift.

Proposition 2. 

For $z\in \mathcal{P}({\Delta }^{\mathbb{N}})$, let ${\varphi }_z:{\Delta }^{\mathbb{N}}\to \mathcal{P}({\Delta }^{\mathbb{N}})$, ${\varphi }_z(\omega ):=z({\varsigma }^{-1}(·)\mid {\xi }_1)(\omega )$. The prediction process $Z_{\mathbb{Z}}$ is a stationary Markov process. The kernel $S:\mathcal{P}({\Delta }^{\mathbb{N}})\to \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$ with $S(z)=z\circ {\varphi }_z^{-1}$, i.e.

$$S(z)(B):=S(z;B):=z(\{{\varphi }_z\in B\}),z\in \mathcal{P}({\Delta }^{\mathbb{N}}),B\in \mathfrak{B}(\mathcal{P}({\Delta }^{\mathbb{N}})),$$

satisfies $P(Z_k\mid Z_{k-1})=S\circ Z_{k-1}$ a.s. In other words, S is the transition kernel of the prediction process.

Proof. 

Stationarity is obvious from stationarity of $X_{\mathbb{Z}}$. We obtain a.s.

$$\begin{array}{cc}\hfill S(Z_0;B)& =Z_0(\left\{Z_0({\varsigma }^{-1}(·)\mid {\xi }_1)\in B\right\})=P\left(\left\{P({\xi }_{[2,\infty [}\mid {\xi }_{]-\infty ,1]})\in B\right\}|{\xi }_{-{\mathbb{N}}_0}\right)\hfill \\ & =P(\{Z_1\in B\}|{\xi }_{-{\mathbb{N}}_0}).\hfill \end{array}$$

In particular, $P(\{Z_1\in B\}|{\xi }_{-{\mathbb{N}}_0})$ is $\sigma (Z_0)$-measurable (modulo P) and together with $\sigma (Z_0)\subseteq \sigma ({\xi }_{-{\mathbb{N}}_0})$ we obtain

$$P(\{Z_1\in B\}|Z_0)=P(\{Z_1\in B\}|{\xi }_{-{\mathbb{N}}_0})=S(Z_0;B),$$

(1)

as claimed. We still have to verify the Markov property. But because the $\sigma$-algebra induced by $Z_{-{\mathbb{N}}_0}$ is nested between those induced by $Z_0$ and ${\xi }_{-{\mathbb{N}}_0}$, i.e. $\sigma (Z_0)\subseteq \sigma (Z_{-{\mathbb{N}}_0})\subseteq \sigma ({\xi }_{-{\mathbb{N}}_0})$, we obtain the Markov property from the first equality in (1). □

Definition 3. 

We call the Markov transition S of the prediction process prediction dynamic.

Note that although the prediction process $Z_{\mathbb{Z}}$ obviously depends on P, the prediction space $\mathcal{P}({\Delta }^{\mathbb{N}})$ and the prediction dynamic S do not. In the case of general Lusin state space, it is non-trivial to prove the existence of the regular versions of conditional probability such that ${\varphi }_z(\omega )$ is jointly measurable in $(z,\omega )$ (see [9]). For countable Δ, however, we can obtain essential continuity in an elementary way. This enables us to prove continuity of the prediction danamic.

Lemma 4. 

Let $z,z_n\in \mathcal{P}({ʔ}^{\mathbb{N}})$ and $z_n\stackrel{*}{\rightharpoonup }z$. There is a clopen (i.e. closed and open) set ${\Omega }_z\subseteq {ʔ}^{\mathbb{N}}$ with $z({\Omega }_z)=1$ such that ${\varphi }_{z_n}\stackrel{*}{\rightharpoonup }{\varphi }_z$, uniformly on compact subsets of ${\Omega }_z$.

Proof. 

Let $A_{\omega }:={{\xi }_1}^{-1}({\xi }_1(\omega ))$ and ${\Omega }_z:=\left\{\omega \in {\Delta }^{\mathbb{N}}|z(A_{\omega })>0\right\}$. Because Δ is discrete and countable, Ω_z id clopen with z(Ω_z) = 1. Uniform convergence on compacta is equivalent to ${\varphi }_{z_n}({\omega }_n)\stackrel{*}{\rightharpoonup }{\varphi }_z(\omega )$ whenever ω_n → ω in Ω_z. For sufficiently large n, ζ₁ (ω_n) = ζ₁ (ω) and because ς⁻¹ maps cylinder sets to cylinder sets, ${\varphi }_{z_n}({\omega }_n)=\frac{z_n(A_{\omega }\cap {\varsigma }^{-1}(°))}{z_n(A_{\omega })}\stackrel{*}{\rightharpoonup }{\varphi }_z(\omega )$.

Proposition 5. 

The prediction dynamic S is continuous.

Proof. 

Let $z_n,z\in \mathcal{P}({\Delta }^{\mathbb{N}})$ with $z_n\stackrel{*}{\rightharpoonup }z$ and ${\Omega }_z$ as in Lemma 4. We have to show

$$\int g\mathrm{d}S(z_n)=\int g\circ {\varphi }_{z_n}\mathrm{d}{z}_n{\displaystyle \stackrel{n\to \infty }{⟶}}\int g\circ {\varphi }_z\mathrm{d}z=\int g\mathrm{d}S(z)$$

(2)

for any bounded continuous g. According to Prokhorov’s theorem, the sequence ${(z_n)}_{n\in \mathbb{N}}$ is uniformly tight and we can restrict the integrations to compact subsets. Because ${lim}_{n\to \infty }{z}_n({\Omega }_z)=z({\Omega }_z)=1$, we can restrict to compact subsets of ${\Omega }_z$. There, the convergence of ${\varphi }_{z_n}$ is uniform, thus (2) holds. □

2.2. Statistical Complexity

In integral representation theory, a measure $\nu \in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$ represents the measure $z\in \mathcal{P}({\Delta }^{\mathbb{N}})$ if

$$z=r(\nu ):={\int }_{\mathcal{P}({\Delta }^{\mathbb{N}})}^{}{\mathrm{id}}_{\mathcal{P}({\Delta }^{\mathbb{N}})}\mathrm{d}\nu ,$$

(3)

where $r:\mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))\to \mathcal{P}({\Delta }^{\mathbb{N}})$ is called resolvent or barycentre map (see [15]) and $\mathrm{id}$ is the identity map. Here, measure valued integrals are Gel’fand integrals. That is, $\mu =\int K\mathrm{d}\nu$ for some kernel K means $\int f\mathrm{d}\mu =\int \int f\mathrm{d}K(·)\mathrm{d}\nu$ for all continuous, real-valued f or, equivalently, $\mu (B)=\int K(·;B)\mathrm{d}\nu$ for all measurable sets B. $z=r(\nu )$ means that z is a mixture (convex combination) of other processes, and the mixture is described by $\nu$. A trivial representation for z is given by ${\delta }_z$, the Dirac measure in z. The measure $\nu$ is called S-invariant if $\nu S=\nu$, where $\nu S:=\int S\mathrm{d}\nu$. In other words, it is S-invariant if the iteration with the prediction dynamic S does not change it. We see in the following lemma that general iteration with S shifts the represented measure, i.e., $\nu S$ represents $z\circ {\varsigma }^{-1}$.

Lemma 6. 

${\displaystyle r(\nu S)=r(\nu )\circ {\varsigma }^{-1}.}$ In particular, S-invariant $\nu$ represent stationary processes.

Proof. 

Because $r(\nu S)=\int \int {\mathrm{id}}_{\mathcal{P}({\Delta }^{\mathbb{N}})}\mathrm{d}S\mathrm{d}\nu$, it is sufficient to consider Dirac measures ${\delta }_z$, $z\in \mathcal{P}({\Delta }^{\mathbb{N}})$ (the general claim follows by integration over $\nu$). For Dirac measures we have

$$r({\delta }_{z}S)=\int {\mathrm{id}}_{\mathcal{P}({\Delta }^{\mathbb{N}})}\mathrm{d}S(z)=\int {\varphi }_z\mathrm{d}z=\int z({\varsigma }^{-1}(·)|{\xi }_1)\mathrm{d}z=z\circ {\varsigma }^{-1}$$

 □

If $\nu$ is S-invariant, we also say that $\nu$ represents the stationary extension of $r(\nu )$ to ${\Delta }^{\mathbb{Z}}$. The marginal of the prediction process is an important such representation, which we call causal state distribution because of its close relation to the causal states of computational mechanics.

Definition 7. 

For $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$, the causal state distribution ${\mu }_{\mathfrak{C}}(P)$ is the marginal distribution of the prediction process, i.e., ${\mu }_{\mathfrak{C}}(P):=P\circ Z_0^{-1}\in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$.

The causal state distribution of P is an S-invariant representation of P.

Lemma 8. 

Let $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$. Then ${\mu }_{\mathfrak{C}}(P)$ is S-invariant and represents P.

Proof. 

From Proposition 2 we know that $P(Z_1\mid Z_0)=S\circ Z_0$ and $Z_{\mathbb{Z}}$ is stationary. Thus

$$\int S\mathrm{d}{\mu }_{\mathfrak{C}}(P)=\int S\circ Z_0\mathrm{d}P=\int P(Z_1\mid Z_0)\mathrm{d}P=P\circ Z_1^{-1}={\mu }_{\mathfrak{C}}(P)$$

Furthermore, ${\mu }_{\mathfrak{C}}(P)$ represents P because we have

$$r({\mu }_{\mathfrak{C}}(P))=\int Z_0\mathrm{d}P=\int P({\xi }_{\mathbb{N}}\mid {\xi }_{-{\mathbb{N}}_0})\mathrm{d}P=P\circ {{\xi }_{\mathbb{N}}}^{-1}$$

 □

Remark. 

The definitions in computational mechanics are slightly different. There, one works with equivalence classes of past trajectories (called causal states) instead of probability distributions on future trajectories. Because past trajectories $x,y\in {\Delta }^{-{\mathbb{N}}_0}$ are identified if $P({\xi }_{\mathbb{N}}\mid {\xi }_{-{\mathbb{N}}_0}=x)=P({\xi }_{\mathbb{N}}\mid {\xi }_{-{\mathbb{N}}_0}=y)$, the two approaches are equivalent. The advantage of working on prediction space $\mathcal{P}({\Delta }^{\mathbb{N}})$ is that it has a natural topology and the prediction processes of all Δ-valued stochastic processes are described in a unified manner on the same space with the same transition kernel.

Example 9. 

${\mu }_{\mathfrak{C}}$ is not continuous. Let P be a non-deterministic i.i.d. (independent, identically distributed) process. Obviously, the causal state distribution of an i.i.d. process is the Dirac measure ${\delta }_{P_{\mathbb{N}}}$ in its restriction $P_{\mathbb{N}}:=P\circ {{\xi }_{\mathbb{N}}}^{-1}$ to positive time. According to [16], periodic measures are dense in the stationary measures and we find an approximating sequence $P_n\stackrel{*}{\rightharpoonup }P$ of periodic measures $P_n$. But the past of a periodic process determines its future. Thus its causal state distribution is supported by the set of Dirac measures on ${\Delta }^{\mathbb{N}}$. Because the set of Dirac measures is closed in $\mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$, the topological supports $supp{\mu }_{\mathfrak{C}}(P_n)$ are disjoint from the support $supp{\mu }_{\mathfrak{C}}(P)=\left\{P_{\mathbb{N}}\right\}$. Consequently, ${\mu }_{\mathfrak{C}}(P_n)$ cannot converge to ${\mu }_{\mathfrak{C}}(P)$. $◊$

With statistical complexity, we measure complexity of a process P by the “diversity” of its expected futures, given observed pasts (i.e., of ${\mu }_{\mathfrak{C}}(P)$). The Shannon entropy $H(\mu )$ is used as the measure of “diversity” of a probability measure $\mu$. With $\phi (x):=-xlog(x)$, it is defined as

$$H(\mu ):=sup\{\sum _{i=1}^n\phi (\mu (B_i))|n\in \mathbb{N},B_i\text{disjoint,}\text{measurable}\}$$

(4)

Definition 10. 

For $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$, the quantity $C_{\mathfrak{C}}(P):=H({\mu }_{\mathfrak{C}}(P))\in {\overline{R}}_{+}$ is called statistical complexity of P.

Note that if the probability space is sufficiently regular (e.g., separable, metrisable), $H(\mu )$ can only be finite if $\mu$ is supported by a countable set A. In this case

$$H(\mu )=\sum _{a\in A}\phi (\mu (\left\{a\right\}))$$

Probably, lower semi-continuity of this entropy functional is well-known. We give a proof in the appendix.

Lemma 11. 

Let M be a separable, metrisable space. Then the entropy $H:\mathcal{P}(M)\to {\overline{R}}_{+}$ is weak-* lower semi-continuous.

3. Partially Deterministic HMMs

The probability measures on prediction space induce hidden Markov models (HMMs) with an additional partial determinism property, and it turns out to be helpful to investigate such HMMs. In Section 3.1., we define HMMs and introduce the notation we need for the further discussion. In Section 3.2., we define the partial determinism property and obtain our results about the HMMs satisfying this property. In Section 3.3., we show how measures on prediction space induce partially deterministic HMMs and apply the results from Section 3.2. to prove that the causal state distribution is the only invariant representation on prediction space that can have finite entropy.

3.1. HMMs

We use the term HMM in a wide sense, meaning a pair $(T,\mu )$, where $\mu$ is an initial probability measure on some Polish space M of internal states and T is a Markov kernel from M to Δ × M. The HMM generates on $(\Omega ,\mathfrak{A},P)$ a Δ-valued output process $X_{\mathbb{N}}$ and a (coupled) M-valued internal process $W_{{\mathbb{N}}_0}$, such that W₀ is μ-distributed and the joint process is Markovian with

$$P(\{X_k\in D,W_k\in B\}|X_{k-1},W_{k-1})=T(W_{k-1};D\times B)\text{a.s.}$$

We call (T,μ) an HMM of $z\in \mathcal{P}({\Delta }^{\mathbb{N}})$ if $z=\mathbb{P}\circ X_{\mathbb{N}}^{-1}$ if $\mu (B)=\int T(·;\Delta \times B)d\mu$, we say that the HMM is invariant and extend the generated processes to stationary processes $X_{\mathbb{Z}}$ and $W_{\mathbb{Z}}$. We need some further notation.

Definition 12. 

Let $(T,\mu )$ be an HMM, $m\in M$, $d\in \Delta$, and $\nu \in \mathcal{P}(M)$.

a)

The output kernel K: $M\to \mathcal{P}(\Delta )$ is defined by $K(m):=K_m:=T(m;·\times M)\in \mathcal{P}(\Delta )$. We also use the notations $\overline{K_d}(m):=K_m(d):=K_m(\{d\})$ and $K_{\nu }:=\int Kd\nu$.

b)

The internal operators $L_d:\mathcal{P}(M)\to \mathcal{P}(M)\bigcup \left\{0\right\}$ are defined as follows. $L_d(\nu )=0$ and

$$L_d(\nu )(B):=\frac{\int T(·;\{d\}\times B)\text{d}\nu }{K_{\nu }(d)}\text{ otherwise}.$$

Remark. 

a)

$K_m$ is the distribution of the next output symbol when the internal state is m, i.e. $K_m=P(X_1\mid W_0=m)$ a.s. Further, $K_{\mu }$ is the law of $X_1$.

b)

The internal operator $L_d$ describes the update of knowledge of the internal state when the symbol $d\in \Delta$ is observed. For Dirac measures, we obtain

$$L_d({\delta }_m)=P(W_1\mid W_0=m,X_1=d)\text{a.s.}$$

Be warned that L_d is not induced by a kernel in the following sense. There is no kernel $l_d:M\to \mathcal{P}(M)$ such that $L_d(\nu )=\int l_d\text{d}\nu$. To see this, note that $L_d(\nu )\ne \int L_d⃘\iota \text{d}\nu$ for $\iota (m)={\zeta }_m$, because L_d(ν) is normalised outside the integral as opposed to an individual normalisation of the L_d(ζ_m) inside the integral on the right-hand side.

It directly follows from the definition of $(X_{\mathbb{N}},W_{{\mathbb{N}}_0})$ by a Markov kernel that the conditional probability, given that the internal state is m, is obtained by starting the HMM in m. In other words, it is generated by the HMM $(T,{\delta }_m)$. Similarly, the conditional probability given an observed symbol $X_1=d$ is obtained by starting the HMM in the updated initial distribution $L_d(\mu )$. We formulate these observations in the following lemma and give a formal proof in the appendix.

Lemma 13. 

Let $(T,\mu )$ be an HMM with internal and output processes $W_{{\mathbb{N}}_0}$, $X_{\mathbb{N}}$ as above. Then a.s. $(T,{\delta }_{W_0(\omega )})$ is an HMM of $P(X_{\mathbb{N}}\mid W_0)(\omega )$, and $(T,L_{X_1(\omega )}(\mu ))$ is an HMM of $P(X_{[2,\infty [}\mid X_1)(\omega )$.

Definition 14. 

(processes $Y_Z$ and $H_Z$) Given an invariant HMM, let $Y_Z$ be the $\mathcal{P}(M)$-valued stochastic process of expectations over internal states, given by $Y_k:=P(W_k\mid X_{]-\infty ,k]})$. Let $H_Z$ be the process of entropies of the random measures $Y_k$, i.e., $H_k(\omega ):=H(Y_k(\omega ))$, where entropy H is defined by (4).

Remark. 

$Y_k$ describes the current knowledge of the internal state, given the past. $H_k$ is the entropy of the value of $Y_k$ and measures “how uncertain” the knowledge of the internal state is. It is important to bear in mind that this is different from the entropy of the random variable $Y_k$. To avoid confusion, we always write $H^P(X)$ when referring to the entropy of a random variable X defined on a probability space with measure P.

The following lemma justifies the idea of the internal operator $L_d$ being an update of knowledge of the internal state. Furthermore, it enables us to condition on $Y_0$ instead of $X_{-{\mathbb{N}}_0}$. The conditional probability of the internal state given the past, $Y_0$, contains as much information about $X_1$ (and in fact $X_{\mathbb{N}}$, but we do not need that here) as the past $X_{-{\mathbb{N}}_0}$ does.

Lemma 15. 

a)

${\displaystyle Y_1(\omega )=L_{X_1(\omega )}(Y_0(\omega ))}$ a.s.

b)

${\displaystyle P(\{X_1=d\}|Y_0)(\omega )=P(\{X_1=d\}|X_{-{\mathbb{N}}_0})(\omega )=K_{Y_0(\omega )}(d)}$ a.s.

Proof. 

Conditional independence of $(X_1,W_1)$ and $X_{-{\mathbb{N}}_0}$ given $W_0$ implies that a.s. $P(X_1,W_1\mid W_0)=P(X_1,W_1\mid W_0,X_{-{\mathbb{N}}_0})$ and thus

$$\int T\mathrm{d}{Y}_0=\int P(X_1,W_1\mid W_0)\mathrm{d}P(·\mid X_{-{\mathbb{N}}_0})=P(X_1,W_1\mid X_{-{\mathbb{N}}_0})$$

(5)

a)

Let $d=X_1(\omega )$ and for $B\in \mathfrak{B}(M)$ set $F_B:=\{X_1=d,W_1\in B\}$. We obtain a.s.

$$L_d(Y_0)(B)\stackrel{(5)}{=}\frac{P(F_B\mid X_{-{\mathbb{N}}_0})}{P(F_M\mid X_{-{\mathbb{N}}_0})}\stackrel{\text{(}d=X_1(\omega ))}{=}P(\{W_1\in B\}|X_{-{\mathbb{N}}_0},X_1)=Y_1(·)(B)$$

b)

The second equality follows directly from (5). The first follows because, due to the second equality, $P(\{X_1=d\}|X_{-{\mathbb{N}}_0})$ is $\sigma (Y_0)$-measurable modulo P. □

The previous lemma enables us to prove that $Y_Z$ is Markovian and compute its transition kernel. We already know that $L_d(\nu )$ is the updated expectation of the internal state when it was previously $\nu$ and is now observed d. Thus it is not surprising that the conditional probability of $Y_k$ given $Y_{k-1}=\nu$ is a convex combination of Dirac measures in $L_d(\nu )$ for different d (note that $Y_k$ is a measure-valued random variable, thus its conditional probability distribution is indeed a distribution on distributions). The mixture is given by the output kernel K, more precisely by $K_{\nu }$.

Lemma 16. 

For an invariant HMM, $Y_Z$ and $H_Z$ are stationary. $Y_Z$ is a Markov process with transition kernel

$$P(Y_{k+1}\mid Y_k=\nu )=\sum _{d\in \Delta }{K}_{\nu }(d)·{\delta }_{L_d(\nu )}\in \mathcal{P}(\mathcal{P}(M))\forall \nu \in \mathcal{P}(M).$$

Proof. 

Stationarity is obvious. For ${\nu }_0,\dots {\nu }_k\in \mathcal{P}(M)$ and $\nu :{\nu }_k$ we obtain

$$\begin{array}{ccc}\hfill P(Y_{k+1}\mid Y_{[0,k]}={\nu }_{[0,k]})& \stackrel{(\text{lem}.15\text{a})}{=}& P(L_{X_{k+1}(·)}(\nu )|Y_{[0,k]}={\nu }_{[0,k]})\hfill \\ & =& \sum _{d\in \Delta }\mathbb{P}(\{X_{k+1}=d\}|Y_{[0,k]}={\nu }_{[0,k]})·{\delta }_{L_d(\nu )}.\hfill \end{array}$$

Because $\sigma (Y_{[0,k]})$ is nested between $\sigma (Y_k)$ and $\sigma (X_{]-\infty ,k]})$, Lemma 15 b) implies that $P(\{X_{k+1}=d\}|Y_{[0,k]}={\nu }_{[0,k]})=K_{{\nu }_k}=K_{\nu }$ and hence the claim.

Partial Determinism

If the transition T of an HMM is deterministic, i.e., if the internal state determines the next state and output (and thus the whole future) uniquely, the HMM is called (completely) deterministic. In a deterministic HMM, all randomness is due to the initial distribution. This is a very strong property, and a weaker partial determinism property is useful. In a partially deterministic HMM, the output symbol is determined randomly, but the new internal state is a function $f(m,d)$ of the last internal state m and the new output symbol d. If the internal space M is finite, such HMMs are stochastic versions of deterministic finite state automata (DFAs), an important concept of theoretical computer science (see [7]). The function f directly corresponds to the transition function of the DFA, but the start state is replaced by the initial distribution and the HMM assigns probabilities to the outputs via the output kernel K. A difference in interpretation is that the symbols from Δ are considered input of the DFA and output of HMMs. To emphasise their close connection to DFAs, partially deterministic HMMs are often called deterministic stochastic automata, although they are not completely deterministic.

Definition 17. 

An HMM $(T,\mu )$ is called partially deterministic if there is a measurable function $f:M\times \Delta \to M$, called transition function, such that $T(m)=K_m\otimes {\delta }_{f_m(·)}$ for all $m\in M$, i.e.,

$$T(m;D\times B)=K_m(D\cap f_m^{-1}(B))\forall m\in M,D\subseteq \Delta B\in \mathfrak{B}(M)$$

where $f_m(d):={\widehat{f}}_d(m):=f(m,d)$ and $\mathfrak{B}(M)$ is the Borel $\sigma$-algebra on M.

Remark. 

For partially deterministic HMMs we obtain

$$L_d(\nu )(B)=\frac{1}{K_{\nu }(d)}{\int }_{{\widehat{f}}_d^{-1}(B)}{\widehat{K}}_d\mathrm{d}\nu \text{and}{L}_d({\delta }_m)={\delta }_{f_m(d)}$$

(6)

The second equation implies that $W_k=f_{W_{k-1}}(X_k)$ a.s., justifying the name transition function for f.

The following proposition is crucial for understanding partially deterministic representations. It states that, given the past output, the uncertainty $H_k=H(Y_k)$ about the internal state is constant over time and the next output symbol is independent of the internal state. The proof is along the following lines. If we know the internal state at one point in time, we can maintain knowledge of the internal state due to partial determinism. More generally, the uncertainty $H_k$ of the internal state cannot decrease on average and thus is a supermartingale. But because it is also stationary, the trajectories have to be constant. If two possible internal states would lead to different probabilities for the next output symbol, we could increase our knowledge of the internal state by observing the next output. But because of partial determinism, this would also decrease the uncertainty of the following internal state, in contradiction to the constant trajectories of $H_Z$.

Proposition 18. 

Let $(T,\mu )$ be a partially deterministic, invariant HMM with $H(\mu )<\infty$. Then $H_Z$ has a.s. constant trajectories, i.e., $H_k=H_0$ a.s. Furthermore, the restriction $K{\upharpoonright }_{supp(Y_0)}$ of the output kernel K to the support $supp(Y_0)\subseteq M$ of the random measure $Y_0$ is a.s. a constant kernel, i.e.,

$$K_m=K_{\widehat{m}}\forall m,\widehat{m}\in supp(Y_0(\omega ))\text{a.s.}$$

(7)

Proof. 

We show that $H_Z$ is a supermartingale to use the following well-known property.

Lemma. 

Every stationary supermartingale has a.s. constant trajectories.

Because $H(\mu )<\infty$, we may assume w.l.o.g. that M is countable. Note that $\phi (x)=-xlog(x)$ satisfies $\phi (\sum x_i)\le \sum \phi (x_i)$. We obtain

$$H(L_d(\nu ))\stackrel{(6)}{=}\sum _{\widehat{m}\in M}\phi \left(\sum _{m\in {\widehat{f}}_d^{-1}(\widehat{m})}\nu (m)\frac{K_m(d)}{K_{\nu }(d)}\right)\le \sum _{m\in {\widehat{f}}_d^{-1}(M)=M}\phi \left(\nu (m)\frac{K_m(d)}{K_{\nu }(d)}\right).$$

We use the filtration ${\mathcal{F}}_k:=\sigma (Y_{]-\infty ,k]})$. Markovianity of $Y_Z$ yields $\mathrm{E}(H_{k+1}\mid {\mathcal{F}}_k)=\mathrm{E}(H_{k+1}\mid Y_k)$.

$$\begin{array}{ccc}\hfill \mathrm{E}(H_{k+1}|Y_k=\nu )& \stackrel{(\text{lem}.16)}{=}& \sum _{d\in \Delta }{K}_{\nu }(d)·H(L_d(\nu ))\le -{\sum }_{d,m}\nu (m)K_m(d)·log\left(\nu (m)\frac{K_m(d)}{K_{\nu }(d)}\right)\hfill \\ & =& H^P(W_k\mid X_{k+1},Y_k=\nu )\le H^P(W_k\mid Y_k=\nu )=H(\nu )\hfill \end{array}$$

where the secong equality holds because $P(\{W_k=m,X_{k+1}=d\}|Y_k=\nu )=\nu (m)K_m(d)$ and $P(\{X_{k+1}=d\}|Y_k=\nu )=K_{\nu }(d)$, This $H_Z$ is a supermartingale w.r.t. ${({\mathcal{F}}_k)}_{k\in Z}$ and has a.s. constant trajectories. In particular, inequality (8) is actually an equality. Because $H(\mu )<\infty$ and $\mu =\int Y_k\mathrm{d}P$, the entropy of $Y_k(\omega )$ is a.s. finite. Thus, $H^P(W_k\mid X_{k+1},Y_k=\nu )=H^P(W_k\mid Y_k=\nu )$ implies that W_k and X_K+1 are independent given $Y_k=\nu$, i.e. $K{\upharpoonright }_{supp(\nu )}$ is constant.  □

Note that the finite-entropy assumption is indeed necessary for the second statement of Proposition 18. For example, the shift defines a deterministic HMM that does not (in general) satisfy (7).

Example 19. (shift HMM) 

The shift HMM is defined as follows. The internal state consists of the whole trajectory, $M:={\Delta }^{\mathbb{Z}}$. $T=T^{\varsigma }$ outputs the symbol at position one and shifts the sequence to the left. More formally with $m={(m_k)}_{k\in Z}\in M$ and $\varsigma (m)={(m_{k+1})}_{k\in Z}$ we have

$$T^{\varsigma }(m)={\delta }_{m_1}\otimes {\delta }_{\varsigma (m)}={\delta }_{(m_1,\varsigma (m))}$$

If $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$, it is obvious that $(T^{\varsigma },P)$ is an invariant, deterministic (in particular partially deterministic) HMM of P. Here, P is the law of both $X_Z$ and $W_0$; in fact even $X_Z=W_0$. We claim that, generically, $(T^{\varsigma },P)$ does not satisfy (7) (and of course the internal state entropy $H(P)$ is infinite). Indeed, $K_m={\delta }_{m_1}$ and thus $K_m=K_{\widehat{m}}$ implies $m_1={\widehat{m}}_1$. Because $Y_0(\omega )=P(X_Z\mid X_{-{\mathbb{N}}_0})(\omega )$, (7) implies that $X_{-{\mathbb{N}}_0}$ determines $X_1$ uniquely, which is generically not true. The analogously defined one-sided shift on $M={\Delta }^{\mathbb{N}}$ also does not satisfy (7). Note that, because future trajectories are equivalent to internal states, the associated process $Y_Z$ is essentially the prediction process in the sense that $Y_k=Z_k\circ X_Z$. $◊$

Proposition 18 tells us that the next output symbol of a partially deterministic HMM is conditionally independent of the internal state, given the past output. But even more is true. The whole future output is conditionally independent of the internal state. Thus, if we know the past, the internal state provides no additional information useful for the prediction of the future output.

Corollary 20. 

Let $(T,\mu )$ be partially deterministic, invariant, and $H(\mu )<\infty$. Then

$$P(X_{\mathbb{N}}\mid W_0=m)=P(X_{\mathbb{N}}\mid W_0=\widehat{m})\forall m,\widehat{m}\in supp(Y_0)\text{a.s.}$$

Proof. 

According to Proposition 18, $P(X_1\mid W_0=·)=K$ is constant on $supp(Y_0)$. To obtain the statement for $X_{[1,n]}$, we consider the n-tuple HMM defined as follows. The output space is Δⁿ, the internal space is M, whereas the output and internal processes ${\widehat{X}}_Z$ and ${\widehat{W}}_Z$ are given by ${\widehat{X}}_k=X_{[(k-1)n+1,kn]}$ and ${\widehat{W}}_k=W_{nk}$. This is achieved by the HMM $(\widehat{T},\mu )$ with $\widehat{T}:M\to \mathcal{P}({\Delta }^n\times M)$, $\widehat{T}(m)=P(X_{[1,n]},W_n\mid W_0=m)$. The HMM is obviously partially deterministic with transition function $f_{d_n}\circ \cdots \circ f_{d_1}$ and invariant. Thus Proposition 18 implies that $P(X_{[1,n]}\mid W_0=·)=P({\widehat{X}}_1\mid {\widehat{W}}_0=·)$ is constant on $supp({\widehat{Y}}_0)$. Because we can couple the processes such that ${\widehat{Y}}_0=Y_0$, the claim follows.  □

3.3. Representations on Prediction Space

We can interpret any probability measure $\mu$ on prediction space $\mathcal{P}({\Delta }^{\mathbb{N}})$ as initial distribution of an HMM. The “internal state update” of the corresponding transition $T^{\mathfrak{C}}$ follows the same rule as the prediction dynamic S, described by the conditional probability given the last observation. The difference is that now we include output symbols from Δ. We want to construct the HMM in such a way that if it is started in the internal state $z\in P({\Delta }^{\mathbb{N}}$, its output process is distributed according to z (which is also a measure on the future). Thus, the distribution of the next output d has to be equal to the marginal of z. The next internal state has to be the conditional z-probability of the future given ${\zeta }_1=d$. Recall that ${\varphi }_z(\omega )=({\varsigma }^{-1}(·)|{\zeta }_1)(\omega )$.

Definition 21. 

We define the Markov kernel $T^{\mathfrak{C}}$ from $\mathcal{P}({\Delta }^{\mathbb{N}})$ to $\Delta \times \mathcal{P}({\Delta }^{\mathbb{N}})$ by

$$T^{\mathfrak{C}}(z;D\times B):=z(\{{\xi }_1\in D,{\varphi }_z\in B\}),z\in \mathcal{P}({\Delta }^{\mathbb{N}}),D\subseteq \Delta B\in \mathfrak{B}(\mathcal{P}({\Delta }^{\mathbb{N}})).$$

Note that $T^{\mathfrak{C}}(z;\Delta \times B)=S(z;B)$, i.e., marginalising $T^{\mathfrak{C}}(z)$ to the internal component yields the prediction dynamic. Thus, if $\mu ={\mu }_{\mathfrak{C}}(P)$ is the causal state distribution (Definition 7) of some process $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$, then the internal state process of the induced HMM $(T^{\mathfrak{C}},\mu )$ coincides with the prediction process $Z_Z$ of P. From the following lemma we conclude that the output process $X_Z$ is, as expected, distributed according to P. More generally, if $\mu \in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$ represents a process $z\in \mathcal{P}({\Delta }^{\mathbb{N}})$ in the sense of integral representation theory as a mixture of other processes, it also induces an HMM of z, namely $(T^{\mathfrak{C}},\mu )$. Recall that r is the resolvent, defined in (3), and associates the represented process to $\mu$.

Lemma 22. 

Let $\mu \in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$. Then $(T^{\mathfrak{C}},\mu )$ is a partially deterministic HMM of $r(\mu )$. In particular, $(T^{\mathfrak{C}},{\mu }_{\mathfrak{C}}(P))$ is an invariant HMM of $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$.

Proof. 

Partial determinism follows directly from the definition of $T^{\mathfrak{C}}$. We have $K_z=z\circ {{\xi }_1}^{-1}$ and the transition function f is given by $f_z\circ {\xi }_1:={\varphi }_z$. It is well defined due to the $\sigma ({\xi }_1)$-measurability of ${\varphi }_z$ and obviously $T^{\mathfrak{C}}(z;D\times B)=K_z(D\cap f_z^{-1}(B))$. We assume w.l.o.g. that $\mu$ is a Dirac measure (the general claim follows by integration over $\mu$). Thus let $\mu ={\delta }_z$ with $z=r(\mu )$. Recall that, according to Lemma 13, $(T^{\mathfrak{C}},T_d^{\mathfrak{C}}({\delta }_z))$ is an HMM of the conditional probability of ${\xi }_{[2,\infty [}$ given that ${\xi }_1=d$ (w.r.t. the output process of $(T^{\mathfrak{C}},{\delta }_z)$). Using $T^{\mathfrak{C}}(z;\left\{d\right\}\times \mathcal{P}({\Delta }^{\mathbb{N}}))=z(\{{\xi }_1=d\})$ and

$$r(T_d^{\mathfrak{C}}({\delta }_z))\stackrel{(6)}{=}r({\delta }_{f_z(d)})=f_z(d)=z({\varsigma }^{-1}(·)\mid {\xi }_1=d),$$

the claim follows by induction.  □

Remark. ($\epsilon$-machine). 

$(T^{\mathfrak{C}},{\mu }_{\mathfrak{C}}(P))$ corresponds to the so-called $\epsilon$-machine of computational mechanics. It is in some sense a minimal predictive model but in general not the minimal HMM of P (see [17]).

Given a process $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$, there are (usually) many invariant representations on prediction space (i.e., S-invariant $\nu \in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$ with $r(\nu )=P_{\mathbb{N}}$). The next proposition shows that the causal state distribution of P is distinguished among them as the only one that can have finite entropy.

Proposition 23. 

Let $\nu \in \mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$ be S-invariant, and $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ the measure it represents. If $\nu \ne {\mu }_{\mathfrak{C}}(P)$, then $H(\nu )=\infty$.

Proof. 

Let $H(\nu )<\infty$. According to Lemma 22, $(T^{\mathfrak{C}},\nu )$ is an invariant HMM of P and satisfies the conditions of Corollary 20. Let $W_Z$ be the corresponding $M=\mathcal{P}({\Delta }^{\mathbb{N}})$-valued internal process. For a.e. fixed $\omega$, Lemma 13 tells us that $(T^{\mathfrak{C}},{\delta }_{W_0(\omega )})$ is an HMM of $P(X_{\mathbb{N}}\mid W_0)(\omega )$, but it is also an HMM of $r({\delta }_{W_0(\omega )})=W_0(\omega )$ due to Lemma 22. Thus, $P(X_{\mathbb{N}}\mid W_0)=W_0$ and

$$z=P(X_{\mathbb{N}}\mid W_0=z)\stackrel{(\text{cor}.20)}{=}P(X_{\mathbb{N}}\mid W_0=\widehat{z})=\widehat{z}\forall z,\widehat{z}\in supp(Y_0(\omega ))$$

This means $\left|supp(Y_0)\right|=1$, i.e. $Y_0(\omega )$ is a Dirac measure. Thus $Y_0=P(W_0\mid X_{-{\mathbb{N}}_0})={\delta }_{W_0}$ a.s. and

$$Z_0\circ X_Z=P(X_{\mathbb{N}}\mid X_{-{\mathbb{N}}_0})=\int P(X_{\mathbb{N}}\mid W_0=·)\mathrm{d}{Y}_0=P(X_{\mathbb{N}}\mid W_0)=W_0\text{a.s.}$$

Because $W_0$ is $\nu$-distributed and ${\mu }_{\mathfrak{C}}(P)$ is the law of $Z_0$, we obtain $\nu ={\mu }_{\mathfrak{C}}(P)$.

We conclude this section with two examples of representations on prediction space. They are extreme cases. The first one, ${\nu }_1$, is maximally concentrated, namely ${\nu }_1$ is the Dirac measure in (the future part of) the process we want to represent. Thus it has no uncertainty in itself, but the (unique) process in its support can be arbitrary. The second example, ${\nu }_2$, is supported by maximally concentrated processes, i.e. by Dirac measures on $({\Delta }^{\mathbb{N}})$, but the mixture ${\nu }_2$ is as diverse as the original process. The HMM corresponding to ${\nu }_2$ is equivalent to the one-sided shift (Example 19).

Example 24. 

Let $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$, $P_{\mathbb{N}}=P\circ X_{\mathbb{N}}^{-1}$ and $\nu ={\delta }_{P_{\mathbb{N}}}$. Then $\nu$ is a representation of $P_{\mathbb{N}}$ with $H(\nu )=0$. This is no contradiction to Proposition 23 because $\nu$ is not S-invariant (if P is not i.i.d.) $◊$

Example 25. (lifted shift). 

Let $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ and $\nu =P_{\mathbb{N}}\circ {\iota }^{-1}$, where $\iota :{\Delta }^{\mathbb{N}}\to \mathcal{P}({\Delta }^{\mathbb{N}})$, $\iota (x)={\delta }_x$ is the embedding as Dirac measures. $\nu$ is an S-invariant representations of P and $(T^{\mathfrak{C}},\nu )$ is equivalent to the one-sided shift. The only difference is that trajectories $x\in {\Delta }^{\mathbb{N}}$ are replaced by the corresponding Dirac measures ${\delta }_x\in \mathcal{P}({\Delta }^{\mathbb{N}})$. In other words, $\iota$ is an isomorphism. This is no contradiction to Proposition 23 because $H(\nu )=\infty$ (if P is not concentrated on countably many trajectories). $◊$

4. Properties of the Statistical Complexity Functional

Recall that the statistical complexity $C_{\mathfrak{C}}(P)$ (Definition 10) of a process $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ is defined as the entropy $H({\mu }_{\mathfrak{C}}(P))$ of its causal state distribution. In this section, we investigate $C_{\mathfrak{C}}$ as a functional on the space of processes. First, we consider the problem of ergodic decomposition. With ergodic decomposition of P, we denote a probability measure $\nu$ on the ergodic measures ${\mathcal{P}}_{\mathsf{e}}({\Delta }^{\mathbb{Z}})\subseteq {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ that satisfies

$$P=r(\nu )={\int }_{{\mathcal{P}}_{\mathsf{e}}({\Delta }^{\mathbb{Z}})}^{}{\mathrm{id}}_{{\mathcal{P}}_{\mathsf{e}}(\mathbb{Z})}\mathrm{d}\nu$$

Such a measure $\nu$ always exists and is uniquely determined by P. In [9,10], Łukasz Dębowski investigated another complexity measure, excess entropy, and gave a formula for its ergodic decomposition. Here, we obtain the corresponding result for statistical complexity. It is the average complexity of the ergodic components plus the entropy of the mixture.

Proposition 26. 

(ergodic decomposition). Let $\nu \in \mathcal{P}({\mathcal{P}}_{\mathsf{e}}({\Delta }^{\mathbb{Z}}))$ be the ergodic decomposition of $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$. Then

$$C_{\mathfrak{C}}(P)=\int C_{\mathfrak{C}}\mathrm{d}\nu +H(\nu )$$

Proof. 

First note that ${\mu }_{\mathfrak{C}}(P_1)$ and ${\mu }_{\mathfrak{C}}(P_2)$ are singular for distinct ergodic $P_1,P_2\in {\mathcal{P}}_{\mathsf{e}}({\Delta }^{\mathbb{Z}})$. Indeed, there exist disjoint $A_1,A_2\in \sigma ({\xi }_{-{\mathbb{N}}_0})$ and disjoint $B_1,B_2\in \sigma ({\xi }_{\mathbb{N}})$ s.t. $P_k(A_k)=1$ and $P_k(B_k\mid {\xi }_{-{\mathbb{N}}_0}){\upharpoonright }_{A_k}\equiv 1$. Consequently, if $\nu$ is not supported by a countable set, ${\mu }_{\mathfrak{C}}(P)$ cannot be supported by a countable set and $C_{\mathfrak{C}}(P)=H(\nu )=\infty$. Thus assume $\nu ={\sum }_{k\in \mathbb{N}}{\nu }_k{\delta }_{P_k}$ for some ${\nu }_k\ge 0$ and distinct $P_k\in {\mathcal{P}}_{\mathsf{e}}({\Delta }^{\mathbb{Z}})$. Then there are disjoint $A_k\in \sigma ({\xi }_{-{\mathbb{N}}_0})$ s.t. $P_k(A_k)=1$. We claim

$$P(·\mid {\xi }_{-{\mathbb{N}}_0})=\sum _{k\in \mathbb{N}}{1}_{A_k}{P}_k(·\mid {\xi }_{-{\mathbb{N}}_0})P\text{-a.s.}$$

Indeed, the $\sigma ({\xi }_{-{\mathbb{N}}_0})$-measurability is clear, and for $A\in \sigma ({\xi }_{-{\mathbb{N}}_0})$, $F\in \mathfrak{B}({\Delta }^{\mathbb{Z}})$ we have

$$\begin{array}{ccc}\hfill {\int }_A^{}\sum _{k\in \mathbb{N}}{1}_{A_k}{P}_k(F\mid {\xi }_{-{\mathbb{N}}_0})\mathrm{d}P& =& \sum _{j\in \mathbb{N}}{\nu }_j{\int }_A^{}\sum _{k\in \mathbb{N}}{1}_{A_k}{P}_k(F\mid {\xi }_{-{\mathbb{N}}_0})\mathrm{d}{P}_j\hfill \\ & \stackrel{\text{(}{P}_j(A_j)=1)}{=}& \sum _j{\nu }_j{\int }_{A\cap A_j}^{}{P}_j(F\mid {\xi }_{-{\mathbb{N}}_0})\mathrm{d}{P}_j\hfill \\ & =& \sum _j{\nu }_j{P}_j(F\cap A\cap A_j)=P(F\cap A)\hfill \end{array}$$

As $P(A_k)={\nu }_k$, it follows that ${\mu }_{\mathfrak{C}}(P)={\sum }_k{\nu }_k{\mu }_{\mathfrak{C}}(P_k)$. Mutual singularity of the ${\mu }_{\mathfrak{C}}(P_k)$ implies

$$C_{\mathfrak{C}}(P)=H\left(\sum _k{\nu }_k{\mu }_{\mathfrak{C}}(P_k)\right)=\sum _k{\nu }_{k}H({\mu }_{\mathfrak{C}}(P_k))+H(\nu )$$

 □

Several corollaries follow directly from this proposition. The set ${\mathcal{P}}_{\mathfrak{C}}:=C_{\mathfrak{C}}^{-1}(\mathbb{R})$ of stationary processes with finite statistical complexity is convex, $C_{\mathfrak{C}}$ is concave but not continuous, and the set ${\mathcal{P}}_{\infty }:={\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})\setminus {\mathcal{P}}_{\mathfrak{C}}$ of processes with infinite statistical complexity is dense.

Corollary 27 

(concavity) ${\mathcal{P}}_{\mathfrak{C}}$ is a convex set and $C_{\mathfrak{C}}$ is concave. Moreover, for all $\nu \in \mathcal{P}({\Delta }^{\mathbb{N}})$, ${\nu }_k:=\nu (k)$ and $P_k\in {\mathcal{P}}_{\mathsf{inv}}(\mathbb{Z})$

$$\sum _{k\in \mathbb{N}}{\nu }_k{C}_{\mathfrak{C}}(P_k)\le C_{\mathfrak{C}}\left(\sum _{k\in \mathbb{N}}{\nu }_k{P}_k\right)\le \sum _{k\in \mathbb{N}}{\nu }_k{C}_{\mathfrak{C}}(P_k)+H(\nu )$$

Proof. 

Use ergodic decomposition of the $P_k$ and Proposition 26.  □

Corollary 28. 

(non-continuity). $C_{\mathfrak{C}}{\upharpoonright }_{{\mathcal{P}}_{\mathfrak{C}}}$ is not continuous in any $P\in {\mathcal{P}}_{\mathfrak{C}}$ w.r.t. variational topology, let alone w.r.t. weak-* topology.

Proof. 

Let $Q_n\in {\mathcal{P}}_{\mathfrak{C}}$ with ${lim}_{n\to \infty }\frac{1}{n}{C}_{\mathfrak{C}}(Q_n)\to \infty$ and $P_n:=\frac{n-1}{n}P+\frac{1}{n}{Q}_n$. Then $P_n\to P$ in variational topology, but $C_{\mathfrak{C}}(P_n)\ge \frac{1}{n}{C}_{\mathfrak{C}}(Q_n)\to \infty$ by Corollary 27.

Corollary 29. 

${\mathcal{P}}_{\infty }$ is dense in ${\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ w.r.t. variational- and a fortiori w.r.t. weak-*-topology.

Proof. 

$P,Q\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ with $C_{\mathfrak{C}}(Q)=\infty$. Then ${\mathcal{P}}_{\infty }\ni \frac{n-1}{n}P+\frac{1}{n}Q\to P$  □

We give a simple example of a situation where statistical complexity is not continuous.

Example 30. 

(non-continuity). Let $Q_p$ be the Bernoulli process on $\Delta =\{0,1\}$ with parameter $0<p<1$, i.e. $Q_p({\xi }_1=1)=p$. Consider the process of throwing a coin which is either slightly biased to 0 or 1, each with probability $\frac{1}{2}$, i.e. $P_{\epsilon }=\frac{1}{2}{Q}_{\frac{1}{2}+\epsilon }+\frac{1}{2}{Q}_{\frac{1}{2}-\epsilon }$ with $0<\epsilon <\frac{1}{2}$. Then $P_{\epsilon }\stackrel{*}{\rightharpoonup }{P}_0=Q_{\frac{1}{2}}$ for $\epsilon \to 0$, but $C_{\mathfrak{C}}(P_{\epsilon })=log(2)$ for $\epsilon >0$ and $C_{\mathfrak{C}}(P_0)=0$. $◊$

The proof of our most important result about statistical complexity, namely its lower semi-continuity, makes use of the propositions given in Section 2.1. and Section 3.. It also uses a compactness argument. To this end we need, in the case of infinite Δ, a lemma guaranteeing that ${\mu }_C$ preserves relative compactness.

Lemma 31. 

Let $\mathcal{M}\subseteq {\mathcal{P}}_{\mathsf{inv}}(\mathbb{Z})$ be relatively compact. Then ${\mu }_{\mathfrak{C}}(\mathcal{M}):=\left\{{\mu }_{\mathfrak{C}}(P)|P\in \mathcal{M}\right\}$ is relatively compact in $\mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))$.

Proof. 

Using Prokhorov’s theorem, we have to show that ${\mu }_{\mathfrak{C}}(\mathcal{M})$ is tight provided that $\mathcal{M}$ is tight. Let $\epsilon >0$ and $K_n\subseteq {\Delta }^{\mathbb{Z}}$ compact with $P(K_n)\ge 1-\epsilon \frac{2^{-n}}{n}$ for all $P\in \mathcal{M}$. We define $K_n^{\prime }:={\xi }_{\mathbb{N}}(K_n)$, $\tilde{K}:=\left\{z\in \mathcal{P}({\Delta }^{\mathbb{N}})|z(K_n^{\prime })\ge 1-\frac{1}{n}\forall n\in \mathbb{N}\right\}$ and $f_n:=P(\{{\xi }_{\mathbb{N}}\in K_n^{\prime }\}|{\xi }_{-{\mathbb{N}}_0})$. For $P\in \mathcal{M}$

$$\int f_n\mathrm{d}P\ge \int P(K_n\mid {\xi }_{-{\mathbb{N}}_0})\mathrm{d}P=P(K_n)\ge 1-\epsilon \frac{2^{-n}}{n}.$$

We obtain $P({\bigcup }_n\{f_n<1-\frac{1}{n}\})\le {\sum }_{n}n(1-\int f_n\mathrm{d}P)\le \sum \epsilon 2^{-n}=\epsilon$ and consequently ${\mu }_{\mathfrak{C}}(P)(\tilde{K})=P(\{Z_0\in \tilde{K}\})=P({\bigcap }_n\{f_n\ge 1-\frac{1}{n}\})\ge 1-\epsilon$ for all $P\in \mathcal{M}$. We still have to show compactness of $\tilde{K}$. It is closed because $z_k\stackrel{*}{\rightharpoonup }z$ implies $z(K_n^{\prime })\ge {lim\; sup}_k{z}_k(K_n^{\prime })$ due to closedness of $K_n^{\prime }$. It is tight by definition because the $K_n^{\prime }$ are compact. Therefore, $\tilde{K}$ is compact.  □

Theorem 32. 

(lower semi-continuity). The statistical complexity functional, $C_{\mathfrak{C}}:{\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})\to {\overline{\mathbb{R}}}_{+}$, is weak-* lower semi-continuous.

Proof. 

Let $P_n,P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ with $P_n\stackrel{*}{\rightharpoonup }P$. Every subsequence of $({\mu }_{\mathfrak{C}}(P_n){)}_{n\in \mathbb{N}}$ has an accumulation point (a.p.), according to Lemma 31. Consequently,

$$\underset{n\to \infty }{lim\; inf}{C}_{\mathfrak{C}}(P_n)=\underset{n\to \infty }{lim\; inf}H({\mu }_{\mathfrak{C}}(P_n))\stackrel{\text{(}H\text{lsc)}}{\ge }inf\left\{H(\nu )|\nu \text{a.p.}\text{of}{({\mu }_{\mathfrak{C}}(P_n))}_{n\in \mathbb{N}}\right\}.$$

Every ${\mu }_{\mathfrak{C}}(P_n)$ is S-invariant. According to Proposition 5, S is continuous and thus every a.p. $\nu$ of $({\mu }_{\mathfrak{C}}(P_n){)}_{n\in \mathbb{N}}$ is also S-invariant. The resolvent $r:\mathcal{P}(\mathcal{P}({\Delta }^{\mathbb{N}}))\to \mathcal{P}({\Delta }^{\mathbb{N}})$ is continuous (see [15]), and thus $\nu$ represents P. Therefore, according to Proposition 23, $H(\nu )\ge C_{\mathfrak{C}}(P)$. In total we obtain

$$\underset{n\to \infty }{lim\; inf}{C}_{\mathfrak{C}}(P_n)\ge C_{\mathfrak{C}}(P).$$

 □

We argue that, from a theoretical point of view, every complexity measure should be lower semi-continuous. While it is not counter-intuitive that it is possible to approximate a simple system by unnecessarily complex ones (and hence the complexity is not continuous), it would be strange to consider a process complex if there is an approximating sequence with (uniformly) simple processes. Therefore, an axiomatic characterisation of complexity measures (although, of course, we are far from having such a characterisation) should include lower semi-continuity. There are also slightly more practical reasons why semi-continuity is a nice property.

In a model selection task, for instance, it might be desirable to impose some upper bound $a\in {\mathbb{R}}_{+}$ on the complexity of considered processes (e.g. to avoid overfitting). An important consequence of lower semi-continuity is that the set $C_{\mathfrak{C}}^{-1}([0,a])=\left\{P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})|C_{\mathfrak{C}}(P)\le a\right\}$ of processes with complexity bounded by a is closed. This makes the complexity constraint technically easier. Consider any complete metric on ${\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ compatible with weak-* (or any stronger) topology (e.g. Prokhorov, Kantorovich-Rubinshtein or variational metric). Then due to the closedness, for every $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$ with arbitrary complexity, there is a (not necessarily unique) closest “sufficiently simple” process $P_a$ with complexity not exceeding a. Another consequence is that the set of processes with infinite complexity is generic in the following sense.

Corollary 33. 

${\mathcal{P}}_{\infty }$ contains a dense ${\mathcal{G}}_{\delta }$-set.

Proof. 

Because all $C_{\mathfrak{C}}^{-1}([0,n])$ are closed, ${\mathcal{P}}_{\infty }$ is a ${\mathcal{G}}_{\delta }$-set. It is dense according to Corollary 29.

Example 34. 

Consider the experiment of first choosing a random coin with success probability p uniformly in $[0,1]$ and then generating an i.i.d. sequence with this coin. More precisely, let $Q_p$ be the Bernoulli process with parameter p on $\Delta =\{0,1\}$ and $P=\int Q_p\mathrm{d}p$. Then P has infinite statistical complexity according to Proposition 26. We might approximate P by $P_n\stackrel{*}{\rightharpoonup }P$ (e.g. with ergodic $P_n$). Then Theorem 32 implies that the complexity of $P_n$ necessarily tends to infinity. $◊$

Example 35. 

Let Δ be finite, then $P_{\mathrm{inv}}({\Delta }^{\mathbb{Z}})$ is compact. Assume we made observations of a Δ-valued process and want to fit some $P\in {\mathcal{P}}_{\mathsf{inv}}({\Delta }^{\mathbb{Z}})$. From the observations, we might derive a set of closed constraints, e.g., $P(\{{\xi }_1={\xi }_2\})\in [a,b]$, $P(\{{\xi }_1=d\})\ge \epsilon$, and $P(\{{\xi }_2=d\}|{\xi }_1=d)\in [a,b]$ (the third is closed only in presence of the second). Further closed constraints may be given by modelling assumptions. Because the resulting set of admissible processes is compact, lower semi-continuity implies that there is at least one process of minimal complexity satisfying all constraints. $◊$