Entropy Production in Stochastics

Abstract

While the modern definition of entropy is genuinely probabilistic, in entropy production the classical thermodynamic definition, as in heat transfer, is typically used. Here we explore the concept of entropy production within stochastics and, particularly, two forms of entropy production in logarithmic time, unconditionally (EPLT) or conditionally on the past and present having been observed (CEPLT). We study the theoretical properties of both forms, in general and in application to a broad set of stochastic processes. A main question investigated, related to model identification and fitting from data, is how to estimate the entropy production from a time series. It turns out that there is a link of the EPLT with the climacogram, and of the CEPLT with two additional tools introduced here, namely the differenced climacogram and the climacospectrum. In particular, EPLT and CEPLT are related to slopes of log-log plots of these tools, with the asymptotic slopes at the tails being most important as they justify the emergence of scaling laws of second-order characteristics of stochastic processes. As a real-world application, we use an extraordinary long time series of turbulent velocity and show how a parsimonious stochastic model can be identified and fitted using the tools developed.

1. Introduction

Entropy was first recognized as a probabilistic concept in 1887 by Boltzmann [1], who established a relationship of entropy with probabilities of statistical mechanical system states, thus explaining the Second Law of Thermodynamics as the tendency of the system to run toward more probable states. In 1948 Shannon [2] used an essentially similar, albeit more general, definition of entropy as a probabilistic concept, a measure of information or, equivalently, uncertainty. In 1957 Jaynes [3] introduced the principle of maximum entropy thus equipping the entropy concept with a powerful tool for logical inference.

A decade later, probabilistic entropy and the principle of maximum entropy were used in geophysical sciences and particularly hydrology, initially for parameter estimation of models [4] and probability distributions [5]. Detailed reviews on the use of entropy in applications in hydrology, and water and environmental engineering have been provided by Singh ([6,7] and more recently [8,9,10,11]). Most applications of probabilistic entropy are static, disregarding time. In studying systems out of the equilibrium, time should necessarily be involved (e.g., [12]) and the paths of the systems should be inferred. The more recently developed, within non-equilibrium thermodynamics, extremum principles of entropy production attempt to predict the most likely path of system evolution [13,14,15]. Central among the latter are Prigogine’s minimum entropy production principle [16] and Ziegler’s maximum entropy production principle [17,18].

Entropy production in thermodynamic systems is typically defined in terms of the derivative of entropy with respect to time, while the entropy flux in open systems is also considered. Niven and Ozawa [19] provide a general definition of entropy production along with a brief review of applications of extremum principles in geophysics and particularly hydrology. It can be seen that in these applications, the entropy concept is typically used with its classical thermodynamic definition and in deterministic terms, without reference to its modern probabilistic definition.

Deterministic descriptions of the evolution of uncertain physical and geophysical systems are generally inefficient [20,21], while a much more powerful alternative is offered by stochastics. The term ‘stochastics’ describes the mathematics of random variables and stochastic processes, and comprises probability theory, statistics and stochastic processes. While the entropy concept is well defined within stochastics, the entropy production does not have a scientifically mature definition. By analogy to classical thermodynamic versions, we may define entropy production in stochastics as the derivative of (probabilistic) entropy with respect to time. As here we deal with a single stochastic process and not with interaction of many processes, there are no entropy fluxes and, thus, entropy production should merely rely on the time derivative of entropy. In an earlier study [22], the derivative with respect to the logarithm of time was introduced and termed ‘entropy production in logarithmic time’ (EPLT). Differentiation with respect to the logarithm of time has some attractive elements, such as:

• It is dimensionless as d(ln t) = dt/t and thus, given that entropy is a dimensionless quantity per se (see next section), the EPLT remains a dimensionless quantity.
• It is consistent with the notion of the ‘arrow of time’. Usually entropy per se is related to the time asymmetry and the Second Law is regarded as the origin of the arrow. However, it may be simpler to think that it is the action of observation that creates time asymmetry. In this respect, we can set t = 0 for the observation time, i.e., the present, while t < 0 denotes the observable past and t > 0 denotes the unknown future. The fact that the past is observable, but the future not, generates the asymmetry. It is clarified that when the future becomes observable, it is no longer future; rather it has become present or past. Once the future (t > 0) is treated separately from the past, it is legitimate to differentiate with respect to the logarithm of time.
• It makes extremization of entropy production easier, particularly for asymptotic times t → 0 and t → ∞, as it avoids infinite or zero values of entropy production [22]; indeed, the asymptotic values of EPLT are always bounded (see Section 2.4).
• It provides the means to study or even explain the scaling behaviour often observed in geophysical processes (see Section 2.2 and Section 2.4).

Here we advance the study of the EPLT concept, with particular emphasis on the conditional entropy production (CEPLT), when the past and present have been observed (Section 2.2). The theoretical properties of EPLT and CEPLT are studied in a general setting and in application to a broad set of specific types of stochastic processes (Section 3.1 and Section 3.2). A main question studied is how the entropy production can be estimated from time series, i.e., realizations of stochastic processes. It turns out that there is a strong link of the EPLT with the climacogram, with the slope of a log-log plot of the latter representing the EPLT. Further, we introduce two new tools, additional to the climacogram and based on it, the differenced climacogram and the climacospectrum, where the latter has properties similar to those of the power spectrum. The slope of a log-log plot of either of the two additional tools can be used as an estimator of CEPLT. Of particular importance are the asymptotic slopes for time scale tending to zero or infinity (Section 3.3). To illustrate the theoretical and empirical properties of EPLT and CEPLT using real-world data and to test the applicability of the specific models proposed, a long time series of very fine resolution is needed in order to allow viewing the real-world behaviour at the far-left and the far-right tails. Fine resolution measurements of turbulence can provide an ideal test series and indeed a time series of length 36 × 10⁶ of turbulent velocities is utilized for illustration and testing (Section 3.4).

To avoid bothering the reader with derivations and theoretical details, most of them have been put in a series of annexed sections organized as Appendix A.1. In addition, the details of the specific stochastic models and their behaviour have been grouped in Appendix A.2. While the two appendices are made in a stand-alone form separate from the body of the article, they are perhaps the most essential part of the study.

2. Methods

2.1. General Context

The evolution of a system state x over time t is usually represented as a trajectory x(t), but such a description is often inefficient. Instead, the evolution can be represented as a stochastic process x(t), which is a collection of (infinitely many) random variables x indexed by t. A realization (sample) x(t) of x(t) is a trajectory; if it is known at certain points t_i, i = 1, 2, …, the realization is called a time series.

A random variable x is an abstract mathematical entity associated with a probability density (or mass) function f(x); the realizations x of x belong to a set of possible numerical values. Notice the different notation of random variables and hence stochastic processes (underlined, according to the Dutch notation [23]) from regular ones.

Most natural processes evolve in continuous time but they are observed in discrete time, typically by averaging. Accordingly, the stochastic processes devised to represent the natural processes should evolve in continuous time and be converted into discrete time, as illustrated in Figure 1.

While a stochastic process denotes, by conception, change (process = a series of changes), there should be some properties that are unchanged in time. This implies the concept of stationarity [25], which is central in stochastics. According to Kolmogorov’s definition [26], a stochastic process x(t) is stationary if the distributions of $\left(x\left(t_1\right),x\left(t_2\right),\dots ,x\left(t_n\right)\right)$ and $\left(x\left(t_1+\tau \right),x\left(t_2+\tau \right),\dots ,x\left(t_n+\tau \right)\right)$ coincide for any $n, t_1, t_2, \dots ,t_n, \tau$. By negation, in a nonstationary process the probability density $f\left(x\left(t_1\right),x\left(t_2\right),\dots ,x\left(t_n\right)\right)$ for some (or all) τ is not equal to $f\left(x\left(t_1+\tau \right),x\left(t_2+\tau \right),\dots ,x\left(t_n+\tau \right)\right)$, which means that the mathematical expression of the latter should explicitly contain the time shift τ, or else that it should be a deterministic function of τ.

Stationarity is closely related to ergodicity, which in turn is a prerequisite to make inference from data, that is, induction. Within the stochastics domain ([20,27] (p. 427)) ergodicity is defined in the following manner: A stochastic process x(t) is ergodic if the time average of any function $g\left(\underset{\_}{x}\left(t\right)\right)$, as time tends to infinity, equals the true (ensemble) expectation $\mathrm{E}\left[g\left(\underset{\_}{x}\left(t\right)\right)\right]$. This allows the estimation (i.e., approximate calculation) of the true but unknown quantity $\mathrm{E}\left[g\left(\underset{\_}{x}\left(t\right)\right)\right]$ (e.g., the true average $\mathrm{E}\left[\underset{\_}{x}\left(t\right)\right]$) from the available data, i.e., from the sample mean). Without stationarity there is no ergodicity and without ergodicity inference from data is impossible. More details about stationarity and ergodicity, and their importance, are provided by Koutsoyiannis and Montanari [25], along with highlights of the misconceptions and abuses of these concepts which abound in the literature.

For the remaining part of this article, unless otherwise stated, the processes are assumed to be stationary and ergodic, noting that nonstationary processes should be converted to stationary before their study. For example, the cumulative process X(t) in Figure 1 is nonstationary, but by differentiating it in time we obtain the stationary process x(t).

Without loss of generality we assume that the mean of the process x(t) is zero (E[x(t)] = 0) and we denote its variance as:

γ₀ ≔ Var[x(t)]

(1)

Let:

Γ(t) ≔ Var[X(t)], γ(k) ≔ Var[(1/k)(X(t + k) − X(t))] = Γ(k)/k²

(2)

be the variances of the cumulative and the averaged process, respectively (note that Γ(0) = 0 and γ(0) = γ₀). While Γ(t), the variance of the nonstationary process X(t), is a function of t, γ(k) is the variance of the stationary process ${\underset{\_}{x}}_{\tau }^{\left(k\right)}$, i.e., $\gamma \left(k\right)=\mathrm{Var}[{\underset{\_}{x}}_{\tau }^{\left(k\right)}]$ and thus it is not a function of time τ. We call Γ(t) and γ(k), as functions of time t and time scale k, the cumulative climacogram and the climacogram of the process, respectively.

The climacogram is the second central moment of the process, as a function of time scale, and thus it is a second-order characteristic of the process. Other customary second order characteristics of a stationary process are the autocovariance function c(h), where h denotes time lag, the power spectrum s(w), where w denotes frequency, and the structure function v(h) (also known, for a stationary process, as the semivariogram or variogram). The latter has a direct analogy based on the climacogram, the climacostructure function ξ(k). Definitions and notation for these second order characteristics are contained in

Table 1. Definition of main characteristics and notations for a stochastic process in continuous time; see Figure 1 for clarification.

Name of Quantity or Characteristic	Symbol and Definition	Remarks	Equation
Stochastic process of interest	x(t)	Assumed stationary
Time, continuous	t	Dimensional quantity
Cumulative process	$\underset{\_}{X}(t)≔{\int }_0^t\underset{\_}{x}(\xi )\mathrm{d}\xi$	Nonstationary	(3)
Variance, instantaneous	γ0 ≔ Var[x(t)]	Constant (not a function of t)	(4)
Cumulative climacogram	Γ(t) ≔ Var[X(t)]	A function of t, Γ(0) = 0	(5)
Climacogram	γ(k) ≔ Var[(1/k)(X(t + k) − X(t))] = Var[X(k)/k] = Γ(k)/k2	Not a function of t, γ(0) = γ0	(6)
Time scale, continuous	k	Units of time
Autocovariance function	c(h) := Cov[x(t), x(t + h)]	c(0) = γ0	(7)
Time lag, continuous	h	Units of time
Structure function (or semivariogram or variogram)	$v(h)≔\frac{1}{2}\mathrm{Var}$[x(t) − x(t + h)]		(8)
Climacostructure function	ξ(k) ≔ γ0 − γ(k)		(9)
Power spectrum (or spectral density)	$s(w)≔4{\int }_0^{\infty }c(h)\cos (2\mathsf{\pi }wh)\mathrm{d}h$		(10)
Frequency, continuous	w = 1/k	Units of inverse time	(11)

for the continuous time domain and in

Table 2. Definition of main characteristics and notations for a stochastic process in discrete time and their relationship with those in continuous time; see Figure 1 for clarification.

Name of Quantity or Characteristic	Symbol and Definition	Remarks	Equation
Stochastic process, discrete time	${\underset{\_}{x}}_{\tau }^{\left(D\right)}≔\frac{1}{D}{{\displaystyle \int }}_{\left(\tau -1\right)D}^{\tau D}\underset{\_}{x}\left(u\right)\mathrm{d}u=$ $\frac{1}{D}\left(\underset{\_}{X}\left(\tau D\right)-\underset{\_}{X}\left(\left(\tau -1\right)D\right)\right)$		(12)
Time unit = discretization time step	D	Time step or length of time window of averaging
Time, discrete	τ := t/D	Dimensionless quantity, integer	(13)
Characteristic variance	Var[ ${\underset{\_}{x}}_{\tau }^{\left(D\right)}$] = γ(D)		(14)
Climacogram	${\gamma }_{\kappa }^{\left(D\right)}≔\gamma \left(\kappa D\right)=\frac{\Gamma \left(\kappa D\right)}{{\left(\kappa D\right)}^2}$	${\gamma }_1^{\left(D\right)}$ = γ(D)	(15)
Time scale, discrete	κ := k/D	Dimensionless quantity	(16)
Autocovariance function	$c_{\eta }^{\left(D\right)}$≔ Cov[${\underset{\_}{x}}_{\tau }^{\left(D\right)},{\underset{\_}{x}}_{\tau +\eta }^{\left(D\right)}$]	$c_0^{\left(D\right)}=\gamma \left(D\right)$
Time lag, discrete	η := h/D	Dimensionless quantity	(17)
Structure function	$v_{\eta }^{\left(D\right)}≔\gamma \left(D\right)-c_{\eta }^{\left(D\right)}$		(18)
Power spectrum	$s_{\mathrm{d}}^{\left(D\right)}\left(\omega \right)≔$ $\frac{1}{D}{{\displaystyle \sum }}_{j=-\infty }^{\infty }s\left(\frac{\omega +j}{D}\right){\mathrm{sinc}}^2\left(\mathsf{\pi }\left(\omega +j\right)\right)$		(19)
Frequency, discrete	ω := wD = 1/κ	Dimensionless quantity	(20)

Note: In time-related quantities, Latin letters denote dimensional quantities and Greek letters dimensionless ones. The Latin i, j, l may also be used as integers to denote quantities τ, η, κ, depending on the context.

for the discrete time case. All of these functions are transformations of one another, as shown in

Table 3. Relationships between several characteristics of a process in continuous and discrete time.

Related Characteristics	Direct Relationship	Inverse Relationship	Equation
$\gamma \left(k\right)\leftrightarrow c\left(h\right)$	$\gamma \left(k\right)=2{{\displaystyle \int }}_0^1\left(1-\chi \right)c\left(\chi k\right)\mathrm{d}\chi$	$c\left(h\right)=\frac{1}{2}\frac{{\mathrm{d}}^2\left(h^2\gamma \left(h\right)\right)}{\mathrm{d}{h}^2}$	(21)
$s\left(w\right)\leftrightarrow c\left(h\right)$	$s\left(w\right)≔4{{\displaystyle \int }}_0^{\infty }c\left(h\right)\cos \left(2\mathsf{\pi }wh\right)\mathrm{d}h$	$c\left(h\right)={{\displaystyle \int }}_0^{\infty }s\left(w\right)\cos \left(2\mathsf{\pi }wh\right)\mathrm{d}w$	(22)
$\gamma \left(k\right)\leftrightarrow s\left(w\right)$	$\gamma \left(k\right)={{\displaystyle \int }}_0^{\infty }s\left(w\right){\mathrm{sinc}}^2\left(\mathsf{\pi }wk\right)\mathrm{d}w$	$s\left(w\right)≔2{{\displaystyle \int }}_0^{\infty }\frac{{\mathrm{d}}^2\left(h^2\gamma \left(h\right)\right)}{\mathrm{d}{h}^2}\cos \left(2\mathsf{\pi }wh\right)\mathrm{d}h$	(23)
$v\left(h\right)\leftrightarrow c\left(h\right)$	$v\left(h\right)$ $={\gamma }_0-c\left(h\right)$	$c\left(h\right)=v(\infty )-v\left(h\right)\left(v(\infty )={\gamma }_0\right)$	(24)
$\xi \left(k\right)\leftrightarrow \gamma \left(k\right)$	ξ(k) ≔ γ0 − γ(k)	γ(k) = ξ(∞) − ξ(k) ($\xi (\infty )={\gamma }_0)$	(25)
$\xi \left(k\right)\leftrightarrow v\left(h\right)$	$\xi \left(k\right)=2{{\displaystyle \int }}_0^1\left(1-\chi \right)v\left(\chi k\right)\mathrm{d}\chi$	$v\left(h\right)=\frac{1}{2}\frac{{\mathrm{d}}^2\left(h^2\xi \left(h\right)\right)}{\mathrm{d}{h}^2}$	(26)
${\gamma }_{\kappa }^{\left(D\right)}\equiv \gamma \left(\kappa D\right)\leftrightarrow c_{\eta }^{\left(D\right)}$	${\gamma }_{\kappa }^{\left(D\right)}=\frac{1}{\kappa }\left(c_0^{\left(D\right)}+2{\displaystyle {\displaystyle \sum }_{\eta =1}^{\kappa -1}}\left(1-\frac{\eta }{\kappa }\right)c_{\eta }^{\left(D\right)}\right)$ Alternatively, ${\gamma }_{\kappa }^{\left(D\right)}=\frac{\Gamma \left(\kappa D\right)}{{\left(\kappa D\right)}^2}$ where, in recursive mode, $\Gamma \left(\kappa D\right)=2\Gamma \left(\left(\kappa -1\right)D\right)-\Gamma \left(\left(\kappa -2\right)D\right)+2c_{j-1}^{\left(D\right)}{D}^2$ with $\Gamma \left(0\right)=0$, $\Gamma \left(D\right)=$ $c_0^{\left(D\right)}{D}^2$	$c_{\eta }^{\left(D\right)}=\frac{1}{D^2}\left(\frac{\Gamma \left(\left|\eta +1\right|D\right)+\Gamma \left((|\eta -1|D\right)}{2}-\Gamma \left(\left|\eta \right|D\right)\right)$	(27)
$c_{\eta }^{\left(D\right)}\leftrightarrow s_{\mathrm{d}}^{\left(D\right)}\left(\omega \right)$	$s_{\mathrm{d}}^{\left(D\right)}\left(\omega \right)=2c_0^{\left(D\right)}+4{{\displaystyle \sum }}_{\eta =1}^{\infty }{c}_{\eta }^{\left(D\right)}\cos \left(2\mathsf{\pi }\eta \omega \right)$	$c_{\eta }^{\left(D\right)}={{\displaystyle \int }}_0^{1/2}{s}_{\mathrm{d}}^{\left(D\right)}\left(\omega \right)\cos \left(2\mathsf{\pi }\omega \eta \right)\mathrm{d}\omega$	(28)
$v_{\eta }^{\left(D\right)}\leftrightarrow c_{\eta }^{\left(D\right)}$	$v_{\eta }^{\left(D\right)}=\gamma \left(D\right)-c_{\eta }^{\left(D\right)}$	$c_{\eta }^{\left(D\right)}≔\gamma \left(D\right)-v_{\eta }^{\left(D\right)}$	(29)

.

The climacogram, like the autocovariance function, is a positive definite function (see proof in Appendix A.1.2) but of the time scale k, rather than the time lag h. It is not as popular as the other tools but it has several good properties due to its simplicity, close relationship to entropy (see below), and more stable behaviour, which is an advantage in model identification and fitting from data. In particular, when estimated from data, the climacogram behaves better than all other tools, which involve high bias and statistical variation [24,28].

The climacogram involves bias too, but this can be determined analytically and included in the estimation. Specifically, assuming that we have n observations of the averaged process ${\underset{\_}{x}}_i^{\left(k\right)}$, at scale k = κD, so that T = nk is the observation period (rounded off to an integer multiple of k), the standard statistical estimator of variance has expected value:

$$\mathrm{E}\left[\underset{\_}{\overset{^}{\gamma }}\left(k\right)\right]=\theta \left(k,{\rm T}\right)\gamma \left(k\right)$$

(30)

where θ is the bias correction coefficient, given by [22,24]:

$$\theta \left(k,{\rm T}\right)=\frac{1-\gamma \left(T\right)/\gamma \left(k\right)}{1-k/T}=\frac{1-{\left(k/{\rm T}\right)}^2\Gamma \left({\rm T}\right)/\Gamma \left(k\right)}{1-k/T}$$

(31)

It becomes clear from the above equations, which are valid for any process, that direct estimation of the variance $\gamma \left(k\right)$ for any time scale k, let alone that of the instantaneous process γ₀, is not possible merely from the data. We need to know the ratio $\gamma \left(T\right)∕\gamma \left(k\right)$ and thus we should assume a stochastic model which evidently influences the estimation of $\gamma \left(k\right)$. Once the model is assumed and its parameters estimated based on the data, we can expand our calculations to estimate the variance for any time scale, including that of the instantaneous scale γ₀.

2.2. Entropy and Entropy Production

Historically entropy was introduced in thermodynamics but its rigorous definition was given within probability theory. Thermodynamic and probabilistic entropy are regarded by many as different concepts having in common only their name. However, here we embrace the view that they are essentially the same thing, as has articulated elsewhere [29,30]. According to the latter interpretation, the mathematical description of thermodynamic systems could be produced by the probabilistic definition of entropy. As an example indicating how powerful this interpretation could be, the law of phase change transition of water (Clausius-Clapeyron equation) has been produced in [30], with impressive agreement with reality, by maximizing, in a formal probabilistic frame, the entropy, i.e., the total uncertainty about the state of a single molecule of water.

In this frame, entropy (often called Boltzmann-Gibbs-Shannon entropy to give credit to its pioneers and distinguish it from other forms of generalized entropies) is a dimensionless measure of uncertainty defined as follows:

• For a discrete random variable x with probability mass function P_j ≔ P{x = x_j}, j = 1, …, w:

  $$\Phi [\underset{\_}{x}]≔\mathrm{E}[-\ln P\underset{\_}{x}]=-{{\displaystyle \sum }}_{j=1}^w{P}_j\ln P_j$$

  (32)
• For a continuous random variable z with probability density function f(z):

  $$\Phi [\underset{\_}{x}]≔\mathrm{E}\left[-\ln \frac{f\left(\underset{\_}{x}\right)}{m\left(\underset{\_}{x}\right)}\right]=-{{\displaystyle \int }}_{-\infty }^{\infty }\ln \frac{f\left(x\right)}{m\left(x\right)}f\left(x\right)\mathrm{d}x$$

  (33)

  where m(x) is the density of a background measure (usually Lebesgue, i.e., m(x) = 1, with dimensions [x⁻¹]).

Entropy acquires its importance from the principle of maximum entropy [3], which postulates that the entropy of a random variable should be at maximum, under some conditions, formulated as constraints, which incorporate the information that is given about this variable. Its physical counterpart, the tendency of entropy to become maximal (Second Law of thermodynamics) is the driving force of natural change.

The definition of the entropy of a stochastic process directly follows that of a random variable. Thus, the entropy of the cumulative process X(t) with probability density function f(X; t) is a dimensionless quantity defined as:

$$\Phi \left[\underset{\_}{X}\left(t\right)\right]≔\mathrm{E}\left[-\ln \frac{f\left(\underset{\_}{X};t\right)}{m\left(\underset{\_}{X}\right)}\right]=-\underset{-\infty }{\overset{\infty }{{\displaystyle \int }}}\ln \frac{f\left(X;t\right)}{m\left(X\right)}f\left(X;t\right)dX$$

(34)

In a stochastic process, which by definition involves time, it is natural to consider the change of entropy in time. Intuitively, for the stochastic process X(t) we can define entropy production as the time derivative, i.e., Φ′[X(t)] := dΦ[X(t)]/dt. This has dimensions of inverse time. Koutsoyiannis [22] defined the entropy production in logarithmic time (EPLT) as the derivative of entropy in logarithmic time:

φ(t) ≡ φ[X(t)] ≔ Φ’[X(t)] t ≡ dΦ[X(t)]/d(lnt)

(35)

and showed that it has some advantages over Φ’[x(t)] as described above.

Assuming that the background density is constant, m(Χ) ≡ m, and using simple constraints related to the preservation of the mean and variance, the principle of maximum entropy yields that the probability distribution of X will be Gaussian. For a Gaussian process the entropy depends on its variance Γ(t) and is:

Φ[X(t)] = (1/2) ln(2πe Γ(t)/m²)

(36)

Hence:

φ(t) = Γ’(t) t/2Γ(t) = 1+ γ’(t) t/2γ(t)

(37)

Because of the simplicity of the Gaussian case, we are basing further analyses on that assumption, noting that for large times, due to the Central Limit Theorem, the cumulative process X(t) will tend to Gaussian, even if for the smallest scales it deviates from Gaussian.

A question arises whether φ(t) is a totally abstract concept useful for theoretical analyses only or it can be related to data, and visualized or estimated from them. The reply is fortunate if, in addition, we consider another concept, the log-log derivative (LLD) of a function f(x), formally expressed by:

$$f^{\#}\left(x\right)≔\frac{\mathrm{d}(\ln f\left(x\right))}{\mathrm{d}\left(\ln x\right)}=\frac{x{f}^{\prime }\left(x\right)}{f\left(x\right)}$$

(38)

This is visualized as the (local) slope on the popular log-log plot (plot of log f(x) vs. log x). It is then seen that (37) can be written as:

φ(t) = ½ Γ^#(t) = 1 + ½ γ^#(t)

(39)

In other words, EPLT is visualized and estimated by the slope of a log-log plot of the climacogram.

The quantity φ(t) represents the entropy production when nothing is known (observed) about the process realization. More interesting is the conditional EPLT (CEPLT, φ_C(t)) when the past (t < 0) and the present (t = 0) are observed. In this case, instead of the unconditional variance Γ(t) we should use a variance Γ_C(t) conditional on the past and present. It has been shown [22,24] that for a Markov process (see definition of the Markov process in Section 3.1 and its details in Appendix A.2.1), the conditional variance is:

$${\Gamma }_{\mathrm{C}}\left(t\right)\approx 2\Gamma \left(t\right)-\frac{\Gamma \left(2t\right)}{2},\hspace{1em} {\gamma }_{\mathrm{C}}\left(k\right)\approx 2\left(\gamma \left(k\right)-\gamma \left(2k\right)\right)$$

(40)

We note that the difference γ(k) − γ(2k) is always nonnegative (this is shown in Appendix A.1.3, Equation (A21)) and thus it can indeed represent a (conditional) variance.

For a non-Markov process the following adaptation is necessary:

$${\gamma }_{\mathrm{C}}\left(k\right)=\epsilon \left(\gamma \left(k\right)–\gamma \left(2k\right)\right)$$

(41)

where ε is a constant, ensuring that as k → ∞, ${\gamma }_{\mathrm{C}}\left(k\right)/\gamma \left(k\right)\to 1$; the rationale of this limit is that the information about the past and present will tend not to affect the far distant future. As shown in the Appendix A.1.4, its value is:

$$\epsilon =\frac{1}{1-2^{{\gamma }^{\#}(\infty )}}$$

(42)

It is easily verified that for a Markov process in which ${\gamma }^{\#}(\infty )=-1$ (see Appendix A.2.1), ε = 2, while for a Hurst-Kolmogorov process (original or filtered; see details in [24], in next session and in Appendix A.2) for which ${\gamma }^{\#}(\infty )=2H-2$, $\epsilon =1/\left(1-2^{2H-2}\right)$, where H is the Hurst parameter.

Due to its mathematical form, we will refer to ${\gamma }_{\mathrm{C}}\left(k\right)$ as the differenced climacogram. The inverse transformation, i.e., that giving the climacogram $\gamma \left(k\right)$ once the differenced climacogram ${\gamma }_{\mathrm{C}}\left(k\right)$ is known, can be easily deduced by iterative evaluations also observing that $\gamma (\infty )={\gamma }_{\mathrm{C}}(\infty )=0$ (this is a necessary condition in order for a stochastic process to be ergodic [27] (p. 429)) and takes both the following forms:

$$\gamma \left(k\right)=\frac{1}{\epsilon }{\displaystyle \sum }_{i=0}^{\infty }{\gamma }_{\mathrm{C}}\left(2^{i}k\right)=\gamma \left(0\right)-\frac{1}{\epsilon }{\displaystyle \sum }_{i=1}^{\infty }{\gamma }_{\mathrm{C}}\left(2^{-i}k\right)$$

(43)

whereas for numerical evaluations it should be kept in mind that for large time scales k, $\gamma \left(k\right)\approx {\gamma }_{\mathrm{C}}\left(k\right)$. Note that (43) entails:

$$\gamma \left(0\right)=\frac{1}{\epsilon }{\displaystyle \sum }_{i=-\infty }^{\infty }{\gamma }_{\mathrm{C}}\left(2^{i}k\right)$$

(44)

and this is valid for any k. The importance of the differenced climacogram stems from the fact that it is directly related to the CEPLT, through its obvious relationship (similar to (39)):

$${\phi }_{\mathrm{C}}\left(k\right)=1+\text{½}{\gamma }_{\mathrm{C}}^{\#}\left(k\right)$$

(45)

As the differenced climacogram can be easily visualized from data, because it only needs calculation of variances, it directly provides a means for estimation and visualization of the CEPLT in terms of the slope in a log-log plot of the differenced climacogram.

In Appendix A.1.5 it is shown that the asymptotic properties of ${\gamma }_C\left(k\right)$ are:

$${\gamma }_{\mathrm{C}}\left(0\right)={\gamma }_{\mathrm{C}}(\infty )=0,\hspace{1em}{\gamma }_{\mathrm{C}}^{\#}\left(0\right)={\xi }^{\#}\left(0\right),\hspace{1em}{\gamma }_{\mathrm{C}}^{\#}(\infty )={\gamma }^{\#}(\infty )$$

(46)

In other words, we need both the climacogram γ(k) and the climacostructure function ξ(k) to express the asymptotic behaviour of ${\gamma }_C\left(k\right)$. Once this is known, the asymptotic behaviour of CEPLT is also known from (45).

As shown in Appendix A.1.6, the differenced climacogram for persistent processes (see their definition in next subsection) is not integrable in (0, ∞). However, by slightly changing it multiplying with k, we can obtain an additional tool which is integrable and has some additional good properties. Specifically, we introduce the following tool, which, as we will see next, resembles the power spectrum and thus we refer to it as the climacospectrum:

$$\zeta \left(k\right)≔\frac{k\left(\gamma \left(k\right)-\gamma \left(2k\right)\right)}{\ln 2}$$

(47)

The climacospectrum is also written in an alternative manner in terms of frequency w = 1/k:

$$\tilde{\zeta }\left(w\right)≔\zeta \left(1/w\right)=\frac{\gamma \left(1/w\right)-\gamma \left(2/w\right)}{(\ln 2)w}$$

(48)

The inverse transformation, i.e., that giving the climacogram $\gamma \left(k\right)$ once the climacospectrum $\zeta \left(k\right)$ is known, can be easily deduced with the same method as before, and takes the form:

$$\gamma \left(k\right)=\ln 2{\displaystyle \sum }_{i=0}^{\infty }\frac{\zeta \left(2^{i}k\right)}{2^{i}k}=\gamma \left(0\right)-\ln 2{\displaystyle \sum }_{i=1}^{\infty }\frac{\zeta \left(2^{-i}k\right)}{2^{-i}k}$$

(49)

A very good approximation in which the sum is replaced by an integral, is:

$$\gamma \left(k\right)\approx \frac{(\epsilon \ln 2)k+2}{2k+2}I\left(k\right),\hspace{1em}I\left(k\right)≔\underset{k}{\overset{\infty }{{\displaystyle \int }}}\frac{\zeta \left(x\right)}{x^2}\mathrm{d}x=\underset{0}{\overset{1/k}{{\displaystyle \int }}}\tilde{\zeta }\left(w\right)\mathrm{d}w$$

(50)

This holds for all k and its details are given in Appendix A.1.7. For small k, it is easy to see that the multiplier of the integral I(k) is 1. It follows that for small time scales k or large frequencies w = 1/k the climacospectrum, expressed in terms of frequency, has an area I(k) on the left of w equal to the variance γ(1/w). As shown in Appendix A.1.6, the entire area under the curve $\tilde{\zeta }\left(w\right)$ (i.e., I(0)) is precisely equal to the variance γ(0) of the instantaneous process, a property also shared by the power spectrum s(w). This is not the only connection with the power spectrum. The climacospectrum has also the same asymptotic behaviour with the power spectrum, i.e.,

$${\tilde{\zeta }}^{\#}\left(0\right)=-{\zeta }^{\#}(\infty )=s^{\#}\left(0\right),\hspace{1em}{\tilde{\zeta }}^{\#}(\infty )=-{\zeta }^{\#}\left(0\right)=s^{\#}(\infty )$$

(51)

(see Appendix A.1.9). This property holds almost always, with the exception of the cases where ${\zeta }^{\#}\left(0\right)$ is a specific integer (${\zeta }^{\#}(\infty )=-1$ or ${\zeta }^{\#}\left(0\right)=3$, as explained in Appendix A.1.9).

Again the connection of the climacospectrum with the CEPLT emerges through log-log derivatives. Specifically, combining (41), (45) and (47) we find that ${\zeta }^{\#}\left(k\right)=1+{\gamma }_C^{\#}\left(k\right)=1+2{\phi }_{\mathrm{C}}\left(k\right)-2$; hence:

$${\phi }_{\mathrm{C}}\left(k\right)=\text{½}\left(1+{\zeta }^{\#}\left(k\right)\right)=\text{½}\left(1-{\tilde{\zeta }}^{\#}\left(1/k\right)\right)$$

(52)

By virtue of (51) and with the exceptions mentioned, the asymptotic CEPLT can also be inferred from the slopes of the power spectrum and, as the climacospectrum slopes do not differ substantially from those of the power spectrum even for intermediate time scales, the power spectrum slopes could also be used for a qualitative estimation if the CEPLT.

Obviously, however, the climacospectrum is more advantageous than the power spectrum in this respect, because the connection with conditional entropy production is more precise and without exceptions at all. In addition, intuitively the variance, on which the definition of the climacospectrum is based, is more closely related to uncertainty, and hence entropy of the process, than the power spectrum and the autocovariance. Other advantages are the easy calculation only using the concept of variance, without any need to perform laborious transformations (such as Fourier) and the fact that, like the climacogram, it is not affected by discretization (while autocovariance and power spectrum are). The empirical climacogram and climacospectrum are easily determined from data using nothing more than the standard statistical estimator of variance and they have a smooth shape, much smoother than those of the empirical autocovariance and power spectrum, thus enabling better model identification and fitting.

When the objective is the model fitting, we should be aware that any of the statistical tools entail estimation bias, which should be accounted for in the fitting. Again the climacogram and the climacospectrum are preferable than other tools as the bias is explicitly calculated from the assumed model (Equations (30) and (31)). In particular, the bias of the climacospectrum is usually very small (see the relevant graphs in Appendix A.2) because of its definition as a difference of two variances, in which the biases tend to cancel out. In comparison with another version of climacogram-based spectrum [24], the climacospectrum introduced here has advantages. Namely, these are its simpler expression, as it is a linear transformation (difference) of the climacogram, and the absence of the variance γ₀ ≡ γ(0) from the definition, which makes possible the calculation of the empirical climacospectrum prior to specifying the model (note that γ₀ is not known before a model is specified and its parameters are estimated).

2.3. Scaling

In the previous subsection we have shown that the EPLT and the CEPLT are related to LLDs or slopes of log-log plots of second order tools such as climacogram, climacospectrum, power spectrum, etc. With a few exceptions, these slopes are nonzero asymptotically, hence entailing asymptotic scaling or asymptotic power laws with the LLDs being the scaling exponents. It is intuitive to expect that an emerging asymptotic scaling law would provide a good approximation of the true law for a range of scales. If the scaling law was appropriate for the entire range of scales, then we would have a simple scaling law. Such simple scaling sounds attractive from a mathematical point of view, but, for reasons that will be explained in the next subsection, it turns out to be impossible to make it appropriate for physical processes.

It is thus physically more realistic to expect two different types of asymptotic scaling laws, one in each of the ends of the continuum of scales. We call these two behaviours local scaling or fractal behaviour, when k → 0 or w → ∞, and global scaling, persistence or Hurst-Kolmogorov behaviour, when k → ∞ or w → 0. The respective scaling exponents are (see Appendix A.1.8):

• For local scaling: ${\gamma }_{\mathrm{C}}^{\#}\left(0\right)={\xi }^{\#}\left(0\right)= v^{\#}\left(0\right)={\zeta }^{\#}\left(0\right)-1= 2{\phi }_{\mathrm{C}}\left(0\right)-2=-s^{\#}(\infty )-1$
• For global scaling: ${\gamma }_{\mathrm{C}}^{\#}(\infty )={\gamma }^{\#}(\infty )=c^{\#}(\infty )={\zeta }^{\#}(\infty )-1= 2{\phi }_{\mathrm{C}}(\infty )-2=-s^{\#}\left(0\right)-1$

Here, the emergence of scaling has been related to maximum entropy considerations, and this may provide the theoretical background in modelling complex natural processes by such scaling laws. However, as shown in Appendix A.1.8, scaling laws are a mathematical necessity and could be constructed for virtually any continuous function defined in (0, ∞). In other words, there is no magic in power laws, except that they are, logically and mathematically, a necessity. (No assumption of criticality, self-organization, fractal or multi-fractal generating mechanisms, is necessary to justify their emergence). Here we have focused on second order characteristics of a stochastic process. If we also considered higher order (raw) moments, it is natural to expect that some other (different) power laws would emerge (or could be constructed using the procedure described in Appendix A.1.8), and then one would speak about multifractals everywhere, etc.

2.4. Bounds of Scaling and Entropy Production

It is not well known that the asymptotic scaling exponents cannot take on any arbitrary values but they should be constrained in rather narrow ranges. Instead, misconceptions and incorrect applications resulting in inconsistent values abound in the literature. One of the most typical examples is the asymptotic scaling of the power spectrum for w → 0, for which often steep slopes are reported for low frequencies (down to 0), based on data analyses. Usually these reports are accompanied with a claim that the process with $-s^{\#}\left(0\right)>1$ is nonstationary, going further to characterize the power spectrum as a tool to identify whether a process is stationary or nonstationary (with values $-s^{\#}\left(0\right)$ below or above 1 suggesting stationarity and nonstationarity, respectively). As thoroughly studied elsewhere [31], this entire line of thought is theoretically inconsistent and flawed, while such results often reported are usually artefacts due to insufficient data or inadequate estimation algorithms. In fact, it has been shown [31] that a process with $-s^{\#}\left(0\right)>1$ is nonergodic. Inference from data is only possible when the process is ergodic and thus, claiming that $-s^{\#}\left(0\right)>1$ based on data is self-contradictory. Furthermore, claiming nonstationarity using the power spectrum as a function of frequency is also self-contradictory as in a nonstationary process both the autocovariance and the spectral density, i.e., the Fourier transform of the autocovariance, are functions of two variables, one being related to “absolute” time (see e.g., [32]).

An example of the conditions leading to misinterpreting a stationary process as nonstationary is discussed in Section 3.3 and Appendix A.2.2. We must clarify though that steep slopes (–s^#(w) > 1) are mathematically and physically possible for medium and large w (see, e.g., Figure A7) and indeed they are quite frequent in geophysical and other processes. In this respect, the estimation from data of steep slopes is possible and not problematic, if we are conscious that such slopes are for intermediate or large frequencies (actually, the estimation algorithms are bandpass filtered with the bounds of the filter determined by the resolution and length of the series). The problem arises when such slopes are interpreted as asymptotic ones and particularly when they are used for inference resulting in self-contradicting conclusions.

Because of the equality of slopes of power spectrum and climacospectrum, the ergodicity limitation holds also for the slope of the climacospectrum, i.e., ${\zeta }^{\#}(\infty )= -{\tilde{\zeta }}^{\#}\left(0\right)<1$. On the other hand, too steep negative asymptotic slopes of the climacospectrum are also impossible. Indeed, ${\zeta }^{\#}\left(k\right)= -{\tilde{\zeta }}^{\#}\left(1/k\right)<-1$ would entail (because of (52)) ${\phi }_{\mathrm{C}}\left(k\right)<0$ and (because of (37) and (45)), ${\Gamma }_{\mathrm{C}}^{\prime }\left(k\right)$ < 0. This means that the variance of the cumulative process would be a decreasing function of time, which is absurd. This holds both for the global case (k → ∞, in which the conditional variance ${\Gamma }_{\mathrm{C}}(\infty )$ equals the unconditional $\Gamma (\infty )$) and the local case (k → 0, for the conditional variance ${\Gamma }_{\mathrm{C}}\left(0\right)$).

However, in the local case there is a more severe limitation imposed by physical reasoning. As demonstrated in Appendix A.1.10, the case ${\zeta }^{\#}\left(0\right)=-s^{\#}(\infty )<1$ would entail infinite variance. Infinite variance would require infinite energy to emerge, which is physically inconsistent. Therefore, the physical lower limit for ${\zeta }^{\#}\left(0\right)=-s^{\#}(\infty )$ is 1. Interestingly, the inconsistency if this constraint is violated stems from the behaviour at high frequencies (see Appendix A.1.10), even though the power spectrum at high frequencies in such cases tends to zero rather diverging to infinity. Therefore, the related “catastrophe” is of “ultraviolet” type, while the infinite value of the power spectrum at low frequencies, which was highlighted in several texts as “infrared catastrophe” [33] is not actually a problem at all, nor does it impose any limitation on the scaling law additional to the limitation for ergodicity discussed above.

A final—and quite severe—limitation is an upper bound of the local scaling exponent, which is 3 for ${\zeta }^{\#}\left(0\right)=-s^{\#}(\infty )$. Proof is provided in Appendix A.1.11. The problem if this limitation is violated is that the resulting autocovariance function is not positive definite or, equivalently, that the resulting power spectrum is not always (for any frequency w) positive but takes on negative values for some w. Likewise, the Fourier transform of the climacogram takes on negative values for some w.

Because of the relationship of the entropy production with the scaling exponents, the bounds on the spectrum slopes translate directly into bounds of the CEPLT, i.e., $1\le {\phi }_{\mathrm{C}}\left(0\right)\le 2$ and $0\le {\phi }_{\mathrm{C}}(\infty )\le 1$. These define the “green square” of admissible values of φ_C in Figure 2, which is also depicted in terms of admissible values of slopes ζ^# and s^# (noting that, as discussed above, s^# can, by exception, take on values out of the square when ${\phi }_{\mathrm{C}}\left(0\right)=2$ or ${\phi }_{\mathrm{C}}(\infty )=0$. The reasons why a process out of the square would be impossible or inconsistent, as discussed above, are also marked in Figure 2.

Large values of ${\phi }_{\mathrm{C}}\left(0\right)$ indicate a smooth process and small ones a rough process. Also, large values of ${\phi }_{\mathrm{C}}(\infty )$ indicate a persistent process and small ones an antipersistent process. The centre of the square, with coordinates ${\phi }_{\mathrm{C}}\left(0\right)=3/2,$ ${\phi }_{\mathrm{C}}(\infty )=1/2$ represents a neutral process. As will be seen in next section, neutrality corresponds to a Markov process, which is neither rough nor smooth, and neither antipersistent nor persistent. Even though sometimes it is said to reflect short-term persistence, we must have in mind that a discretized Markov process at time scale k tends to be uncorrelated in time as k increases and therefore we may not have it in mind as persistent. In other words, here persistence is used as synonymous to long-term persistence, while high autocorrelations at small lags or scales, which die off at large lags or scales, are better described with the term smoothness. Both smoothness and persistence reflect high entropy production, locally and globally, respectively.

A useful observation in Figure 2 is that the entire “green square” lies below the equality line, which means that the same scaling exponent is not possible for both local and global behaviour, or else, it is impossible to have a physically realistic simple scaling process. There is one exception, the upper-left corner of the “green square”, which corresponds to the so-called “pink noise” or “1/f noise” and will be discussed further in Section 3.3.

Since we have pointed out several inconsistent or contradictory results that are reported in the literature, it may be useful to mention another case, which is frequently met in geostatistics. The so-called intrinsic models are common in geostatistics but susceptible to inconsistent use, even though the literature presents them as nonstationary to avoid some of the emerging problems. Such models are defined in terms of their structure function (variogram), v(h), which tends to infinity as lag h tends to infinity (e.g., [34]). An example is v_I(h) = a h^b with b > 0, in which we note that its formulation with respect to a single scalar argument, h, does not reveal the nonstationarity. Thus, it may be treated as if it was stationary. In that case, from (24) we have ${\gamma }_0=v_{\mathrm{I}}(\infty )=\infty$ and thus the intrinsic process violates the constraint for physical consistency. Moreover, since v_I(h) = γ₀ − c(h) < ∞ for h < ∞, it turns out that c(h) = ∞ for any h and, hence, (21) entails that γ(k) = ∞ for any k. In other words, the infinity of $v_{\mathrm{I}}(\infty )$ is transferred to the entire autocorrelation function and the entire climacogram. In particular, the property γ(k) = ∞ means that the process is non-ergodic (in ergodic processes γ(k) should tend to zero as k tends to infinity; see [27] (p. 429)). Thus, in addition to being physically inconsistent such a treatment of the process is mathematically inconsistent and logically self-contradictory. A consistent way of treatment is to identify the intrinsic process with the (nonstationary) cumulative process X(t) [35], derive the stationary process x(t) and treat the latter further regularly. In that case the variogram is no longer related to the structure function of x(t) but to its climacogram; namely, it is identical to the cumulative climacogram v_I(h) = Γ(h) = γ(h) h².

Another peculiarity in geostatisical analyses is the so-called “nugget effect”, which is also problematic or enigmatic [36]. Namely, this is a discontinuity of the structure function at the origin. An example is v_E(h) = a (h +c)^b, h > 0, with c, b > 0, while v_E(0) = 0. Investigation shows that the “nugget effect” does not necessarily create inconsistency. It is obviously associated with an infinite derivative of the structure function at the origin, i.e., $v^{\prime }\left(0\right)=\infty$. However, the LLD, $v^{\#}\left(h\right)=v^{\prime }\left(h\right)h/v\left(h\right)$ can be finite at the limit as h → 0 (because of the multiplication by h). Therefore, the resulting ${\phi }_{\mathrm{C}}\left(0\right)=1+v^{\#}\left(0\right)/2$ does not necessarily lie out of the “green square”.

Figure 3 illustrates both the “intrinsic” case and the “nugget effect” and provides hints how to avoid both, adopting much better modelling alternatives. The above example of v_E(h) was used with parameters indicated in the figure caption. It is easy to find that a FHK-C model which is both persistent (H = 0.75 > ½) and rough (M = 0.2 < ½) can replace the v_E(h) and be used further.

Conversely, we can imagine that in the model identification and fitting of v_E(h), persistence was regarded as an “intrinsic” characteristic and roughness was interpreted as “nugget effect”. Note that because M < ½, $v^{\prime }\left(h\right)=-c^{\prime }\left(h\right)=\infty$. Thus, a rough process seems like if it exhibited the “nuggest effect”. However, if instead of the standard plot of the variogram, a logarithmic plot was made (also shown in Figure 3 with respect to the upper horizontal axis), then the “nugget” would disappear.

3. Results

3.1. Revisiting Earlier Results

In an earlier study (already mentioned, [22]) on subjects related to the present one, it was suggested that extremization of EPLT in a continuous time representation could determine the entire dependence structure of the process of interest based on simple constraints. The specific premises for EPLT extremization were:

(a)

Lebesgue background measure;

(b)

constrained mean μ and variance γ(1) at a specified (observation) time scale;

(c)

constrained lag-one autocorrelation ρ at the specified time scale;

(d)

an inequality constraint φ_C(t) ≥ φ(t) to ensure physical realism as, naturally, by observing the present and past state of a process, the future entropy is reduced, whereas as t → ∞ conditional and unconditional entropies should tend to be equal, which, however, cannot happen if the entropy production is consistently lower in the conditional than in the unconditional case;

(e)

extremization of entropy production at asymptotic time scales, i.e., t → 0 and t → ∞.

These premises after systematic analyses resulted in two processes extremizing entropy production:

• A Markov process:

  $$c\left(h\right)=\lambda {\mathrm{e}}^{-h/\alpha }, \gamma \left(k\right)=\frac{2\lambda }{k/\alpha }\left(1-\frac{1-{\mathrm{e}}^{-k/\alpha }}{k/\alpha }\right)$$

  (53)

  maximizes entropy production for small times (t → 0) but minimizes it for large times (t → ∞).
• A Hurst-Kolmogorov (HK) process:

  $$\gamma \left(k\right)=\lambda {\left(\alpha /k\right)}^{2-2{\rm H}}$$

  (54)

  maximizes entropy production for large times (t → ∞) but minimizes it for small times (t → 0).

In these definitions α and λ are scale parameters with dimensions of [t] and [x²], respectively. The parameter H, known as the Hurst parameter, determines the global properties of the process with the notable property:

$$H=\phi (\infty )={\phi }_{\mathrm{C}}(\infty )$$

(55)

Here we revisit these results as well as the premises, in light of the theoretical analyses of the previous section and incorporating empirical experience based on various data sets. These allow the following notes:

• The minimal entropy production of the HK process at small times may not be plausible for complex processes; rather we should generally expect that complex processes maximize entropy production at both small and large times (asymptotically as t → 0 and ∞).
• The HK process is characterized by infinite variance ($\gamma \left(0\right)=\infty )$, which, as discussed in the previous section, makes it physically inconsistent.
• The premise φ_C(t) ≥ φ(t) should certainly hold asymptotically (for t → 0 and t → ∞) but not necessarily at intermediate times. A specific example is the process shown in Figure 4h which includes a periodic component where the curve of φ_C(t) intersects that of φ(t) (the latter is not shown in the figure), even though asymptotically the inequality holds. It can thus be conjectured that if a process incorporates a deterministic component (e.g., periodic) while it is treated stochastically without separating the deterministic component, the inequality for EPLT and CEPLT could be violated at intermediate times.
• While the Markov process represents the maximal EPLT as t → 0 (φ(0) = 1), this property is shared by many other processes (see next subsection). Furthermore, in terms of CEPLT, while the Markov process corresponds to φ_C(0) = 3/2, there are processes which have higher CEPLT than the Markov, up to φ_C(0) = 2. These are smoother than the Markov process and have structure function with v^#(0) = ξ^#(0) > 1, where the value 1 corresponds to the Markov process.

Based on these remarks, the Markov process remains a valid candidate of physically consistent processes maximizing entropy production, if only the local asymptotic behaviour is of interest. However, additional constraints to the above premises (a)–(e) (from [22]) are needed to confirm its appropriateness, both on small scales (to enable determination of φ_C(0)) and large scales (to enable determination of φ_C(∞) = φ(∞)).

On the contrary, the HK process should not be regarded as a candidate of general validity, even though it is valid for large scales. Indeed, it is still quite useful as it results in high entropy production at large scales, a property that cannot hold for processes with exponential decrease of autocovariance such as the Markov process. We can thus regard the HK process as a useful mathematical construct which does not appear in nature but can be filtered appropriately to make a physically consistent process. This is the same with white noise (WN), which again is not physically consistent as it involves infinite variance in continuous time (see the specific location and the relationship of the WN and HK processes in the diagram of Figure 2).

It is well known that if a WN process $\underset{\_}{y}\left(t\right)$ is the input to a moving average filter $\underset{\_}{x}\left(t\right)=\frac{1}{2T}{{\displaystyle \int }}_{t-T}^{t+T}\underset{\_}{y}\left(a\right)\mathrm{d}a$, then it produces a physically consistent process $\underset{\_}{x}\left(t\right)$ with finite variance and autocovariance linearly varying in the interval [0, 2T] ([27] (p. 325)). Also, if $\underset{\_}{y}\left(t\right)$ is the input to a linear system corresponding to the linear differential equation ${\underset{\_}{x}}^{\prime }\left(t\right)+\alpha \underset{\_}{x}\left(t\right)=\underset{\_}{y}\left(t\right)$, then the output $\underset{\_}{x}\left(t\right)$ is a Markov process ([27] (p. 326)), physically consistent.

Likewise if in these two cases the input $\underset{\_}{y}\left(t\right)$ is an HK process, then it is easy to see that the filtered output is a physically realistic process with finite variance γ(0), practically unaffected climacogram γ(k) at large scales, with γ^#(∞) = 2H − 2 (as in the original HK process) but highly modified climacogram at small scales, thus having a valid structure (and climacostructure) function with v^#(0) = ξ^#(0) = 2H.

3.2. Specific Processes

Following the theoretical analyses of Section 2 and the remarks of the Section 3.1, we describe here a number of stochastic processes specified through any of their second order functions. All of these processes respect the limitations discussed in Section 2.4 and thus they are consistent mathematically and physically. Some of them correspond to high entropy production, but we also discuss processes with low entropy production. The complete details of these processes, along with graphs illustrating their properties and results of their application, are contained in Appendix A.2.

We start with a general Filtered Hurst-Kolmogorov (FHK) process (which in some cases, e.g., [24], has also been called the Hybrid Hurst-Kolmogorov—HHK—process). In Section 3.1 we briefly discussed two types of linear filters to the HK process, whose asymptotic properties are determined based on a single parameter, H. Here we generalize this result also making the asymptotic behaviour on the left independent from that on the right (cf. [37]) through introducing another parameter denoted as M (in honour of Mandelbrot, following [31]) which is called the smoothness (or fractal) parameter. We avoid specifying the linear filter needed to convert the HK process into the types of the FHK process given as this is not necessary (and in some cases it would be too involved). Rather we specify these types in terms of a convenient expression of the climacogram. The first case is defined through a generalized Cauchy-type climacogram (FHK-C) (see also [24]):

$$\gamma \left(k\right)=\lambda {\left(1+{\left(k/\alpha \right)}^{2M}\right)}^{\frac{H-1}{M}}$$

(56)

Both M and H are dimensionless parameters, and M and 1 − H vary in the interval (0, 1] with M < ½ or > ½ indicating a rough or a smooth process, respectively, and with H < ½ or > ½ indicating an antipersistent or a persistent process, respectively.

The second case is defined through a Dagum-type (FHK-D) climacogram:

$$\gamma \left(k\right)=\lambda \left(1-{\left(1+{\left(k/\alpha \right)}^{2\left(H-1\right)}\right)}^{\frac{M}{H-1}}\right)$$

(57)

and, despite its different expression, its behaviour, as well as the meaning and ranges of parameters are the same as in the Cauchy-type climacogram. For M = 1 − H both result in precisely the same special case:

$$\gamma \left(k\right)=\frac{\lambda }{1+{\left(k/\alpha \right)}^{2\left(1-H\right)}}$$

(58)

The third case is obtained by summing a Cauchy-type climacogram with M = 1 and a Dagum-type climacogram with H = 0. The climacogram of the thus produced model (FHK-CD) is:

$$\gamma (k)={\lambda }_1{(1+{(k/{\alpha }_1)}^2)}^{H-1}+{\lambda }_2(1-{(1+{(k/{\alpha }_2)}^{-2})}^{-{\rm M}})$$

(59)

This is a convenient form, in which the first additive term determines merely the persistence of the process and the second one the smoothness of the process. In addition, it is more flexible and richer than its constituents, as it contains two couples of scale parameters; however, if parsimony is sought, then it can take the same number of parameters as each of the constituents by setting α₁ = α₂ and λ₁ = λ₂ (note that, for dimensional consistency, one λ and one α are minimal parameter requirements).

Even in its most parsimonious form, the FHK in any of the above three variants can cover the entire admissible range, i.e., the entire “green square” in Figure 2. The different patterns in time series generated by different M and H (specifically for the Cauchy-type climacogram) are illustrated in some plots of Figure 4 also in comparison with other models, first of which is the most customary Markov model (Figure 4a). The Markov model is good as a benchmark for comparisons because, as already discussed in Section 2, it is fully neutral (neither rough nor smooth as φ_C(0) = 3/2, and neither antipersistent nor persistent as φ_C(∞) = 1/2).

Each of the panels shows the first fifty terms of time series produced by each of the model implementations at time scales k = 1 and 20. In addition, each panel contains a “stamp” of the specific model represented by the plot of CEPLT, φ_C(k). Additional plots for all second order tools of each of the models are given in Appendix A.2. The time series for these models were generated quite easily using the generic model proposed in [24]; their length is 1024 and this is also the length of the series of coefficients a used for the generating symmetric moving average (SMA) scheme. All calculations are of algebraic type (based on the equations of Section 2 and Appendix A.2) and were performed in Excel spreadsheets without difficulties.

In Figure 4b the CEPLT is close to the absolute maximum both for small and large scales (H = M = 0.97 so as to obtain φ_C(0) = 1.97 ≈ 2 and φ_C(∞) = 0.97 ≈ 1); notable is the very smooth shape at scale 1 and the large departures from the mean (which is 0) at scale 20. On the contrary, in Figure 4c the CEPLT is close to the absolute minimum for all scales (H = M = 0.05, so as to obtain φ_C(0) = 1.05 ≈ 1 and φ_C(∞) = 0.05 ≈ 0—for better visualization it was preferred not to use values of H and M < 0.05). Furthermore, in Figure 4d the CEPLT is close to the absolute maximum for large scales (H = φ_C(∞) = 0.99 ≈ 1) and close to the absolute minimum for small scales (M = 0.01 resulting in φ_C(0) = 1.01 ≈ 1). Finally, in Figure 4f the conditions are opposite to those in 4d, i.e., the CEPLT is equal to the absolute minimum for large scales (H = φ_C(∞) = 0.01 ≈ 0) and to the absolute maximum (2) for small scales (M = 0.99 resulting in φ_C(0) = 1.99 ≈ 2).

The particular case of Figure 4d is close to what is usually called “pink noise” or  ”1⁄f noise”, as the power spectrum has almost constant slope −1 for the entire frequency domain (which is the same in the climacospectrum). This means that using the FHK model we can theoretically represent and practically produce even “pink noise” in a consistent stationary setting without linking it to a nonstationary process [38,39], which actually involves several theoretical inconsistencies [31]. Indeed, as can be seen more thoroughly in the detailed graphs of Figure A5 in Appendix A.2.2, which are for the same application with that of Figure 4d, the small change of slope of from 0.99 to 1.01 is not actually visible, especially in view of the very rough shape of the empirical periodogram, which certainly cannot support differentiation between 0.99 and 1. The FHK model can be used also in other ways to produce “pink noise”, that is, by selecting a very large (small) parameter α so as to expel from our field of vision the asymptotic behaviour on large (small) scales. And we can imagine that in several cases of empirical explorations using observations of natural processes, the observation resolution and length, compared to characteristic scale(s) of the process, are such as to hide the asymptotic behaviour of the process.

We can use this as a trick to obtain virtually constant power spectrum slopes much steeper than −1. This is illustrated Figure A7 of Appendix A.2.2 where the FHK was used with H = M = 0.75 and α = 100. These yield theoretical slopes ${\tilde{\zeta }}^{\#}(\infty )=s^{\#}(\infty )=-\left(1+2M\right)=-2.5$ and ${\tilde{\zeta }}^{\#}\left(0\right)=s^{\#}\left(0\right)=-\left(2H-1\right)=-0.75$. However, the large α does not allow viewing the asymptotic behaviour at low frequencies or large scales and the slope $-2.5$ dominates everywhere. Actually the empirical periodogram estimates an even steeper constant slope, s^# = −2.6. This should not mislead us to conclude that the process is nonstationary because the slope is steeper than −1 (as happens in many studies in the literature). Likewise, in the case that the time series represented observations of a natural process, such a result must not be misinterpreted as evidence that natural processes can lead to slopes constant for all scales and steeper than −1. Clearly here, the process is stationary, the slope −2.5 refers to large frequencies or small scales but, because of the large characteristic scale and the limited observation period (1024 in this example), we cannot see what happens at larger scales. If we saw, then the slope would be −0.75, in accord with the theory.

As discussed, the Markov process is neutral but, by modifying its power spectrum, we can obtain processes which are smooth or antipersistent. Two types of modifications are studied in Appendix A.2, introduced in terms of their power spectrum, i.e.,

$$s\left(w\right)=\frac{4^{n+1}{\left(n!\right)}^2}{\left(2n\right)!}\frac{\lambda \alpha }{{(1+{\left(2\mathsf{\pi }\alpha w\right)}^2)}^{n+1}}$$

(60)

and:

$$s\left(w\right)=\frac{4^{n+1}{\left(n!\right)}^2}{\left(2n\right)!}\frac{\lambda \alpha {\left(2\mathsf{\pi }\alpha w\right)}^{2n}}{{(1+{\left(2\mathsf{\pi }\alpha w\right)}^2)}^{n+1}}$$

(61)

where n = 0, 1, 2…, and the value n = 0 results precisely in a Markov process. For n > 0, (60) corresponds to a smooth process with φ_C(0) = 2 and (61) corresponds to an antipersistent process with φ_C(∞) = 0. The resulting autocovariance function retains the characteristic exponential form of the Markov process but is multiplied by a polynomial function of lag (see Appendix A.2). Therefore, we call these two processes the smooth exponential (SE) and the antipersistent exponential (AE), respectively. In each of the cases, a most extreme value of CEPLT is achieved (maximal for small scales in SE and minimal for large scales in AE). Figure 4g, shows an example of AE for n = 2; the hole in the function φ_C(k) for moderate k is a characteristic of this process.

An interesting process that is simultaneously extremely smooth (φ_C(0) = 2) and extremely antipersistent (φ_C(∞) = 0) is defined by the following power spectrum:

$$s\left(w\right)=\frac{c_n\lambda \alpha {\left(\alpha w\right)}^{2n+1}}{{\mathrm{e}}^{2\pi \alpha w}-1}$$

(62)

where c_n is a normalizing constant making the area of the power spectrum equal to ${\gamma }_0=\lambda$. This process, which we call generalized Planck (GP) is obtained by a generalization of Planck’s law of black-body radiation, whereas it is identical to this law if n = 1. The detailed equations of this process are also given in Appendix A.2. A characteristic plot of time series generated from this process is shown in Figure 4e, and is almost indistinguishable from Figure 4f which, as already discussed, was produced as an approximation by FHK.

A final case examined in the Appendix A.2 is a harmonic oscillation, which is very easily modelled as a deterministic process but here it is treated as a stochastic process. Of course stochastic treatment is not advisable in this case, but often such periodic behaviours appear as components of stochastic processes. Obviously the second-order characteristics of such processes are affected by periodic components and therefore we need to know which equations should be superimposed in those of the pure stochastic process (see also [40,41]). As an example of such a case, Figure 4h shows the behaviour of a process defined as the average of a FHK with H = M = 0.8 and a harmonic oscillation. It is difficult to identify the presence of the oscillation from visual inspection of the time series, but the detailed graph of any of the second order characteristics, plotted in Appendix A.2., captures the periodic behaviour.

3.3. Comparison of Asymptotic Properties

Most of the models examined in the previous subsection can only reproduce specific values of asymptotic properties. An exception is the FHK model which is the most powerful as it can perform in the entire admissible domain of asymptotic properties. A visual comparison of all models examined, it terms of the asymptotic values of the different second order characteristics is made in Figure 5, while Figure 6 provides an envelope for all models, with classification of the ranges in terms of smoothness and persistence.

Further comparative information about which model can perform for the most extremal cases is provided in

Table 4. Models that can perform in each of the specified extremal or neutral cases of smoothness and persistence.

Persistence	Smoothness
	Maximal Roughness ${\mathit{\phi }}_{\mathbf{C}}\left(0\right)=1$	Neutral Smoothness ${\mathit{\phi }}_{\mathbf{C}}\left(0\right)=3/2$	Maximal Smoothness ${\mathit{\phi }}_{\mathbf{C}}\left(0\right)=2$
Maximal Persistence, ${\phi }_{\mathrm{C}}(\infty )=1$	~FHK	~FHK	~FHK
Neutral Persistence ${\phi }_{\mathrm{C}}(\infty )=1/2$	~FHK	Markov, FHK	SE, GP, FHK
Maximal Antipersistence ${\phi }_{\mathrm{C}}(\infty )=0$	~FHK	AE, FHK	GP, FHK

Note: the symbol ‘~’ indicates that the model can perform in close approximation but not precisely.

. These figures and the table can be useful for the model selection based on data.

3.4. Real World Case Study

Empirical testing of the theoretical analyses presented here needs fine resolution of measurements and long time series, in order to reliably determine the asymptotic properties from data. Geophysical time series usually do not satisfy either of these two conditions. However, laboratory measurements of turbulent velocity can meet the conditions. Indeed, in laboratory experiments at sampling intervals of μs, very large samples can be formed which can enable viewing the asymptotic behaviour. Here we use grid data of nearly isotropic turbulence from the Corrsin Wind Tunnel at a high-Reynolds-number [42], which were made available on the Internet by the Johns Hopkins University. This dataset consists of 40 time series with n = 36 × 10⁶ data points of wind velocity along the flow direction and an equal number of time series of cross-stream velocity, all measured at a sampling time interval D = 25 μs by X-wire probes placed downstream of the grid. Here we use part of the data, namely the series of wind velocity along the flow direction at the first of the probes (first column of files in http://pages.jh.edu/~cmeneve1/datasets/Activegrid/M20/H1/). More data and analyses have been used by Dimitriadis and Koutsoyiannis [43] at a somewhat different context.

Characteristic plots of the turbulent velocity time series are shown in the Figure 7a for various time scales. The large persistence of turbulence is evident from the large variability at the largest time scales. Impressively, an FHK-CD model with five parameters describes well the data at the entire range of available time scales which spans almost seven orders of magnitude.

The fitted model parameters are: H = 5/6 = 0.83, M = 1/3 = 0.33, a₁= a₂ = 2170 D = 54 ms, λ₁ = 1.13 m²/s², λ₂ = 2.39 m²/s². These confirm the intense persistence and a small roughness, which is reflected in a climacospectrum slope of −5/3 for large frequencies (the neutral would be −2), in accord with Kolmogorov’s theory.

4. Discussion and Conclusions

The question whether the probabilistic (information) entropy incorporates, as a special case, thermodynamic entropy or the two are different concepts, is still debated [44,45]. If we adopt the former proposition of the dilemma, as we have done here, the requirement emerges that entropy production should be formulated in a stochastic context, while current attempts mostly use the classical thermodynamic entropy definition as in heat transfer. This is not an easy task and certainly requires exploration of a broad spectrum of alternatives for a definition of the concept of entropy production. Exploration of extremum principles for entropy production in a stochastic framework is also crucial. The two versions proposed here, entropy production in logarithmic time, unconditionally (EPLT) or conditionally on the past and present having been observed (CEPLT), may have some usefulness and potential as explanatory and modelling tools. The findings of this study include the following:

• Definition of entropy production with respect to logarithmic time has some attractive characteristics and is consistent with the notion of the time arrow.
• The asymptotic values of entropy production for time zero and infinity seem to be crucial for its extremization and define important features of a process.
• In particular, the asymptotic value of CEPLT for zero time defines the roughness (fractality) of the stochastic process while the common asymptotic value of EPLT and CEPLT for infinite time defines the (long-term) persistence of the process and equals the Hurst parameter.
• Both ELPT and CEPLT are finite and constrained within specific theoretically justified bounds. The scaling laws of all second order characteristics of stochastic processes are a result of these finite values and are also subject to bounds (with few exceptions fully described). The proofs of existence of these bounds offer the means to fight common misunderstanding, misuse, false reporting and inconsistent estimation of characteristics of geophysical processes, as well as to develop consistent estimation algorithms.
• The climacogram and its derivative tools introduced here, namely, the differenced climacogram and the climacospectrum are very powerful and support the estimation of entropy production, in either of its forms, from data series.
• The broad family of models for stochastic processes investigated here covers the entire admissible range of asymptotic behaviours and offers the means for modelling any type of processes, either using a single model or a combination of more than one model.
• The real-world application presented with the extraordinary long time series of turbulent velocity shows how a parsimonious stochastic model can be identified and fitted using the tools developed.

Measurements of diverse turbulent phenomena can potentially provide equally long time series with fine resolution, shaping a broader empirical basis for future work on identifying natural behaviours, specifically on larger spatial and temporal scales, and further testing the adequacy of the models. Characteristics of order higher than second, along with the effects of non-Gaussianity on entropy production, are also interesting to examine in light of the developments for second-order ones. A recent study [43] has already provided interesting results with respect to high-order moments. Connections of microscale turbulence and macroscale atmospheric phenomena are also a broad field for future studies. Further applications, also extending similar recent works in hydrology, geophysics and ecosystems [30,40,41], would enrich our knowledge on natural behaviours.