Internal Energy, Fundamental Thermodynamic Relation, and Gibbs’ Ensemble Theory as Emergent Laws of Statistical Counting

Abstract

Statistical counting ad infinitum is the holographic observable to a statistical dynamics with finite states under independent and identically distributed N sampling. Entropy provides the infinitesimal probability for an observed empirical frequency $\widehat{\mathit{\nu }}$ with respect to a probability prior $\mathbf{p}$, when $\widehat{\mathit{\nu }}\ne \mathbf{p}$ as $N\to \infty$. Following Callen’s postulate and through Legendre–Fenchel transform, without help from mechanics, we show that an internal energy $\mathbf{u}$ emerges; it provides a linear representation of real-valued observables with full or partial information. Gibbs’ fundamental thermodynamic relation and theory of ensembles follow mathematically. $\mathbf{u}$ is to $\widehat{\mathit{\nu }}$ what chemical potential $\mu$ is to particle number N in Gibbs’ chemical thermodynamics, what $\beta =T^{-1}$ is to internal energy U in classical thermodynamics, and what $\omega$ is to t in Fourier analysis.

1. Introduction

It is a pleasure to be a part of this celebration for Signe Kjelstrup. She has made significant contributions to nonequilibrium thermodynamics, in both theory and applications that include electrochemistry, transport in heterogeneous media, and T. L. Hill’s small systems [1,2,3]. In this work we extend Gibbs’ and Hill’s approach to equilibrium thermodynamics [4] and show that the new logical path via the “crucial step” advocated in [5] is in fact a consequence of a limit theorem [6] in the mathematical theory of probability [7,8]. The results perfectly fit P. W. Anderson’s notion of emergent phenomenon [9].

Sometimes a mathematical transform can provide a fundamental concept beyond just being a technique for solving a problem, and through which a new representation of a natural phenomenon emerges. A case in point is the Fourier transform (FT) that leads to the theory of harmonics in music instruments [10] and the very concept of optical spectrum. FT represents a function of time $f\left(t\right)$ in terms of $\tilde{f}\left(\omega \right)$, where $\omega$ is introduced as a novel notion, the temporal frequency of a sinusoidal oscillatory component in time [11]. The solutions to a large class of problems in differential calculus involving t can be very efficiently expressed through FT.

We show in the present paper that the fundamental notion of internal energy first appeared in the theory of thermodynamics in the 19th century, collectively developed by J. R. von Mayer, W. Rankine, R. Clausius, and W. Thomson among many others [12], is a concept that can be understood, and generalized, in statistical counting. The transformation in question is the Legendre–Fenchel transform (LFT) [13,14], a more refined mathematical formulation of the traditional Legendre transform [15].

When a simple statistical analysis is carried out on a set of data, correlated or not, it is usually supposed that they are from an identical probability distribution. One of the best understood systems that exhibit an invariant probability is the ergodic dynamical system [16]. The ergodic theory of classical Hamiltonian dynamics has been an intense research area in both physics and mathematics for more than a century [17,18]. Even when the data are from seemingly different “objects”, say different individuals within a biological species, it is understood that an ergodic mating or mutational process is behind the statistical practice; and the conclusions drawn are most meaningful in this regard. Such an ergodic stochastic dynamic perspective has transformed cell biology through the notion of phenotypic switching in recent years [19].

2. Energetic Theory of Statistical Counting

Let us consider the repeated statistical samples ad infinitum of a system with finite state space $\mathcal{S}=\{0,1,\cdots ,n\}$. In the present work we shall restrict our discussion for independent and identically distributed (i.i.d.) samples. More general sampling of Markov data will be published elsewhere. The number counting $\mathit{\nu }=({\nu }_0,\cdots ,{\nu }_n)$ with ${\nu }_0+\cdots +{\nu }_n=N$ and counting frequency $\widehat{\mathit{\nu }}=\mathit{\nu }/N$, not to be confused with the $\omega$ in FT above, has a homogeneous degree 1 neg-entropy function with respect to a given probability prior $\mathbf{p}=(p_0,\cdots ,p_n)$ [8,20]:

$$\mathrm{\Phi }\left(\mathit{\nu }\right)=\sum _{i=0}^n{\nu }_{i}ln\left({\displaystyle \frac{{\nu }_i}{p_i{\sum }_{j=0}^n{\nu }_j}}\right).$$

(1)

The Appendix A provides the mathematical origin of the non-negative $\mathrm{\Phi }\left(\mathit{\nu }\right)$ as a result of statistical counting. In information theory, it is interpreted as the “surprise” in observing the $\mathit{\nu }$ under the assumption $\mathbf{p}$ [21,22]. It is a double-edged sword which tells the rareness of $\mathit{\nu }$ (or $\widehat{\mathit{\nu }}$) with respect to $\mathbf{p}$ or erroneous model $\mathbf{p}$ with respect to empirical $\mathit{\nu }$. The Kolmogorov probability $\mathbf{p}$ has two rather different roles in statistical inference and in statistical physics. In the former it has been identified as only half to the other “half of probability theory as it is needed in current applications—the principles for assigning probabilities by logical analysis of incomplete information—is not present at all in the Kolmogorov system” [20].

The application of modern probability to statistical physics involves the limit of sample ad infinitum represented by $N\to \infty$ [6]. In this case, the probability $\mathbf{p}$ is for all the systems with the same state space $\mathcal{S}$; it is not meant to be realistic for any particular system. It simply provides a “metric” under which each and every particular system has its own representation in terms of its complete information, $\mathit{\nu }$. The $\mathrm{\Phi }\left(\mathit{\nu }\right)$ is introduced to further gauge the differences among systems with different $\mathit{\nu }$’s on the same $\mathcal{S}$; it becomes a “theory of everything.” Because the $N\to \infty$ limit, there are no uncertainties in $\widehat{\mathit{\nu }}$; it is a definitive characterization of an i.i.d. statistical distribution with state space $\mathcal{S}$.

Therefore, statistical inference is about the mathematical model of a particular system, and statistical physics is about the mathematical representation of all systems with the same $\mathcal{S}$ under the supposition of i.i.d. data ad infinitum. The entropy in (1) is an emergent characterization in the limit of $N\to \infty$, with the starting point in terms of generative models [9]. It provides the relationship between $\mathit{\nu }$ and $\mathbf{p}$ in the sampling process. It is an Eulerian degree 1 homogeneous function of $\mathit{\nu }$: $\mathrm{\Phi }\left(\lambda \mathit{\nu }\right)=\lambda \mathrm{\Phi }\left(\mathit{\nu }\right)$. This fits naturally to the fundamental thermodynamic postulate formulated by H. B. Callen [23]. The LFT of $\mathrm{\Phi }$ as a function of the normalized $\widehat{\mathit{\nu }}$ then yields [13,14,24]:

$$\Psi \left(\mathbf{u}\right)=\underset{\widehat{\mathit{\nu }}}{inf}\left\{\sum _{i=0}^n{\widehat{\nu }}_i{u}_i+\mathrm{\Phi }\left(\widehat{\mathit{\nu }}\right)\right\}=-ln\sum _{i=0}^n{p}_i{e}^{-u_i},$$

(2a)

with corresponding optimal ${\widehat{\mathit{\nu }}}^{*}\left(\mathbf{u}\right)$

$${\left({\widehat{\mathit{\nu }}}^{*}\right)}_i={\displaystyle \frac{p_i{e}^{-u_i}}{{\sum }_{\ell =0}{p}_{\ell }{e}^{-u_{\ell }}}},\mathrm{and}{u}_i=-\left({\displaystyle \frac{\partial \mathrm{\Phi }\left({\widehat{\mathit{\nu }}}^{*}\right)}{\partial {\widehat{\nu }}_i}}\right).$$

(2b)

Note that the second equation in (2b) is obtained when one uses calculus to solve the infimum in (2a); this recovers the traditional Legendre transform. Normalizing $\mathit{\nu }$ to $\widehat{\mathit{\nu }}$ induces a gauge freedom in (2), an arbitrary additive constant to $u_i$. In statistical thermodynamics, the conjugate variable $u_k$ introduced in Equation (2) has been interpreted as the internal energy of the state k, in $k_{B}T$ unit [25]; then $\widehat{\mathit{\nu }}·\mathbf{u}$ is the mean internal energy of “the statistical system”.

In a real-world laboratory working on a particular system, the $\mathit{\nu }$ tends to infinity as $N\to \infty$ but $\widehat{\mathit{\nu }}$ converges to the intrinsic property of the statistical system. In statistical inference, the assumed $\mathbf{p}$, as a prior, then is expected to be replaced by the observed, real, posterior $\widehat{\mathit{\nu }}$ according to conditional probability and/or Bayesian statistical logic [24,25]. This concludes the statistical investigation of the particular system with respect to the type of observations. The neg-entropy function in (1) actually provides a meta-statistical theory for all possible observed $\widehat{\mathit{\nu }}$, assessing their respective infinitesimal probability (rate) with respect to the prior $\mathbf{p}$ (see Appendix A).

3. Maximization of Entropy $-\mathrm{\Phi }$ Under Constraint by Empirical Mean Value

However, The complete counting for the entire state space $\mathcal{S}$ in terms of empirical frequencies $\widehat{\mathit{\nu }}$ is only a gedankenexperiment. The significance of Gibbs’ ensemble theory is in dealing with observations from a small set of real-valued observables $g_1\left(i\right),g_2\left(i\right),\cdots ,g_J\left(i\right)$, where $i\in \mathcal{S}$ but $J\ll n$. These g’s are random variables on the state space $\mathcal{S}$. In fact, their empirical mean values are linear combinations of the $\widehat{\mathit{\nu }}$:

$$x_j=\sum _{i=0}^n{\widehat{\nu }}_i{g}_j\left(i\right).$$

(3)

To fix mathematical notations, we append $g_0\left(i\right)=1$ and $x_0=1$, which represent the fact that $\widehat{\mathit{\nu }}$ is always normalized, and denote $(n+1)\times (J+1)$ matrix ${\mathbf{G}}_J$ with elements

$${\left({\mathbf{G}}_J\right)}_{ij}=\left\{\begin{array}{ccc}1& & j=0,\hfill \\ g_j\left(i\right)& & j=1,\cdots ,J.\hfill \end{array}\right.$$

(4)

Equation (3) shows that if all the g’s are linearly independent and $J=n$, then one can solve the normalized $\widehat{\mathit{\nu }}$ uniquely from each set of x’s: $\widehat{\mathit{\nu }}=\mathbf{x}{\mathbf{G}}_n^{-1}$. We refer to such a set of observables as holographic with full information. In the following discussion, we shall always imagine the $(g_1,\cdots ,g_J)$ as the first J component of a holographic observable $(g_0, g_1,\cdots ,g_n)$. When $J<n$, there is missing information [20,22,25].

With a set of observed values ${\mathbf{x}}^{\prime }=(x_1,\cdots ,x_J)$ in hand where $J<n$, the maximum entropy principle (MEP) from classical thermodynamics [23] and the contraction principle from the mathematical theory of probability [8] assert that the most probable ${\widehat{\mathit{\nu }}}^{*}$ that is consistent with the set of ${\mathbf{x}}^{\prime }$ corresponds to minimum neg-entropy:

$${\widehat{\mathit{\nu }}}^{*}=arg\underset{\widehat{\mathit{\nu }}}{inf}\left\{\mathrm{\Phi }\left(\widehat{\mathit{\nu }}\right)|\widehat{\mathit{\nu }}{\mathbf{G}}_J={\mathbf{x}}^{\prime }\right\}.$$

(5)

The entire Gibbs’ ensemble theory arises in solving the mathematical problem posed in Equation (5) through LFT. See Appendix A for its origin.

Entropy functions for different observables are different. First, for invertible ${\mathbf{G}}_n$, one has the entropy function for the holographic observable $\mathbf{x}=(1,x_1,\cdots ,x_n)$:

$${\mathrm{\Phi }}_{\mathbf{x}}\left(\mathbf{x}\right)\equiv \mathrm{\Phi }\left(\mathbf{x}{\mathbf{G}}_n^{-1}\right).$$

(6)

This is simply a change in the independent variables from $\mathit{\nu }$ to $\mathbf{x}$. Then in terms of this entropy function ${\mathrm{\Phi }}_{\mathbf{x}}$, (5) becomes

$$\begin{array}{ccc}\hfill \phi \left({\mathbf{x}}^{\prime }\right)& =& \underset{\widehat{\mathit{\nu }}}{inf}\left\{\mathrm{\Phi }\left(\widehat{\mathit{\nu }}\right)|\widehat{\mathit{\nu }}{\mathbf{G}}_J={\mathbf{x}}^{\prime }\right\}\hfill \\ & =& \underset{x_{J+1},\cdots ,x_n}{inf}\left\{{\mathrm{\Phi }}_{\mathbf{x}}\left(\mathbf{x}\right)|x_1=x_1^{\prime },\cdots ,x_J=x_J^{\prime }\right\}.\hfill \end{array}$$

(7)

Intimately related to the generating function of a probability distribution, the LFT provides a powerful mathematical transform of the entropy functions $\mathrm{\Phi }\left(\mathit{\nu }\right)$, ${\mathrm{\Phi }}_{\mathbf{x}}\left(\mathbf{x}\right)$, and $\phi \left({\mathbf{x}}^{\prime }\right)$ in terms of their conjugates in the energy representation: Parallel to the $\Psi \left(\mathbf{u}\right)$ in (2) are,

$$\begin{array}{ccc}\hfill {\Psi }_{\mathbf{y}}\left(\mathbf{y}\right)& =& \underset{\mathbf{x}}{inf}\left\{\sum _{i=1}^n{x}_i{y}_i+{\mathrm{\Phi }}_{\mathbf{x}}\left(\mathbf{x}\right)\right\},\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill \psi \left({\mathbf{y}}^{\prime }\right)& =& \underset{{\mathbf{x}}^{\prime }}{inf}\left\{\sum _{j=1}^J{x}_j^{\prime }{y}_j^{\prime }+\phi \left({\mathbf{x}}^{\prime }\right)\right\}.\hfill \end{array}$$

(9)

These psi’s are now related through linear transformation:

$${\Psi }_{\mathbf{y}}\left(\mathbf{y}\right)=\Psi \left({\mathbf{G}}_n\mathbf{y}\right),$$

(10)

and projection:

$$\begin{array}{ccc}\hfill \psi \left({\mathbf{y}}^{\prime }\right)& =& {\Psi }_{\mathbf{y}}\left(y_1^{\prime },\cdots ,y_J^{\prime },0,\cdots ,0\right)=\Psi \left({\mathbf{G}}_J{\mathbf{y}}^{\prime }\right)\hfill \end{array}$$

(11a)

$$\begin{array}{ccc}& =& -ln\sum _{i=0}^n{p}_{i}exp\left[-\sum _{j=1}{g}_j\left(i\right)y_j^{\prime }\right].\hfill \end{array}$$

(11b)

And finally, since $\psi$ is convex, the inverse LFT yields

$$\begin{array}{ccc}\hfill -\phi \left({\mathbf{x}}^{\prime }\right)& =& \underset{{\mathbf{y}}^{\prime }}{inf}\left\{\sum _{i=1}^J{x}_i^{\prime }{y}_i^{\prime }-\psi \left({\mathbf{y}}^{\prime }\right)\right\}\hfill \\ & =& \left\{\begin{array}{ccc}-\phi & =& {\mathbf{y}}^{\prime }·\nabla \psi \left({\mathbf{y}}^{\prime }\right)-\psi \left({\mathbf{y}}^{\prime }\right)\hfill \\ {\mathbf{x}}^{\prime }& =& \nabla \psi \left({\mathbf{y}}^{\prime }\right)\hfill \end{array}\right.\hfill \end{array}$$

(12)

The optimization in (5) is completely “solved” in closed form, through LFT and its inverse, as a parametric function in terms of ${\mathbf{y}}^{\prime }$ given in (12).

The equation $-\phi ={\mathbf{y}}^{\prime }·\nabla \psi -\psi$ in (12) should be recognized as a generalization of the celebrated “entropy = mean internal energy − free energy”, where

$${\left(\nabla \psi \right)}_k={\displaystyle \frac{{\displaystyle \sum _{i=0}^n{g}_k\left(i\right)p_{i}exp\sum _{j=1}^J{g}_j\left(i\right)y_j}}{{\displaystyle \sum _{i=0}^n{p}_{i}exp\sum _{j=1}^J{g}_j\left(i\right)y_j}}}$$

(13)

is the mean value of $g_k$ following Equation (11b), whose conjugate variable is $y_k^{\prime }$. The identification of $\mathbf{u}={\mathbf{G}}_J{\mathbf{y}}^{\prime }$ in (11a) with the first law of thermodynamics as formulated by Gibbs seems natural.

The $y_{J+1}=\cdots =y_n=0$ in (11a) has a very clear thermodynamic interpretation: Since the conjugate variable $\mathbf{y}$ are the partial derivatives of the entropy function ${\mathrm{\Phi }}_{\mathbf{x}}$ with respect to $\mathbf{x}$, finding x’s with maximum entropy in Equation (7) is simply setting the corresponding $y=0$, e.g., letting the entropic force be zero. For each independent observable $g_j$, $y_j$ is its “custom-designed” conjugate force and $y_j\times \mathrm{d}{x}_j$ contributes a term to the internal energy as the “thermodynamic work” associated with $g_j$: The internal energy $\mathbf{u}$ is a highly flexible, adaptive representation of the $\widehat{\mathit{\nu }}$. When $J=n$, $\mathbf{u}={\mathbf{G}}_n\mathbf{y}$ and Equation (10) provides a complete “detailing” of the internal energy in terms of a set of holographic observables. MEP is for missing information [20].

4. Gibbs Distribution and Linear Algebraic Representation

There is a geometric picture associated with the above “thermodynamic analysis”. As we have stated, counting frequency ad infinitum $\widehat{\mathit{\nu }}$ is a fundamental, intrinsic property of an ergodic dynamical system. The space of all possible frequency distributions $\widehat{\mathit{\nu }}$, with ${\widehat{\nu }}_0+\cdots +{\widehat{\nu }}_n=1$, is a n-dimensional hyper-plane in the positive quadrant of ${\mathbb{R}}^{n+1}$, known as a probability simplex ${\mathcal{M}}_n$. For a given set of observables $(g_1,\cdots ,g_J)$, the ${\mathcal{M}}_n$ is foliated by $\widehat{\mathit{\nu }}{\mathbf{G}}_j={\mathbf{x}}^{\prime }$ with different ${\mathbf{x}}^{\prime }$. On each leave of the foliation, there is the most probable ${\mathit{\nu }}^{*}\left({\mathbf{x}}^{\prime }\right)$, which is located at the tangent point between the $(n-J)$-dimensional leave and a $(n-1)$-dimensional level set of the $\mathrm{\Phi }\left(\widehat{\mathit{\nu }}\right)$ function. At this point, ${\nabla }_{\mathit{\nu }}\mathrm{\Phi }\left(\widehat{\mathit{\nu }}\right)=-\mathbf{u}\left(\widehat{\mathit{\nu }}\right)$ is the normal vector to the ${\mathbf{x}}^{\prime }$-leave in ${\mathbb{R}}^{n+1}$, and $\nabla \phi \left({\mathbf{x}}^{\prime }\right)=-{\mathbf{y}}^{\prime }$ is its projection onto the J-manifold of ${\mathbf{x}}^{\prime }$:

$${\mathit{\nu }}^{*}\left({\mathbf{x}}^{\prime }\right)=\left\{\begin{array}{c}{\displaystyle {\nu }_i^{*}=\frac{1}{Z\left({\mathbf{y}}^{\prime }\right)}{p}_{i}exp\left[-\sum _{j=1}^J{g}_j\left(i\right)y_j^{\prime }\right]}\hfill \\ {\displaystyle x_j^{\prime }=\frac{1}{Z\left({\mathbf{y}}^{\prime }\right)}\sum _{i=0}^n{g}_j\left(i\right)p_{i}exp\left[-\sum _{j=1}^J{g}_j\left(i\right)y_j^{\prime }\right]}\hfill \end{array}\right.$$

(14a)

in which

$$Z\left({\mathbf{y}}^{\prime }\right)=\sum _{i=0}^n{p}_{i}exp\left[-\sum _{i=1}^J{g}_j\left(i\right)y_j^{\prime }\right].$$

(14b)

All the other points on the same ${\mathbf{x}}^{\prime }$-leave are no longer relevant: they are deemed statistically impossible under the prior $\mathbf{p}$ and observed ${\mathbf{x}}^{\prime }$. The foliation therefore represents a partition of the ${\mathcal{M}}_n$ into macro- and micro-worlds: Transversing between different ${\mathbf{x}}^{\prime }$-leaves are macroscopic thermodynamic processes that follow the ${\mathbf{y}}^{\prime }\left({\mathbf{x}}^{\prime }\right)$. According to the logic of Bayesian statistics, one should use the most suitable probability frequency distribution ${\widehat{\nu }}^{*}\left({\mathbf{x}}^{\prime }\right)$ to update the prior $\mathbf{p}$ for the particular system with observed ${\mathbf{x}}^{\prime }$. The microscopic world is still random, due to missing information, but its prior is now updated. This is Gibbs’ statistical ensemble.

With a given set of $(g_1,\cdots ,g_J)$, the ${\mathcal{M}}_n$ is collapsed into J-manifold in ${\mathbb{R}}^{n+1}$, which is parametrized by the ${\mathbf{x}}^{\prime }$, or equivalently ${\mathbf{y}}^{\prime }$. There is no uncertainty in this “macroscopic” description. For a different set of g’s and $J^{\prime }$, there will be a different $J^{\prime }$-manifold. It will be desirable to treat different g’s through transformations. We note that even though ${\mathcal{M}}_n$ is a “plane” in ${\mathbb{R}}^{n+1}$, it is not a linear Euclidean space since for any $c\ne 1$, $c\widehat{\mathit{\nu }}\notin {\mathcal{M}}_n$, and neither are the ${\mathbf{x}}^{\prime }$-leaves. They are affine manifolds [26]. The locating of ${\widehat{\mathit{\nu }}}^{*}\left({\mathbf{x}}^{\prime }\right)$ is a highly nonlinear procedure in the space of energies.

The LFT, in terms $\Psi \left(\mathbf{u}\right)$, ${\Psi }_{\mathbf{y}}\left(\mathbf{y}\right)$ and $\psi \left({\mathbf{y}}^{\prime }\right)$, etc., enters as a powerful algebraic linear representation of the MEP procedure. The “collapse” of a holographic $\mathbf{y}$ to ${\mathbf{y}}^{\prime }$ with missing information means simply neglecting all the extra dimensions: $y_{J+1}=\cdots =y_n=0$. This is because due to the convexity of $\mathrm{\Phi }\left(\widehat{\mathit{\nu }}\right)$, there is a one-to-one relation between $\widehat{\mathit{\nu }}$ and $\mathbf{u}=-\nabla \mathrm{\Phi }$ under a proper gauge fixing. And since the constrains to MEP in (5) are all linear due to the nature of observables being random variables, each g determines a 1-dimensional linear subspace in the space of $\mathbf{u}$.

5. Generalized Clausius Inequality

A combination of Equations (9) and (12a) yields a Clausius’ inequality-like relation:

$$\phi \left({\mathbf{x}}^{\prime }\right)+{\mathbf{x}}^{\prime }·{\mathbf{y}}^{\prime }-\psi \left({\mathbf{y}}^{\prime }\right)\ge 0.$$

(15)

The thermodynamics equilibrium is between the observed mean value ${\mathbf{x}}^{\prime }$ and its conjugate “force” ${\mathbf{y}}^{\prime }$. When the equality holds, there is a relation between ${\mathbf{x}}^{\prime }$ and ${\mathbf{y}}^{\prime }$ which should be identified as a “the equation of state”, with ${\nabla }_{\mathbf{x}}\phi =-{\mathbf{y}}^{\prime }$ and ${\nabla }_{\mathbf{y}}\psi ={\mathbf{x}}^{\prime }$. When the ${\mathbf{x}}^{\prime }\ne {\nabla }_{\mathbf{y}}\psi \left(\mathbf{y}\right)$, the difference ${\mathbf{x}}^{\prime }·{\mathbf{y}}^{\prime }-\psi \left({\mathbf{y}}^{\prime }\right)$ can be interpreted as the nonequilibrium heat and $\phi$ again as the entropy; then, the inequality in (15) becomes the Clausius’ inequality.

6. Generalized Gibbs–Duhem Equation

The celebrated Gibbs–Duhem equation in classical thermodynamics is a consequence of the entropy being a Eulerian degree 1 homogeneous function. Thus, for the $\mathrm{\Phi }\left(\mathit{\nu }\right)$ in (1), we have

$$\begin{array}{cc}& {\displaystyle \mathrm{\Phi }\left(\mathit{\nu }\right)=\sum _{i=0}^n{\nu }_i\left(\frac{\partial \mathrm{\Phi }}{\partial {\nu }_i}\right),\sum _{i=0}^n{\nu }_i\left(\frac{{\partial }^2\mathrm{\Phi }}{\partial {\nu }_i\partial {\nu }_j}\right)=0,}\\ & {\displaystyle \sum _{j=0}^n\mathrm{d}{\nu }_j\sum _{i=0}^n{\nu }_i\left(\frac{{\partial }^2\mathrm{\Phi }}{\partial {\nu }_i\partial {\nu }_j}\right)=\sum _{i=0}^n{\nu }_i\sum _{j=0}^n\left(\frac{\partial u_i}{\partial {\nu }_j}\right)\mathrm{d}{\nu }_j=0,}\\ & {\displaystyle \mathrm{that}\mathrm{is},\sum _{i=0}^n{\nu }_i\mathrm{d}{u}_i=0,}\end{array}$$

(16)

in which we have used (2b). We identify (16) as a generalized Gibbs–Duhem equation.

7. Conclusions

The mathematical theory of probability deals with a set of elementary events $\mathcal{S}$, on which the probability $\mathbf{p}$ and random variables g’s are introduced. Applying this mathematics to the real world, each ergodic dynamical system with state space $\mathcal{S}$ has its own unique steady-state probability distribution which can be obtained as the $\widehat{\mathit{\nu }}$ from i.i.d. sampling ad infinitum.

Our present theory is to statistical inference obtaining particular $\widehat{\mathit{\nu }}$’s what dynamics is to kinematics in classical mechanics [27]. The entropy function in (1) arises in this context as a measure of the quantitative relationship between the assumed, “hypothesis” ($\mathbf{p}$) and the observed “data” ($\mathit{\nu }$ and $\widehat{\mathit{\nu }}$), as “missing information” or “surprise” [21,22]. Motivated by the analogy to the Fourier analysis, our generalized Gibbs’ theory suggests that the notion of thermo-energetics is a powerful mathematical transformation of the statistical description; $\widehat{\mathit{\nu }}$ and $\mathbf{u}$ are simply two representations of a same physical reality, the former being statistical while the latter thermo-energetic. $\mathbf{u}$ is to $\widehat{\mathit{\nu }}$ what chemical potential $\mu$ is to particle number N in J. W. Gibbs’ chemical thermodynamics. With a fixed $\mathbf{p}$, the theory of probability [8] revealed a powerful, dual energetic representation for various different systems, with the same state space $\mathcal{S}$, in terms of their respective internal energy functions $\mathbf{u}$ [25]. This fundamental duality between counting frequency and internal energy of course has been recognized by L. Boltzmann already in 1880s, when he was developing the statistical mechanics as a foundation of classical thermodynamics under the principle of equal probability a priori. The present work shows that while the probability and statistics are fundamental as the foundation of thermodynamics, mechanics is not necessary. A similar conclusion was reached in the 1925 thesis of L. Szilard [28,29].

For sufficiently large N, the probability of observing a particular $\widehat{\mathit{\nu }}$ is asymptotically zero except $\widehat{\mathit{\nu }}=\mathbf{p}$. The significance of $\mathrm{\Phi }\left(\widehat{\mathit{\nu }};\mathbf{p}\right)$ is to provide a “high-resolution magnifying glass” for the asymptotically small

$$exp\left\{-N\mathrm{\Phi }\left(\widehat{\mathit{\nu }};\mathbf{p}\right)\right\}.$$

(17)

This is known as the large deviations rate function in the modern theory of probability [8]. The entropy $-\mathrm{\Phi }(\widehat{\mathit{\nu }},\mathbf{p})$ is a function of both $\widehat{\mathit{\nu }}$ and $\mathbf{p}$, $\mathrm{\Phi }(\widehat{\mathit{\nu }};\mathbf{p})\ge 0$ and $\mathrm{\Phi }(\mathbf{p};\mathbf{p})=0$. For a given $\mathbf{p}$, it views each possible $\widehat{\mathit{\nu }}$ from a real system as a part of an entire class of systems under a common $\mathbf{p}$, a meta-statistics. If one chooses the true steady-state probability $\mathit{\pi }$ of a particular system to replace $\mathbf{p}$, then Equation (17) gives the probability distribution of the uncertainties in the measurement $\widehat{\mathit{\nu }}$ from N samples. The second-order Taylor expansion near $\mathit{\pi }$,

$$e^{-N\mathrm{\Phi }(\widehat{\mathit{\nu }};\mathit{\pi })}\simeq exp\left[-N\sum _{i,j=0}^n{\displaystyle \frac{({\widehat{\nu }}_i-{\pi }_i)({\delta }_{ij}-{\pi }_i)({\widehat{\nu }}_j-{\pi }_j)}{{\pi }_i}}\right],$$

is the central limit theorem for the statistics of counting frequency $\widehat{\mathit{\nu }}$, with $\mathrm{Var}\left[{\widehat{\nu }}_i\right]={\pi }_i(1-{\pi }_i)/N$ and $\mathrm{Cov}[{\widehat{\nu }}_i,{\widehat{\nu }}_j]=-{\pi }_i{\pi }_j/N$. This is not the fluctuations within the $\mathit{\pi }$ of the system itself. Gibbs’ theory of ensemble is about statistical measurements of a whole system; not about the individuals within.

We choose to present our theory with finite state space $\mathcal{S}$ for mathematical simplicity. Formal generalization to continuous state space is straight forward if mathematical rigor is not required. Beyond the finite state space, it is known that modern probability and the theory of measures encounter challenges, c.f., de Finetti’s treatment of infinite sets and the axiom of choice of nonempity subsets [20]. In addition to continuous ${\mathbb{R}}^n$ [30], there are even larger Hilbert spaces of functions on ${\mathbb{R}}^n$ and/or von Neumann algebra of operators acting on a Hilbert space.