Legendre Transformation and Information Geometry for the Maximum Entropy Theory of Ecology ^†

Abstract

Here I investigate some mathematical aspects of the maximum entropy theory of ecology (METE). In particular I address the geometrical structure of METE endowed by information geometry. As novel results, the macrostate entropy is calculated analytically by the Legendre transformation of the log-normalizer in METE. This result allows for the calculation of the metric terms in the information geometry arising from METE and, by consequence, the covariance matrix between METE variables.

1. Introduction

The method of maximum entropy (MaxEnt) is usually associated with Jaynes’ work [1,2,3] connecting statistical physics and the information entropy proposed by Shannon [4]—although its mathematics is known since Gibbs [5]. It consists of selecting probability distributions by maximizing a functional—namely entropy—usually under a set of expected values constraints, arriving at what is known as Gibbs distributions. Since Shore and Johnson [6] MaxEnt has been understood as a general method for inference—see also [7,8,9]—hence it is not surprising that (i) Gibbs distributions are what is known in statistical theory as exponential family—the only distributions for which sufficient statistics exist (see e.g., [10]), (ii) MaxEnt encompasses the methods of Bayesian statistics [11], and (iii) MaxEnt has found successful applications in several fields of science (e.g., [12,13,14,15,16,17,18,19,20,21,22]).

One of the scientific fields in which MaxEnt has been successfully applied is macroecology. The work of Harte and collaborators [23,24,25,26,27] presents what is known as the maximum entropy theory of ecology (METE). It consists of finding, through MaxEnt, a joint conditional distribution for the abundance of a species and the metabolic rate of its individuals. From the marginalization and expected values of the MaxEnt distribution, it is possible to obtain (i) the species abundance distribution (Fisher’s log series), (ii) the species-area distribution, (iii) the distribution for metabolic rates over individuals, and (iv) the relationship between the metabolic rate of individuals in a species and that species abundance—for a comprehensive confirmation of METE with experimental data see [28]. In a recent article Harte [29] brings forward the need for dynamical models based on MaxEnt, as METE assume the variables to be static—It is relevant to say that Jaynes applied dynamical methods based on information theory for nonequilibrium statistical mechanics [30] leading to what is known as maximum caliber [31,32]. However, maximum caliber assumes a Hamiltonian dynamics and, therefore, does not generalize to ecology and other complex systems.

The field known as information geometry (IG) [33,34,35,36] assigns a Riemannian geometry structure to probability distributions. In information geometry the distances are given by the Fisher-Rao information metric (FRIM) [37,38], which is the only metric in accordance with the grouping property of probability distributions [39]. IG has found important applications for probabilistic dynamical systems [34,40,41,42,43]. Here the FRIM terms for the distributions arising from METE will be calculated. In a future publication I will evolve METE into entropic dynamical models for ecology, as explained in [43], in order to do so it is necessary to calculate the macrostate entropy and the FRIM terms—which can be obtained from differentiating the macrostate entropy. Therefore, present article performs the calculations necessary for an entropic dynamics model for macroecology.

The layout of the paper is as follows: The following section (2) presents MaxEnt in general terms followed by the MaxEnt process in METE. In particular we obtain the macrostate entropy through the Legendre transform, and the Lambert W special function [44,45], which is a novel result to the best of my knowledge. Section 3 presents some general results of IG and calculate the information metric terms for METE. Section 4 concludes the present article by commenting on possible applications and perspectives for IG in a dynamical theory of macroecology.

2. Maximum Entropy

In information theory, probability distributions encode the available information about a system’s variables $x\in \mathcal{X}$. MaxEnt consists of updating from a prior distribution $q\left(x\right)$—usually, but not necessarily, taken to be uniform—to a posterior $\rho \left(x\right)$ that maximizes the entropy functional under a set of constraints meant to represent the known information about the system. Usually these constraints are the expected values $A^i$ of a set of real valued functions $\left\{a^i\left(x\right)\right\}$ namely sufficient statistics. The distribution $\rho$ is found as the solution to the following optimization problem

$$\begin{array}{cc}\hfill \underset{\rho }{max}& H\left[\rho \right]=-\int \mathrm{d}x\rho \left(x\right)log\left(\frac{\rho \left(x\right)}{q\left(x\right)}\right),\hfill \end{array}$$

(1a)

$$\begin{array}{cc}\hfill s.t.& \int \mathrm{d}x\rho (x)=1\hfill \end{array}$$

(1b)

$$\begin{array}{cc}\hfill & \int \mathrm{d}x{a}^i\left(x\right)\rho \left(x\right)=A^i.\hfill \end{array}$$

(1c)

where $\int \mathrm{d}x$ refers to the appropriate measure of the set $\mathcal{X}$; if one is interested in a discrete set $\mathcal{X}=\left\{x_{\mu }\right\}$, where $\mu$ corresponds to an enumeration of $\mathcal{X}$, we have $\int \mathrm{d}x={\sum }_{\mu }$, if one is interested in a continuous subset of real variables, e.g., $\mathcal{X}=[a,b]$, we have $\int \mathrm{d}x={\int }_a^b\mathrm{d}x$.

The solution of (1) is the Gibbs distribution

$$\rho \left(x\right|{\lambda }_1,{\lambda }_2,...,{\lambda }_n)=\frac{q\left(x\right)}{Z\left(\lambda \right)}exp\left(-{\displaystyle \sum _{i=1}^n}{\lambda }_i{a}^i\left(x\right)\right),$$

(2)

where $\lambda =\left\{{\lambda }_i\right\}$ is the set of Lagrange multipliers dual to the expected values $A=\left\{A^i\right\}$ and $Z\left(\lambda \right)$ is a normalization factor given by

$$Z\left(\lambda \right)=\int \mathrm{d}xq\left(x\right)exp\left(-{\lambda }_i{a}^i\left(x\right)\right).$$

(3)

Above, and on the remainder of this article, we use Einstein’s summation notation $A_i{B}^i={\sum }_i{A}_i{B}^i$. The expected values can be recovered as

$$A^i=-\frac{1}{Z}\frac{\partial Z}{\partial {\lambda }_i}=\frac{\partial F}{\partial {\lambda }_i},\mathrm{where}F\left(\lambda \right)\doteq -log\left(Z\left(\lambda \right)\right).$$

(4)

We will refer to F as the log-normalizer, which displays a role similar to free energy in statistical mechanics.

If one is able to invert the equations arriving from (4), obtaining this way ${\lambda }_i\left(A\right)$ they can express the probability distributions in terms of the expected values, $\rho \left(x\right|A)=\rho (x\left|\lambda \right(A\left)\right)$. This also allows one to calculate the entropy H at its maximum—that means $H\left[\rho \right(x\left|A\right)]$ for $\rho$ in (2)—as a function of the expected values, rather than a functional of $\rho$, obtaining

$$H\left(A\right)\doteq H\left[\rho \left(x\right|\lambda \left(A\right))\right]=-\int dx\rho \left(x\right|\lambda \left(A\right))log\frac{\rho \left(x\right|\lambda \left(A\right))}{q\left(x\right)}={\lambda }_i\left(A\right)A^i-F\left(\lambda \left(A\right)\right).$$

(5)

We will refer to $H\left(A\right)$ as the macrostate entropy, which is what we refer to in statistical mechanics as thermodynamical entropy—meaning the one that appears in the laws of thermodynamics (Since the arguments that identify the macrostate entropy as the thermodynamical entropy assume that the sufficient statistics are conserved quantities in a Hamiltonian dynamics [2], analogous ‘laws of thermodynamics’ - e.g., conservation of $A^2$ in (12) or an impossibility of H in (15) to decrease— are not expected in ecological systems). One can see from (5) that $H\left(A\right)$ is the Legendre transformation [46] of $F\left(\lambda \right)$. It also follows that ${\lambda }_i=\frac{\partial H}{\partial A^i}$.

METE

The first step towards a MaxEnt description involves choosing the appropriate variables for the problem at hand. In METE [24] one assumes an ecosystem of S species supporting N individuals with a total metabolic rate E, meaning in a unit of time the ecosystem consumes a quantity E of energy. The state of the system x on MaxEnt is defined for a singular species as the number of individuals (abundance) n, $n\in \{1,2,\dots ,N\}$ and the metabolic rate of an individual of that species $\epsilon$, $\epsilon \in [1,E]$—note that one can choose a system of units so that the smallest metabolic rate is the unit, ${\epsilon }_{min}=1$. We represent the state as $x=(n,\epsilon )$.

The second step consists of assigning the sufficient statistics that appropriately captures the information about the system. In METE [24] the statistics chosen are the number of individuals in the species $a^1(n,\epsilon )\doteq n$ and the total metabolic rate $a^2(n,\epsilon )\doteq n\epsilon$. Substituting these into the defined expected value constrains for the sufficient statistics (1), we obtain constraints on average abundance per species

$$A^1=\sum _{n=1}^N{\int }_1^E\mathrm{d}\epsilon n\rho (n,\epsilon |\lambda )=\frac{N}{S}\doteq N^{\prime },$$

(6)

and a constrain on the average metabolic consumption per species

$$A^2=\sum _{n=1}^N{\int }_1^E\mathrm{d}\epsilon n\epsilon \rho (n,\epsilon |\lambda )=\frac{E}{S}\doteq E^{\prime }.$$

(7)

The defined variable $N^{\prime }$ and $E^{\prime }$ will replace $A^1$ and $A^2$, respectively, when convenient.

Having the state variables and the sufficient statistics chosen, we can compute all quantities defined in the previous subsection for the specific system defined by METE. With a uniform prior q, justified by the fact that at its level of complexity organisms should be considered as distinguishable, this leads to the canonical distribution (2) of the form

$$\rho (n,\epsilon |\lambda )=\frac{1}{Z\left(\lambda \right)}{e}^{-{\lambda }_{1}n}{e}^{-{\lambda }_{2}n\epsilon },$$

(8)

where the normalization factor (3) is given by

$$Z\left(\lambda \right)=\sum _{n=1}^N{\int }_1^E\mathrm{d}\epsilon e^{-{\lambda }_{1}n}{e}^{-{\lambda }_{2}n\epsilon }=\sum _{n=1}^N{e}^{-{\lambda }_{1}n}\left(\frac{e^{-{\lambda }_{2}n}-e^{-{\lambda }_{2}nE}}{{\lambda }_{2}n}\right),$$

(9)

from which the expected values (4) can be calculated as

$$\begin{array}{cc}\hfill A^1=N^{\prime }& =\frac{1}{{\lambda }_{2}Z\left(\lambda \right)}\sum _{n=1}^N{e}^{-{\lambda }_{1}n}(e^{-{\lambda }_{2}n}-e^{-{\lambda }_{2}nE}),\hfill \end{array}$$

(10a)

$$\begin{array}{cc}\hfill A^2=E^{\prime }& =\frac{1}{{\lambda }_2}\left[1+\frac{1}{Z\left(\lambda \right)}\sum _{n=1}^N{e}^{-{\lambda }_{1}n}(e^{-{\lambda }_{2}n}-E{e}^{-{\lambda }_{2}nE})\right].\hfill \end{array}$$

(10b)

These are complicated equations, however some approximations may make them more treatable.

A fair assumption, knowing what the variables are supposed to represent, is that there are far more individuals than species, $N\gg S$ and the average metabolic rate per individual is far greater than the unit of metabolic rate $E/N=E^{\prime }/N^{\prime }\gg 1$. This allows for a sequence of approximation that we will treat like assumptions here, namely (i) $e^{-{\lambda }_{2}nE}\ll e^{-{\lambda }_{2}n}$, (ii) $E{e}^{-{\lambda }_{2}nE}\ll e^{-{\lambda }_{2}n}$, (iii) ${\lambda }_1+{\lambda }_2\ll 1$, and (iv) $e^{-({\lambda }_1+{\lambda }_2)N}\ll 1$. Further explanation on the validity of these assumptions, under $S\ll N\ll E$, can be seen in [24,26] and their confirmation by numerical calculation can be seen in [24]. Under this understanding we can substitute (9) into (10a) obtaining

$$\begin{array}{cc}\hfill N^{\prime }& =\frac{{\displaystyle \sum _{n=1}^N}{e}^{-{\lambda }_{1}n}(e^{-{\lambda }_{2}n}-e^{-{\lambda }_{2}nE})}{{\displaystyle \sum _{n=1}^N}\frac{1}{n}{e}^{-{\lambda }_{1}n}\left(e^{-{\lambda }_{2}n}-e^{-{\lambda }_{2}nE}\right)}\approx \frac{{\displaystyle \sum _{n=1}^N}{e}^{-({\lambda }_1+{\lambda }_2)n}}{{\displaystyle \sum _{n=1}^N}\frac{1}{n}{e}^{-({\lambda }_1+{\lambda }_2)n}}\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill N^{\prime }& \approx -\left[\frac{1}{({\lambda }_1+{\lambda }_2)log\left({\lambda }_1+{\lambda }_2\right)}\right].\hfill \end{array}$$

(11b)

We can also rewrite (10b) obtaining

$$E^{\prime }=\frac{1}{{\lambda }_2}+\frac{{\displaystyle \sum _{n=1}^N}{e}^{-{\lambda }_{1}n}(e^{-{\lambda }_{2}n}-E{e}^{-{\lambda }_{2}nE})}{{\displaystyle \sum _{n=1}^N}\frac{1}{n}{e}^{-{\lambda }_{1}n}\left(e^{-{\lambda }_{2}n}-e^{-{\lambda }_{2}nE}\right)}\approx \frac{1}{{\lambda }_2}+N^{\prime }.$$

(12)

In order to obtain the macrostate entropy analytically (5) one needs to perform the Legendre transformation for METE, which includes inverting (11) and (12) obtaining ${\lambda }_1(N^{\prime },E^{\prime })$ and ${\lambda }_2(N^{\prime },E^{\prime })$. In page 149 of [24] it is said to be unfeasible. However, it is possible to do so obtaining

$${\lambda }_1=\beta \left(N^{\prime }\right)-\frac{1}{E^{\prime }-N^{\prime }},\mathrm{and}{\lambda }_2=\frac{1}{E^{\prime }-N^{\prime }},$$

(13)

where

$$\beta \left(N^{\prime }\right)\doteq -{\left[N^{\prime }{W}_{-1}\left(-\frac{1}{N^{\prime }}\right)\right]}^{-1},\dot{\beta }\left(N^{\prime }\right) \doteq \frac{\mathrm{d}\beta }{\mathrm{d}{N}^{\prime }}={\left[N^{\prime 2}-\frac{N^{\prime }}{\beta \left(N^{\prime }\right)}\right]}^{-1},$$

(14)

and $W_{-1}$ refers to the second main branch of the Lambert W function (see [44,45]). The details on how (13) inverts (11) and (12) are presented in Appendix A. The macrostate entropy can be calculated directly from (5) as

$$H(N^{\prime },E^{\prime })=N^{\prime }\beta \left(N^{\prime }\right)+log(E^{\prime }-N^{\prime })-log\left(N^{\prime }\beta \left(N^{\prime }\right)\right)+1.$$

(15)

With the calculation of the macrostate entropy finished, we can move into a geometric description of METE.

3. Information Geometry

This section presents the elementary notions of IG—for more in depth discussion and examples see e.g., [33,34,35,36]—and some useful identities for the IG of Gibbs distributions. IG consists of assigning a Riemmanian geometry structure to the space of probability distributions, meaning if a set of distributions $p\left(x\right|\theta )$ is parametrized by a finite number of coordinates, $\theta =\left\{{\theta }^i\right\}$, the distances—which are a measure of distinguishability—$\mathrm{d}\ell$ between the neighbouring distributions $P\left(x\right|\theta +\mathrm{d}\theta )$ and $P\left(x\right|\theta )$ are given by $\mathrm{d}{\ell }^2=g_{ij}\mathrm{d}{\theta }^i\mathrm{d}{\theta }^j$. The work of Cencov [39] demonstrated that the only metric invariant under Markov embeddings—and, therefore, the only one adequate to represent a space of probability distributions—is the metric of the form

$$g_{ij}=\int \mathrm{d}xP\left(x\right|\theta )\frac{\partial logP\left(x\right|\theta )}{\partial {\theta }^i}\frac{\partial logP\left(x\right|\theta )}{\partial {\theta }^j},$$

(16)

know as FRIM.

Considering the MaxEnt results presented in previous section, we can restrict our investigation to the Gibbs distributions using the expected values A as coordinates—${\theta }^i=A^i$ and $P\left(x\right|\theta )=\rho (x\left|A\right)$ as in (2). Two useful expressions arise in that case—for proofs see e.g., [33]—first: the metric terms are the Hessian of the negative of macrostate entropy, meaning

$$g_{ij}=-\frac{{\partial }^{2}H}{\partial A^i\partial A^j}=-\frac{\partial A^i}{\partial A^j},$$

(17)

and second: the covariance matrix between the sufficient statistics $a^i\left(x\right)$ is the inverse matrix of $g_{ij}$, meaning

$$C^{ij}{g}_{jk}={\delta }_k^i,\mathrm{where}{C}^{ij}=\langle a^i\left(x\right)a^j\left(x\right)\rangle -A^i{A}^j.$$

(18)

We can, then, see how these quantities are calculated for METE.

Information Geometry of METE

By substituting the macrostate entropy for METE (15) in (17) we obtain the FRIM terms:

$$\begin{array}{cc}\hfill g_{11}=-\dot{\beta }\left(N^{\prime }\right)+\frac{1}{{(E^{\prime }-N^{\prime })}^2},g_{12}=& g_{21}=-\frac{1}{{(E^{\prime }-N^{\prime })}^2},\hfill \\ \hfill g_{22}=\frac{1}{{(E^{\prime }-N^{\prime })}^2},\mathrm{and}g=& -\frac{\dot{\beta }\left(N^{\prime }\right)}{{(E^{\prime }-N^{\prime })}^2}.\hfill \end{array}$$

(19)

where $g=det{g}_{ij}$. Per (18) and from the general form of inverse matrix of a two dimensional matrix, the covariance matrix terms can be calculated directly inverting (19) obtaining

$$\begin{array}{c}\hfill C^{11}=\frac{g_{22}}{g}=\frac{N^{\prime }}{\beta \left(N^{\prime }\right)}-N^{\prime 2},C^{12}=C^{21}=-\frac{g_{12}}{g}=\frac{N^{\prime }}{\beta \left(N^{\prime }\right)}-N^{\prime 2},\\ \hfill \mathrm{and}{C}^{22}=\frac{g_{11}}{g}=E^{\prime 2}-2E^{\prime }{N}^{\prime }+\frac{N^{\prime }}{\beta \left(N^{\prime }\right)};\end{array}$$

(20)

completing the calculation. The matrix $C^{ij}$ can be interpreted directly as the covariance between a species abundance and its total metabolic rate—METE sufficient statistics. The information metric terms presented in (19) allow for further studies on dynamical ecology from a information theory background, as we will comment in the following section.

4. Discussion and Perspectives

The present article calculates the macrostate entropy (15) for METE. This was made possible by the analytical calculation of the Lagrange multipliers (13) as functions of the expected values (10), previously believed to be unfeasible. This allows for a complete description of METE in terms of the average abundance $N^{\prime }$ and the expected metabolic rate $E^{\prime }$ of each of the ecosystem species. This opens a broad range of investigations possible by analytical calculations. In particular, the IG arising from METE is presented by calculating the FRIM terms in (19). Independently of any geometric interpretation, that was equivalent to calculate the covariance between METE sufficient statistics (20).

The variables that define an ecosystem’s state are not expected to remain constant. Because of this, and the growing relevance of IG in dynamical systems, the calculations made in the present article are an important step into expanding maximum entropy ideas into further investigation in macroecology. The calculations done here allow for evolving METE into an entropic dynamics for ecology, as in the framework developed in [43], this venue of research will be explored in future publication.