Remarks on the Maximum Entropy Principle with Application to the Maximum Entropy Theory of Ecology

Abstract

In the first part of the paper we work out the consequences of the fact that Jaynes’ Maximum Entropy Principle, when translated in mathematical terms, is a constrained extremum problem for an entropy function $H\left(p\right)$ expressing the uncertainty associated with the probability distribution p. Consequently, if two observers use different independent variables p or $g\left(p\right)$, the associated entropy functions have to be defined accordingly and they are different in the general case. In the second part we apply our findings to an analysis of the foundations of the Maximum Entropy Theory of Ecology (M.E.T.E.) a purely statistical model of an ecological community. Since the theory has received considerable attention by the scientific community, we hope to give a useful contribution to the same community by showing that the procedure of application of MEP, in the light of the theory developed in the first part, suffers from some incongruences. We exhibit an alternative formulation which is free from these limitations and that gives different results.

1. Introduction

Maximum Entropy Principle MEP (E.T. Jaynes, 1957, see [1,2]) is a powerful inference principle which allows to determine the probability distribution that describes a system on the basis of the information available, usually in the form of averages of observables (random variables) of interest for the system. It is based on: (i) the enumeration of the system states $i=1,\dots ,N$; (ii) the introduction of one or more functions that translate the information available of the system in the form of constraints on the probability i.e., as $f\left(p\right)=c$ where c is a vector of average values; and (iii) a function measuring the uncertainty associated to a probability distribution candidate to describe the system. We consider here only systems with a finite number of states since the extra generality of considering an infinite number of states is not necessary for the aims of this work. The sought distribution is the one for which uncertainty is maximal on the set of distributions for which $f\left(p\right)=c$. This distribution thus represents the least biased estimate on the basis of the given information whether or not its prescriptions are empirically confirmed by experiments. Each the above outlined Steps (i)–(iii) has a profound meaning and impact on the implementation of the MEP procedure. For example, if the states of the system are a priori equally probable, the uncertainty function to use is Shannon entropy $H\left(p\right)$, while if they are statistically described by a prior distribution q the uncertainty function is (minus) the relative entropy $D\left(p\right|q)$. If the systems states are the result of a coarse-graining procedure from a bigger set of a priori equiprobable states, then a prior distribution q which counts the number of coarse-grained states has to be used for $D\left(p\right|q)$ in order to render the MEP procedure invariant with respect to the coarse graining as explained in [3]. See also [4] where a number of different probability distribution used in ecology are derived for a system of individuals arranged in different cells by simply making different assumptions on the individuals (indistinguishable or not), on the constraints and on the coarse graining. The variety of MEP responses simply reflects the facts that different pieces of information are considered; if the previsions fail to be confirmed by the experiment, this is the signal that relevant information for the statistical description on the system under study has been neglected.

A basic tenet of the Maximum Entropy Principle is that two observer which are given the same information expressed by Steps (i) and (ii) above must obtain the same result upon application of the MEP, i.e., the solution to the constrained extremum problem has to be unique. Notice that the MEP procedure does not specify the probability $p=(p_1,\dots ,p_N)$ to use; thus if one observer uses p and another uses $p^{\prime }$ which is related to p by a one-to-one transformation $p=g\left(p^{\prime }\right)$, provided that they set up the constraints in the form $f\left(p\right)=c$ and $f\left(g\left(p^{\prime }\right)\right)=c$ respectively, they are translating in mathematical terms the same information on the system. This represents the same degree of freedom of using different (but related by an invertible transformation) systems of coordinates to describe the position of a point in physics. Therefore, to ensure that the results of the application of the MEP procedure are independent of the choice of the p used the uncertainty function has to be defined accordingly. Proposition 1 in Section 2 below is the main tool to determine the form of the uncertainty function using g-related distributions.

The analysis of this subtile point of the application of MEP is the main aim of this paper. We argue that since there seems not to be a preferred choice for p they have to be considered all equivalent; therefore the related uncertainty functions are also equivalent and can be derived by a single given one. The question is: which is the translation in mathematical terms of the initial information which is entitled to use as uncertainty function the Shannon entropy? We give an answer by resorting to the Boltzmann statistical derivation of the entropy function, which has the same form of Shannon one. This statistical procedure, based on independent tosses of indistinguishable particles in bins, can be applied to different models and allows determining from a sort of “first principle” the form of the entropy function to use. We think that there is room for further investigation of this subtile issue of the application of MEP.

In the second part of the paper we apply the above findings to a specific application of MEP to ecology called Maximum Entropy Theory of Ecology (M.E.T.E.) as exposed in [5]. M.E.T.E. is a purely statistical model of an ecological community. Using the Maximum Entropy Principle the theory aims at inferring the form of some of the most used distributions in ecology from the knowledge of macroscopic information on the system: number of individuals, number of species and total metabolic requirement. MEP has been applied in the field of statistical ecology mainly to derive one of the most important probability distributions, the Species Abundance Distribution (SAD) but not exclusively (see [6] or [7] where it is used for modeling species geographic distributions with presence-only data). Several papers ([3,8,9]) have revised the application of the MEP inference principle to ecological problems, stressing the importance of the choice of the system’ coordinate [10,11,12], of the prior distribution [4,8] and of the entropy function [3,6]. The interest of [5] is that it considers simultaneously information on the distribution of individuals among species (SAD) and on the distribution of metabolic energy rate among individuals. In this richer scenario a number of probability distributions for different observables of interest are derived. The output of the work has grown to become a comprehensive theory of application of MEP to ecological problems, called Maximum Entropy Theory of Ecology (see the book [13,14]) which has been extensively tested with real ecosystem data in [15,16,17]. In Section 4.1 we review the assumptions of the model in [5] and the related MEP solution; in Section 4.2 we propose an alternative application of the MEP procedure starting from the same initial information but giving a different result. In Section 5 we discuss the non equivalence of the two procedures using the Proposition 1 below and motivate why the MEP solution in [5] is flawed by some inconsistences.

2. MEP Formulation Using Different Variables

The Maximum Entropy Principle is generally expressed as: p is the distribution that maximizes Shannon entropy $H\left(p\right)$ over the set of distributions allowed by the constraints. Let $p^{\prime }$ defined by $p^{\prime }=g^{-1}\left(p\right)$ with g invertible be a different choice of the variable to be used to express the probability distribution and the constraints and suppose that for the solution of this constrained extremum problem we use Lagrange multipliers method. Then, the following Proposition holds (see Appendix A for a proof).

Proposition 1.

$\widehat{p}$ is the constrained extremum of $H\left(p\right)$ over the set $f\left(p\right)=c$ if and only if ${\widehat{p}}^{\prime }=g^{-1}\left(\widehat{p}\right)$ is the constrained extremum of $H\left(g\left(p^{\prime }\right)\right)$ over the set $f\left(g\left(p^{\prime }\right)\right)=c$.

From the above Proposition 1 we learn that if we are given the entropy function and the constraints as $(p,f(p),H(p\left)\right)$ and we want to use another variable $p^{\prime }=g^{-1}\left(p\right)$, to find the same MEP solution we have to transform the constraints and the entropy function accordingly using $(p^{\prime },f\circ g\left(p^{\prime }\right),H\circ g\left(p^{\prime }\right))$. In this way, changing variable and finding the maximum are operations whose order can be interchanged. A natural question arises in this setting: which one is the couple $(p,f(p\left)\right)$ translating in mathematical terms the initial information on the system that “is justified” in using the entropy function in Shannon form? We will give a partial answer to this question in Section 3 below.

Note that the invariance requirement with respect to the choice of the independent variable is different from the requirement, introduced in the axiomatic derivation of MEP in [18], of invariance in the form of $H\left(p\right)$ with respect to a transformation of the type $y=\Gamma \left(x\right)$, where $x\in D$ are the (possibly infinite) states of a system and $p=p\left(x\right)$. For a system with finite states the transformation $\Gamma$ (called coordinate transformation in [18]) reduces to a relabeling of the states. Shannon entropy $H\left(p\right)=-{\sum }_{i=1}^N{p}_i\ln p_i$ being additive is clearly invariant in form, expressing the fact that different observers may use different labels for the N system states. This relabeling transformation

$$p_k^{\prime }=g_k(p_1,\dots ,p_N)=p_{{\phi }_k(1,\dots ,N)}k=1,\dots ,N$$

where ${\phi }_k$ is a permutation of the N state labels is thus a particular case of the more general g considered in the above Proposition 1. It is clearly an invertible transformation in which $H\left(p\right)$ and $H\left(g\left(p^{\prime }\right)\right)$ do coincide, but this is no longer assured for more general transformations g; see Section 3 below.

3. A Simple, Ecologically Oriented Example

To throw light on the questions considered above we take a very simple system used in a number of physical models that for our purposes can be considered as a minimal example of an ecological community. Consider a set $\mathcal{I}$ of $N_0$ identical individuals (balls) which belong to S species (indistinguishable urns). Suppose that each species contains at least one individual so that the maximal number of individuals allowed to one species is $N=N_0-(S-1)$. Suppose that the species labels can be interchanged so that the observable quantity is not the number of individuals $n_{\alpha }$ of a given species $\alpha$ but the number $S_n$ of species which have exactly n individuals, for $n=1,\dots ,N$.

Suppose that the set of species $\mathcal{S}$ is equipped with the uniform probability $1/S$ and consider the random variable $\nu :\mathcal{S}\to \mathcal{N}$, $\alpha \mapsto \nu \left(\alpha \right)=n_{\alpha }$, $\mathcal{N}=\{1,\dots N\}$ that assigns to a species its abundance. Let $\varphi \left(n\right)$ be the density associated to the random variable $\nu$ (in the sequel P denotes probability)

$$\varphi \left(n\right)=P(\nu =n)=P(\alpha \in {\nu }^{-1}\left(n\right))=\frac{|{\nu }^{-1}\left(n\right)|}{S}=\frac{S_n}{S},n\in \mathcal{N}$$

(1)

where $\left|A\right|$ is the number of elements of A. Set for later use ${\mathcal{S}}_n={\nu }^{-1}\left(n\right)\subset \mathcal{S}$, therefore $|{\mathcal{S}}_n|=S_n$.

Let the set of individuals $\mathcal{I}$ be equipped with the uniform probability $1/N_0$, let $\alpha :\mathcal{I}\to \mathcal{S}$, $i\mapsto \alpha \left(i\right)$ be the assignation of each individual to a species and consider the random variable $\nu \circ \alpha :\mathcal{I}\to \mathcal{N}$, $i\mapsto \nu \left(\alpha \right(i\left)\right)$. Moreover, consider the following subset ${\mathcal{I}}_n$ of $\mathcal{I}$ obtained by grouping together all the individuals belonging to the species having n individuals (we call this subset the “multispecies” n)

$${\mathcal{I}}_n=\{i\in \mathcal{I}:\nu \left(\alpha \left(i\right)\right)=n\}\subset \mathcal{I},I_n=\left|{\mathcal{I}}_n\right|.$$

(2)

As a straightforward consequence, we have that $I_n=n{S}_n$. Let ${\varphi }_I\left(n\right)$ be the density associated to the random variable $\nu \circ \alpha$ that is

$${\varphi }_I\left(n\right)=P(\nu \circ \alpha =n)=P(i\in {\mathcal{I}}_n)=\frac{I_n}{N_0},n\in \mathcal{N}.$$

(3)

Both $\varphi$ and ${\varphi }_I$ are probability distributions on $\mathcal{N}$; it is easy to see that since $I_n=n{S}_n$, they are related by an invertible transformation ${\varphi }_I=g\left(\varphi \right)$ defined as

$${\varphi }_I\left(n\right)=g_n\left(\varphi \right)=\frac{nS}{N_0}\varphi \left(n\right)n=1,\dots ,N.$$

(4)

In ecology the distribution $\varphi$ is called Species Abundance Distribution (SAD) and is probably the most important (and investigated) indicator of how an ecological community is structured. While $\varphi \left(n\right)$ gives the probability of extracting a species of abundance n from the set of species, ${\varphi }_I\left(n\right)$ gives the probability that an individual extracted from the set $\mathcal{I}$ belongs to a species of abundance n. Real communities are composed of many species with few individuals (rare species) and few ones which contain the majority of the individuals (common species). As a consequence, the two probability distribution are very different.

The information on the system resumed by the numbers $N_0,S$ sets the following equalities

$$N_0=\sum _{n=1}^{N}n{S}_n=\sum _{n=1}^N{I}_n\mathrm{and}S=\sum _{n=1}^N{S}_n=\sum _{n=1}^N\frac{1}{n}{I}_n.$$

(5)

If we divide the above two equalities by S or $N_0$ respectively, we obtain the following equations to be satisfied by $\varphi$

$$\frac{N_0}{S}=\sum _{n=1}^{N}n\varphi \left(n\right),1=\sum _{n=1}^N\varphi \left(n\right)$$

(6)

and ${\varphi }_I$ respectively

$$1=\sum _{n=1}^N{\varphi }_I\left(n\right),\frac{S}{N_0}=\sum _{n=1}^N\frac{1}{n}{\varphi }_I\left(n\right).$$

(7)

Constraints (6) and (7) are g-related, that is $\varphi$ satisfies Equation (6) if and only if $g\left(\varphi \right)$ satisfies Equation (7). Note that in first case we we are constraining the weighted arithmetic average of $1,\dots ,N$ while in second one we are constraining the weighted harmonic average. Note also that the actual information being used is the ratio $N_0/S$ or equivalently $S/N_0$, so that the MEP description of the system is independent of the size $N_0,S$ of the system provided that the ratio $N_0/S$ is kept constant.

Suppose that two different observers translate in mathematical terms the information contained in the data $N_0,S$ using the two above introduced variables $\varphi$ and ${\varphi }_I$ and that both invoke the MEP to find the least informative probability distribution allowed by the constraints. The application of MEP with entropy function $H\left(\varphi \right)=-{\sum }_{n=1}^N\varphi \left(n\right)ln\varphi \left(n\right)$ and Constraint (6) gives, upon application of the Lagrange multipliers methods, the geometric probability distribution

$$\widehat{\varphi }\left(n\right)=\frac{e^{-\lambda n}}{Z\left(\lambda \right)},Z\left(\lambda \right)=\sum _{n=1}^N{e}^{-\lambda n}$$

(8)

while the MEP procedure with the same entropy function $H\left({\varphi }_I\right)=-{\sum }_{n=1}^N{\varphi }_I\left(n\right)\ln {\varphi }_I\left(n\right)$ and Constraint (7) gives the solution

$${\widehat{\varphi }}_I\left(n\right)=\frac{e^{-\gamma \frac{1}{n}}}{Z\left(\gamma \right)},Z\left(\gamma \right)=\sum _{n=1}^N{e}^{-\gamma \frac{1}{n}}$$

(9)

whose corresponding SAD is

$$\tilde{\varphi }\left(n\right)=g^{-1}({\widehat{\varphi }}_I)\left(n\right)=\frac{N_0}{Sn}\frac{e^{-\gamma \frac{1}{n}}}{Z\left(\gamma \right)}.$$

(10)

It is easy to see that $\widehat{\varphi }\ne g^{-1}({\widehat{\varphi }}_I)$, therefore with this example we have shown that changing independent variables and using the MEP with the Shannon entropy are non commuting operations in the general case. Had we used Constraint (7) with the transformed entropy

$$H^{\prime }\left({\varphi }_I\right)=H\left(g^{-1}\left({\varphi }_I\right)\right)=-\sum _{n=1}^N{\varphi }_I\left(n\right)\frac{N_0}{nS}\ln [{\varphi }_I\left(n\right)\frac{N_0}{nS}]$$

(11)

we would have obtained the solution $g(\widehat{\varphi })$ with $\widehat{\varphi }$ in Equation (8). Note that the transformed entropy in Equation (11) does not have the form of a relative entropy function therefore it can not be obtained by introducing a suitable prior probability distribution. Note also that the non commutativity considered above is different from the non commutativity of MEP procedure with respect to coarse graining of the system state considered in [3,4].

Which One is the Correct Application of MEP?

Both $\varphi \left(n\right)$ in Equation (8) and $\tilde{\varphi }\left(n\right)$ in Equation (10) result from the application of the MEP starting from the same initial information $N_0,S$ on the system and provide non equivalent answers to the question of how the abundance of individuals is distributed among the species. Since the set of distributions allowed by the constraints (feasible set) in the two formulations Equations (6) and (7) are related by an invertible transformation $D_I=g\left(D\right)$, the non equivalence must reside in the choice of the form of the entropy functions which are not g-related according to Proposition 1. On the basis of the same Proposition, once the form of the entropy function is established in a given formulation of MEP $(p,f(p),H(p\left)\right)$, it is also determined for all g-related formulations. It seems thus necessary to invoke a sort of first principle to establish the correct form of the entropy function in the initial MEP formulation.

A possible approach could be to compare the form of the resulting probability distribution against field data. This is not entirely satisfactory since the discrepancy between the MEP solution and the empirical curve could be due to the fact that we have neglected (or we are not aware of) some information on the system which is relevant for the question addressed. In this case the discrepancy between the model and the empirical pattern is a signal that other factors shaping the probability distribution are acting on the system. By the way, for the problem considered above, it is generally acknowledged in the ecological literature (see e.g., [5]) that the geometric SAD in Equation (8) has a poor fit with the empirical SAD, while the one introduced in (10) could provide a better fit since is more flexible (it can be monotonically decreasing or display an interior mode depending on the value of the Lagrange multiplier).

At least for systems with a discrete number of states another criterion can be introduced, which uses the combinatorial argument providing one of the justifications for the form of Shannon entropy historically preceding the axiomatic derivation in [18].

This is the celebrated Boltzmann problem of statistical mechanics [19]. It is useful to briefly review its solution here because it tells us how to determine the correct form of the entropy function from first principles in cases where its determination is not obvious. It deals with the distribution of K indistinguishable particles (individuals) among M equally spaced energy levels $ϵ=1,\dots ,M$. Boltzmann original formulation dealt with arbitrarily spaced energy levels but this extra generality is not needed here. If $n_{ϵ}$ is the number of particles having energy $ϵ$ and $\mathbf{n}=(n_1,\dots ,n_M)$ is the vector of occupation numbers of the energy levels, the logarithm of the probability of $\mathbf{n}$ is

$$lnW\left(\mathbf{n}\right)=ln\frac{1}{M^K}\frac{K!}{n_1!\dots n_M!}$$

(12)

which using Stirling approximation $\ln n!\approx n\ln n$ can be rewritten as

$$\ln W\left(\mathbf{n}\right)=K\ln K-\sum _{ϵ}{n}_{ϵ}\ln n_{ϵ}.$$

(13)

The maximization of $\ln W\left(\mathbf{n}\right)$ under the constraints ${\sum }_{ϵ}{n}_{ϵ}=K$ and ${\sum }_{ϵ}ϵn_{ϵ}=E_0$ gives the most common, i.e., least informative vector of occupation numbers. The solution of the constrained maximization problem is the celebrated Boltzmann-Gibbs distribution

$$n_{ϵ}=\frac{K{e}^{-\beta ϵ}}{{\sum }_{ϵ=1}^M{e}^{-\beta ϵ}},\psi \left(ϵ\right)=\frac{n_{ϵ}}{K}=\frac{e^{-\beta ϵ}}{{\sum }_{ϵ=1}^M{e}^{-\beta ϵ}}$$

(14)

Note that writing $n_{ϵ}=\psi \left(ϵ\right)K$, $\ln W$ in Equation (13) coincides up to a constant with

$$H\left(\psi \right)=-\sum _{ϵ}\psi \left(ϵ\right)\ln \psi \left(ϵ\right).$$

(15)

In the same manner, if we have S indistinguishable species to be assigned in the N levels (abundances) of $\mathcal{N}$ let $\mathbf{s}=(S_1,\dots ,S_N)$ be the occupation vector with constraints ${\sum }_n{S}_n=S$ and ${\sum }_{n}n{S}_n=N_0$. As before, setting $S\varphi \left(n\right)=S_n$ the least informative probability distribution is

$$\varphi \left(n\right)=\frac{e^{-\lambda n}}{{\sum }_n{e}^{-\lambda n}}=\frac{e^{-\lambda n}}{Z\left(\lambda \right)}$$

(16)

and the associated entropy is

$$H\left(\varphi \right)=-\sum _n\varphi \left(n\right)\ln \varphi \left(n\right)$$

(17)

Therefore the form of the entropy function for the formulation given by Equation (6) of the MEP problem using the variable $\varphi$ can be derived by Boltzmann combinatorial argument.

Let us see which is the resulting entropy function if we apply Boltzmann argument to the formulation given by Equation (7) using ${\varphi }_I$. Remember that

$$\left(\mathrm{A}\right)\varphi \left(n\right)=Prob(\alpha \in {\mathcal{S}}_n),\left(\mathrm{B}\right){\varphi }_{\mathcal{I}}\left(n\right)=Prob(i\in {\mathcal{I}}_n)$$

(18)

so basically in (A) we are “tossing” S species in the N bins ${\mathcal{S}}_n$ while in (B) we are “tossing” $N_0$ individuals in the N multispecies $I_n$. In the Boltzmann argument the tosses are supposed to be independent and this is true for (A) but not for (B). Indeed a multispecies $I_n$ is the grouping of $S_n$ species each having n individuals so the number of individuals in ${\mathcal{I}}_n$ can be updated only by multiples of n, i.e., adding or subtracting a species of n individuals. Hence the tosses of single individuals does not represent independent tosses. If we want to support the choice of the entropy function by Boltzmann argument we have to consider tosses of groups of n individuals. Let $i_n$ be the number of individuals tossed in ${\mathcal{I}}_n$. Therefore the related occupation vector has the form

$$\mathbf{i}=(\frac{i_1}{1},\dots \frac{i_n}{n},\dots \frac{i_N}{N}),W\left(\mathbf{i}\right)=\frac{1}{N^{N_0}}\frac{N_0!}{\frac{i_1}{1}!\dots \frac{i_N}{N}!}$$

(19)

which leads to an entropy (recall that $i_n=N_0{\varphi }_{\mathcal{I}}\left(n\right)$) of the form

$$H\left({\varphi }_{\mathcal{I}}\right)=-\sum _n\frac{N_0}{n}{\varphi }_{\mathcal{I}}\left(n\right)\ln \left[\frac{N_0}{n}{\varphi }_{\mathcal{I}}\left(n\right)\right]$$

and under the Constraints (7) above to the solution

$${\varphi }_{\mathcal{I}}\left(n\right)=n\frac{S}{N_0}\frac{e^{-\lambda n}}{Z\left(\lambda \right)}=g(\widehat{\varphi }\left(n\right)).$$

(20)

We have shown that in the case where the probability distribution being used can be linked at least conceptually to independent repetitions of an experiment, we have a criterion to derive the correct form of the entropy function from a first principle. Of course to build a general theory this simple example deserves further generalization.

4. Maximum Entropy Theory of Ecology

4.1. Review of M.E.T.E. Assumptions

Let us review the assumptions of M.E.T.E. as exposed in [5]. In [5], various metrics (probability distributions) are derived but here we will content ourselves with the analysis of the derivation of two of them, namely the Species Abundance Distribution $\varphi \left(n\right)$ and the distribution of metabolic energy $\psi \left(ϵ\right)$ between the individuals.

Following [5], we consider a community made of $N_0$ individuals living on an area $A_0$, subdivided into S species and having total metabolic energy rate $E_0$. Assume that each species has at least one individual and that the individual metabolic rate is a discrete quantity which ranges from 1 to a maximum M in a suitable energy unit. With these assumptions, the abundance of a species $\alpha$ is $\nu \left(\alpha \right)=1,\dots N$ with $N=N_0-S+1$ and the individual metabolic rate is $m\left(i\right)\in \mathcal{M}=\{1,\dots ,M\}$, with $M=E_0-S_0+1$. Each individual is labelled by a double label $(\alpha ,ϵ)$ indicating the species $\alpha =1,\dots ,S$ and the metabolic rate $ϵ=1,\dots ,M$. Here we depart from [5] by assuming for the sake of simplicity that the individual metabolic rate is a discrete quantity while in [5] it is assumed to be a continuous one. This change does not affects our argument since for dealing with the continuous case it is enough to substitute the sums with integrals.

The information on the system is limited to the values $N_0$, S, $E_0$, and $A_0$, but for our aims the knowledge of $A_0$ is not relevant. Therefore we are dealing with a non spatial model of a community of S non interacting species. On the basis of this sole knowledge we would like to answer to two questions: ($Q_1$) How is the energy distributed among the individuals? ($Q_2$) How are the individuals distributed among the species? Note that, since the species’ label can be exchanged, the correct formulation of ($Q_2$) is how many of the species have exactly n individuals for $n=1,\dots ,N$, which is the information contained in $\varphi \left(n\right)$.

Both questions can be answered in a non deterministic manner; for ($Q_2)$, we use the density $\varphi$ introduced in Equation (1), while, for ($Q_1)$, we introduce the map $m:\mathcal{I}\to \mathcal{M}$, $i\mapsto m\left(i\right)$ that we will consider as a random variable and its density $\psi \left(ϵ\right)$ on $\mathcal{M}$

$$\psi \left(ϵ\right)=P(m=ϵ)=P(i\in m^{-1}\left(ϵ\right))=\frac{I_{ϵ}}{N_0},\forall ϵ\in \mathcal{M}$$

(21)

where ${\mathcal{I}}_{ϵ}=m^{-1}\left(ϵ\right)\subset \mathcal{I}$ and $I_{ϵ}=\left|{\mathcal{I}}_{ϵ}\right|$. For later use, consider also the joint probability distribution

$$P(n,ϵ)=Prob(i\in {\mathcal{I}}_n,i\in {\mathcal{I}}_{ϵ})=\frac{1}{N_0}|{\mathcal{I}}_n\cap {\mathcal{I}}_{ϵ}|$$

(22)

having marginals, respectively, ${\varphi }_{\mathcal{I}}\left(n\right)$ and $\psi \left(ϵ\right)$, since the space $\mathcal{I}$ is doubly partitioned in multispecies and energy classes.

4.2. Our Solution of M.E.T.E. Problem

The following constraints contain all the information $(N_0,S,E_0)$ available on the system

$$\sum _{ϵ}\psi \left(ϵ\right)=1,\sum _{ϵ}ϵ\psi \left(ϵ\right)=\frac{E_0}{N_0}$$

(23)

$$\sum _n\varphi \left(n\right)=1,\sum _{n}n\varphi \left(n\right)=\frac{N_0}{S}$$

(24)

and a straightforward application of MEP with the constraints in Equation (23) and the entropy in Equation (15) [with the constraints in Equation (24) and the entropy in Equation (17) respectively] gives the answers $\widehat{\psi }$ in Equation (14) [$\widehat{\varphi }$ in Equation (16)] to the above questions ($Q_1)$ and ($Q_2)$, i.e.,

$$\widehat{\psi }\left(ϵ\right)=\frac{e^{-\beta ϵ}}{Z\left(\beta \right)}=P(i\in {\mathcal{I}}_{ϵ}),\widehat{\varphi }\left(n\right)=\frac{e^{-\lambda n}}{Z\left(\lambda \right)}=P(\alpha \in {\mathcal{S}}_n).$$

Basically, we are solving twice Boltzmann problem of statistical mechanics. Now, if one is interested in the joint probability

$$Q(n,ϵ)=P(\alpha \in {\mathcal{S}}_n,i\in {\mathcal{I}}_{ϵ})$$

(25)

we proceed as before, introducing the vector $\mathbf{c}=(C_{1,1},\dots C_{N,M})$ where $C_{n,ϵ}$ counts the number of times the object $(\alpha ,i)$ is assigned to the bin ${\mathcal{S}}_n\times {\mathcal{I}}_{ϵ}$. Hence

$$W\left(\mathbf{c}\right)=\frac{1}{{\left(NM\right)}^{N_{0}S}}\frac{\left(N_{0}S\right)!}{C_{1,1}!\dots C_{N,M}!}$$

(26)

with

$$\ln W\left(\mathbf{c}\right)=-N_{0}S\ln N_{0}S-\sum _{n,ϵ}{C}_{n,ϵ}\ln C_{n,ϵ}$$

(27)

to be maximized under the decoupled constraints (${\sum }_{ϵ}{C}_{n,ϵ}=S_n,{\sum }_n{C}_{n,ϵ}=n_{ϵ}$)

$$\sum _{n,ϵ}{C}_{n,ϵ}=N_{0}S,\sum _{n,ϵ}n{C}_{n,ϵ}=N_0,\sum _{n,ϵ}ϵC_{n,ϵ}=E_0.$$

(28)

The related entropy is, setting $C_{n,ϵ}=Q(n,ϵ)N_{0}S$

$$H_{\mathcal{N}\times \mathcal{M}}\left(Q\right)=-\sum _{n,ϵ}Q(n,ϵ)\ln Q(n,ϵ)$$

(29)

and the maximum entropy distribution with Constraint (28) can be shown to be

$$Q(n,ϵ)=\varphi \left(n\right)\psi \left(ϵ\right)=\frac{e^{-\lambda n}}{Z\left(\lambda \right)}\frac{e^{-\beta ϵ}}{Z\left(\beta \right)}.$$

(30)

We claim that Equation (30) is the correct solution of the MEP problem dealt with in [5,13] in the sense that the choice of the entropy function adopted is supported by the Boltzmann argument. We do not claim that this solution has a better fit with empirically derived patterns with respect to others but only that is derived in a consistent way. In this solution the two random variables $\nu$ and m are uncorrelated, therefore the energy and species constraints can be dealt with separately giving Equations (14) and (16) or at the same time. This is a well known property of MEP: if the constraints concern only the marginals of a joint probability, then the MEP solution is the product of two unidimensional distributions.

5. Analysis of the Application of MEP in M.E.T.E.

The starting point of M.E.T.E. theory is the following probability distribution in Equation (31) on $\mathcal{N}\times \mathcal{M}$ introduced in [5], formula (4c). Quoting [5], page 2702 above formula (1a) : “$R(n,ϵ)$ is the probability that if a species is picked at random form the species list, then it has abundance n, and if an individual is picked at random from that species with abundance n, then its metabolic rate is $ϵ$” (is between $[ϵ,ϵ+dϵ]$ in the original continuum energy formulation, italics is ours)

$$R(n,ϵ)=\varphi \left(n\right)\Theta \left(ϵ\right|n).$$

(31)

This a delicate point: we think that using “that species” makes the above definition logically inconsistent because if there are more than one species with abundance n it is impossible to know which species is being picked and so the probability of picking an individual with a given metabolic rate is not uniquely defined. One can readily convince oneself of this by considering a toy model of the systems along the lines of example in Box 7.1 in [13], page. 144, but with at least two species having equal abundance and containing individuals with the same metabolic rate. The only way for us to be logically consistent is to substitute “that species” with “a species”. This is equivalent to picking an individual of given metabolic rate from the multispecies ${\mathcal{I}}_n$. In this paper we have thus intended the definition of $R(n,ϵ)$ as amended in this way.

Note however that in the sequel of [5], below formula (4c), it is (correctly) written “a species”. Note also that the same ambiguity is present in the book [13], Chapter 7, pages 142 and 143. In our framework, we thus rephrase the definition as

$$R(n,ϵ)=\varphi \left(n\right)P(i\in {\mathcal{I}}_{ϵ}|i\in {\mathcal{I}}_n)$$

(32)

and using the definition of conditional probability $P(A,B)=P\left(A\right)P\left(B\right|A)$ and (4), (22) also as

$$R(n,ϵ)=\varphi \left(n\right)\frac{P(i\in {\mathcal{I}}_{ϵ},i\in {\mathcal{I}}_n)}{P(i\in {\mathcal{I}}_n)}=\varphi \left(n\right)\frac{P(n,ϵ)}{{\varphi }_I\left(n\right)}=\frac{N_0}{Sn}P(n,ϵ)$$

(33)

Therefore, the probability distributions $P(n,ϵ)$ and $R(n,ϵ)$ are related by the invertible transformation g (we use the same symbol for the sake of simplicity)

$$P(n,ϵ)=g_n\left(R(n,ϵ)\right)=\frac{nS}{N_0}R(n,ϵ).$$

Moreover, the information $N_0,S,E_0$ sets the following constraints on $R(n,ϵ)$, see $\left(1a\right)$–$\left(1c\right)$ in [5]

$$\sum _{n,ϵ}nR(n,ϵ)=\frac{N_0}{S}$$

(34)

$$\sum _{n,ϵ}ϵnR(n,ϵ)=\frac{E_0}{S}$$

(35)

$$\sum _{n,ϵ}R(n,ϵ)=1$$

(36)

hence also $R(n,ϵ)$ is properly normalized. In [5], the MEP procedure is applied as follows: the probability distribution $R(n,ϵ)$ is the one that maximize the Shannon entropy

$$H\left(R\right)=-\sum _{n,ϵ}R(n,ϵ)\ln R(n,ϵ)$$

(37)

on the set defined by Constraints (34)–(36) above. It is therefore the least informative probability distribution on the basis of the sole knowledge of the macroscopic ratios $E_0/S$, $N_0/S$. Note that, as observed in [5], if the previous ratios are known, $E_0/N_0$ is also known.

Standard application of the MEP gives the probability distribution depending on the unknown multipliers $\lambda$ and $\beta$ (see [5], formula (6))

$$R(n,ϵ)=\frac{e^{-\lambda n-\beta ϵn}}{Z(\lambda ,\beta )}.$$

(38)

The Lagrange multipliers $\lambda$ and $\beta$ have to be determined by inserting Equation (38) into the constraints in Equations (34) and (35) but an analytical solution of the resulting equations does not exist. Adopting some approximations in [5] we are lead to the following explicit formulae for the marginal (the SAD distribution)

$$\mathrm{\Phi }\left(n\right)=\sum _{ϵ}R(n,ϵ)\approx \frac{1}{\ln {\lambda }^{-1}}\frac{e^{-\lambda n}}{n}$$

(39)

which is the celebrated Fisher log-series ([20]) and to

$$\mathrm{\Psi }\left(ϵ\right)=\frac{S}{N_0}\sum _{n}R(n,ϵ)\approx \lambda \beta \frac{e^{-\lambda -\beta ϵ}}{{(1-e^{-\lambda -\beta ϵ})}^2}$$

(40)

for the energy marginal. Moreover, the conditional probability is

$$\Theta \left(ϵ\right|n)\approx \beta n{e}^{-\beta nϵ}.$$

(41)

What is striking in the above result, is that it prescribes a non zero correlation (Equation (41)) between the distribution of energy among individuals and the distribution of abundance among species, even if in the initial information nothing seems to hint at a possible correlation. This is for us the signal of a flaw in the above application of MEP.

The Correct Form of Entropy Function of M.E.T.E. Problem

Our aim now is to derive the form of the entropy function for the MEP problem in the $R(n,ϵ)$ variable with constraints on energy and species in Equations (34)–(36) using the Boltzmann argument. Since $R(n,ϵ)$ is not a joint probability distribution, we derive the entropy function for the distribution $P(n,ϵ)=P(i\in {\mathcal{I}}_n,i\in {\mathcal{I}}_{ϵ})$ in (22) which is related to $R(n,ϵ)$ by the change of variable (33) and use the Proposition 1. The g-related constraints of Equations (34)–(36) for $P(n,ϵ)$ are

$$\sum _{n,ϵ}P(n,ϵ)=1,\sum _{n,ϵ}\frac{1}{n}P(n,ϵ)=\frac{S}{N_0},\sum _{n,ϵ}ϵP(n,ϵ)=\frac{E_0}{N_0}.$$

Note that using $P(n,ϵ)$ the constraints are only on the marginals of $P(n,ϵ)$ hence unlike for $R(n,ϵ)$ the MEP solution is the product of the marginals, see (48) below. Given the occupation vector $\mathbf{i}=(i_1,\dots ,i_N)$ of the $N_0$ individuals in the N multispecies, for every n we have to distribute $i_n$ individuals among the M energy groups ${\mathcal{I}}_{ϵ}$ and each arrangement ${\mathbf{i}}_n=(i_{n,1},\dots ,i_{n,M})$ with $i_n={\sum }_{ϵ}{i}_{n,ϵ}$ has probability

$$W_n\left({\mathbf{i}}_n\right)=\frac{1}{M^{i_n}}\frac{i_n!}{i_{n,1}!\dots i_{n,M}!}$$

(42)

Therefore, taking into account that we have $i_n/n$ independent tosses in the bin ${\mathcal{I}}_n$, we have to use the multiplicity factor $W\left(\mathbf{i}\right)$ in Equation (19)

$$W=W\left(\mathbf{i}\right)\prod _{n=1}^N{W}_n\left({\mathbf{i}}_n\right)$$

(43)

hence

$$\ln W=\ln W\left(\mathbf{i}\right)+\ln \prod _{n=1}^N{W}_n\left({\mathbf{i}}_n\right).$$

(44)

The maximum of $\ln W$ is reached when both terms in the right hand side are maximized, but the first term $\ln W\left(\mathbf{i}\right)$ is independent of the arrangement in the energy levels. Therefore we can maximize the second term in the right hand side of Equation (44) for fixed values of $\mathbf{i}$ and then maximize the first term with respect to $\mathbf{i}$. Now

$$\ln \prod _{n=1}^N{W}_n\left({\mathbf{i}}_n\right)=\sum _n\ln W_n\left({\mathbf{i}}_n\right)=-\sum _{n,ϵ}{i}_{n,ϵ}\ln i_{n,ϵ}+f\left(\mathbf{i}\right)$$

(45)

is to be maximized under the $N+1$ constraints (note that the constraint ${\sum }_{n,ϵ}{i}_{n,ϵ}=N_0$ is already enforced since ${\sum }_n{i}_n=N_0$)

$$\sum _{n,ϵ}ϵi_{n,ϵ}=E_0,\mathrm{and}\sum _{ϵ}{i}_{n,ϵ}=i_n,n=1,\dots ,N.$$

(46)

The unconstrained extremum problem for the Lagrange function

$$G=-\sum _{n,ϵ}{i}_{n,ϵ}\ln i_{n,ϵ}-\beta (\sum _{n,ϵ}ϵi_{n,ϵ}-E_0)-\sum _n{\mu }_n(\sum _{ϵ}{i}_{n,ϵ}-i_n)$$

(47)

gives the solution

$$i_{n,ϵ}=i_n\frac{e^{-\beta ϵ}}{{\sum }_n{e}^{-\beta ϵ}}=i_n\frac{e^{-\beta ϵ}}{Z\left(\beta \right)}.$$

Since $i_{n,ϵ}=P(n,ϵ)N_0$ and $i_n={\varphi }_{\mathcal{I}}\left(n\right)N_0$ we have

$$P(n,ϵ)={\varphi }_{\mathcal{I}}\left(n\right)\frac{e^{-\beta ϵ}}{Z\left(\beta \right)}$$

(48)

and to get the complete solution we have to maximize $W\left(\mathbf{i}\right)$, which has already been dealt with in the previous section, Equations (19) and (20). Therefore, the MEP procedure for the distribution $P(n,ϵ)$ with the constraints gives the solution

$$P(n,ϵ)=\frac{nS}{N_0}\frac{e^{-\lambda n}}{Z\left(\lambda \right)}\frac{e^{-\beta ϵ}}{Z\left(\beta \right)}.$$

(49)

Using the change of variables in Equation (33) and Proposition 1, we find the solution of the MEP problem in [5] with respect to the variables $R(n,ϵ)$

$$R(n,ϵ)=\frac{N_0}{Sn}P(n,ϵ)=\frac{e^{-\lambda n}}{Z\left(\lambda \right)}\frac{e^{-\beta ϵ}}{Z\left(\beta \right)}$$

(50)

which coincides with $Q(n,ϵ)$ in (30).

6. Discussion

A non trivial task in the application of the MEP is the translation in mathematical terms (i.e., as constraints on the sought probability distribution) of the information available on the system. In some cases one may hesitate between mathematically equivalent formalizations. What is not so well known is that the use of Shannon entropy $H\left(p\right)$ may not be justified in all cases. We have provided a criterion based on Boltzmann method of the most probable occupation vector to derive the correct form of the entropy function in a given formulation. Note that the problem addressed here is different from the search of a suitable prior distribution for the relative entropy $D\left(p\right|q)$. In the second part of the paper we have applied this analysis to critically examine the Maximum Entropy Theory of Ecology (M.E.T.E.). Even if the application of MEP contained in M.E.T.E. seems flawed, the resulting SAD proposed by M.E.T.E. is the Fisher log-series, which is widely accepted in the ecological community and considered as giving a best fit for many ecological communities (although the use of log-series SAD has been questioned recently, see [21]). Therefore, the previsions of species abundance based on M.E.T.E. may well be in good agreement with field data. What seems to be hardly giving a sound base in M.E.T.E. is the claim that the information contained in $N_0,S,E_0$ produces a coupling between the distribution of abundance between species and the distribution of metabolic energy between individuals.