1. Problem Formulation and MaxEnt Rationale
In the context of testable information that is, when a statement about a probability distribution whose truth or falsity is well-defined, the principle of maximum entropy states that the probability distribution which best represents the current state of knowledge is the one with largest entropy. In this spirit, Maximum Entropy (MaxEnt) methods are traditionally used to select a probability distribution in situations when some (prior) knowledge about the true probability distribution is available and several (up to an infinite set of) different probability distributions are consistent with it. In such a situation MaxEnt methods represent correct methods for doing inference about the true but unknown underlying distribution generating the data that have been observed.
Suppose that
X be an absolutely continuous random variable having probability density function (pdf)
f defined on an unbounded support
and that
, with
, be
M finite integer moments whose values are pre-determined that is,
for an arbitrary
. Quantities such as in (
1) may be intended to represent the available (pre-determined) information relatively to
X.
The Stieltjes (Hamburger) reduced moment problem [
1] consists of recovering an unknown pdf
f, having support
(
), from the knowledge of prefixed moment set
.
Due to the non-uniqueness of the recovered density, the best choice among the (potentially, infinite) competitors may be done by invoking the Maximum Entropy (MaxEnt) principle [
2] which consists in maximizing the Shannon-entropy
under the constraints (
1). Since entropy may be regarded as an objective measure of the uncertainty in a distribution, “... the MaxEnt distribution is uniquely determined as the one which is maximally non-committal with regard the missing information” ([
2], p. 623) so that “...It agrees with is known but expresses maximum uncertainty with respect to all other matters, and thus leaves a maximum possible freedom for our final decisions to be influenced by the subsequent sample data” ([
3], p. 231). In other words, the MaxEnt method dictates the most "reasonable and objective" distribution subject to given constraints.
More formally, in such situation we have to manage a constrained optimization problem involving Shannon entropy and a set of given constraints (here the first
M integer moments and the normalization constraint given by (
1) when
).
This problem is typically solved using the method of Lagrange multipliers leading to a MaxEnt distribution whose density function
is given by [
4], p. 59
fulfils the given constraints
since
and has entropy
where
and
From now on, for sake of brevity, we will write each member f of omitting the dependency on , predetermined set of moments; hence, f and will stand for and , respectively. The same will be done for the corresponding entropies: we will write and in place of and .
A few words about our notation are now opportune. Since in the sequel an arbitrary moment may play different roles, we establish to use
for prescribed moments;
for variable (free to vary) moments;
for the j-th moment of , that is (in general )
to indicate the smallest value of , once are prescribed.
Our attention is solely addressed towards sequences whose underlying density f has finite entropy . More precisely, only distributions with are not considered. Indeed, once is assigned, is not feasible, as it is well known in MaxEnt setup that is finite because Lyapunov’s inequality (Hamburger case) and is finite for every (Stieltjes case).
Here
is the vector of Lagrange multipliers, with
to guarantee integrability of
. If it is possible to determine Lagrange multipliers from the constraints
then the moment problem admits solution and
is MaxEnt solution (which is unique in
S due to strict concavity of (
4)).
The above non negativity condition on which is a consequence of unbounded support , is crucial and renders the moment problem solvable only under certain restrictive assumptions on the prescribed moment vector . This is the ultimate reason upon which the present paper relies.
The existence conditions of the MaxEnt solution
have been deeply investigated in literature ([
5,
6,
7,
8,
9] just to mention some widely cited papers); over the years an intense debate—combining the results of the above papers—has established the correct existence conditions underlying the Stieltjes and Hamburger moment problem (more details on this topic may be found in the
Appendix A).
On the other hand, when the existence conditions for are not satisfied, the non-existence of the MaxEnt solution in Stieltjes and Hamburger reduced moment problem poses a series of interesting and important questions about how to find an approximant of the unknown density f least committed to the information not given to us (still obeying to Jaynes’ Principle). This problem is addressed the present paper.
More formally, take
to be the set of the density functions satisfying the
moment constraints (that is, they share the same
predetermined moments) and let
be the moment space associated to
; hence, the indeterminacy of the moment problem (
1) follows.
A common way to regularize the problem, as recalled before, consists in applying the MaxEnt Principle obtaining
, the set of MaxEnt densities functions which is a subset of
; consequently, let
be the moment space relative to the set of MaxEnt densities functions
. Because, in general,
strictly includes
there are admissible moment vectors in
, the interior of
, for which the moment problem (
1) is solvable but the MaxEnt problem (
3) has no solution and the usual regularization based on MaxEnt strategy is therefore precluded.
The implications of such issue are often understated in practical applications where the usual procedure limits itself to:
In the Stieltjes or symmetric Hamburger cases: to replace the support
(
) with an arbitrarily large interval
. As a consequence of it, the problem is numerically solved within a proper interval
, changing the original Stieltjes (Hamburger) moment problem into Hausdorff one. In the MaxEnt setup, the latter admits a solution for each set
([
7], Theorem 2).
If
does not exist (conclusion drawn uniquely from numerical evidence),
(Stieltjes case) or
(symmetric Hamburger case) always exist (see
Appendix A). In such a case, although the first
M moments are known, we have to settle for a density constrained by
or
. However, this is not completely coherent with the MaxEnt principle that prescribes to use not only the available but all the available information; hence discarding available information seems to be conceptually in contrast with the MaxEnt spirit. However, from the point of practical applications, to consider or not to consider the prefixed moment
seems to have negligible effects on the summarizing quantities of the underlying distribution (mostly expected values of suitable functions) in which we may be interested in. We will resume this issue, after having carefully motivated and proved the proposed solution, in the last section of the paper devoted to discussion and conclusions.
We call the solutions 1. and 2. “forced” pseudo-solutions; they might indeed lead to the unpleasant fact that a MaxEnt solution always exists, although the original Stieltjes (Hamburger) moment problem does not admit any solution. Hence the crucial question is: does there exist a way to regularize the (indeterminate) moment problem (
1) coherently with all and only the available information exploiting the MaxEnt rationale setup without forcing to unnatural solutions, i.e., based on totally inappropriate application of the MaxEnt principle?
Before proceed recall
and define the following class of density functions:
with
, whose entries satisfy the given constraints expressed in terms of
assigned integer moments
.
Now the question is: once
are pre-determined (that is, the MaxEnt problem does not admit solution) what is the optimal choice of the pdf that we can select in place of
? Relying upon the MaxEnt rationale, the best substitute of the missing
should be given by suitable one
having the overall largest entropy; that is, select
actually satisfying the relationship
for an arbitrarily small
.
We are aimed to find
,
and the corresponding entropy
, proving that it may be accomplished by MaxEnt machinery (see Equations (
9)–(
11) below).
The remainder of the paper is organized as follows.
Section 2 and
Section 3 are devoted to evaluating the best pdf in Stieltjes and Hamburger cases respectively. We devote
Section 4 to numerical aspects and in
Section 5 we round up with some concluding remarks. In
Appendix A. the existence conditions of MaxEnt distributions in Stieltjes and Hamburger case are shortly reviewed.
2. Stieltjes Case
Let us consider ; consequently does not exist. In this section we provide a formal justification about motivation (rationale) and optimality of the proposed substitute of the MaxEnt density . We deal with the issue of selecting the "best” pdf both satisfying the constraints (given by predetermined integer moments) and with the overall largest entropy.
Before start, some relevant facts need to be collected together. Since MaxEnt density
does not exist both
with its
M-th moment
and
exist; the latter exists for any value
(see
Appendix A for more details).
Since the procedure here adopted remains valid for each value , as from Lyapunov’s inequality, we have and consequently too. As well, since MaxEnt density does not exist, some additional information not given to us must be added; of course, is the most suitable candidate to represent it.
Once this is established, the relevant question is: what value for
? Recalling that
is monotonic increasing ([
4], Equation (2.73), p. 59), with upper bound
so that
exists,
should assume the overall largest value, so that the decreasing of entropy is as small as possible.
Since
,
and then
are meaningless,
includes infinitely many
f and
must be calculated. The moment set
is considered too, where
is the
M-th order moment of
and
varies continuously within
with
If
is density corresponding to the set of moments (
8), the following theorem holds.
Theorem 1. The following two relationships holdand Now, is identified with where is such thatand ε indicates a fixed tolerance. Proof. If does not exist, exists with entropy and M-th moment respectively. The function , with , is monotonic increasing. As
one has . The latter represents the maximum attainable entropy once are prescribed;
, from monotonicity of it follows independent on . Equivalently, is strict upper bound for the entropies of all densities which have same lower moments as but whose highest moment exceeds .
Hence Equation (
9) is proved.
Let us now consider the suitable class
where
is assumed as parameter and
is the
M-th order moment of
. Equivalently, the entries of
are MaxEnt pdfs constrained by
, belong to
and, primarily, they all have analytically tractable entropy.
In (
5), (
6) and (
12) three classes of functions
,
and
had been defined. Relying upon the identity
we investigate the entropy
of functions
f belonging to (a)
, (b)
and (c)
respectively.
Consider that
, bounded by
from above, is a differentiable monotonic increasing function of
and then it tends to a finite limit so that
Each
has entropy
and its
-th finite moment, say
. Since
f and
share same moments
then
holds, from which
In analogy with (
12), let us introduce the following class
where
assumes arbitrary values. For a fixed
, each
satisfies the following inequality
Taking
,
coincides with
and then
Collecting together both the achieved results in above items (a), (b), (c) and taking into account (
9) one has
Hence Equation (
10) is also proved.
Equation (
10) is restated as follows: if
indicates a fixed tolerance, there exists a value
of
such that
holds. Next
is identified with
so that its entropy
coincides with
. From which the wanted result
(or, equivalently (
7)) follows. As a consequence
is the proposed substitute of
and Equation (
11) is proved. □
In conclusion:
4. Numerical Aspects
The procedure just above described and rooted on MaxEnt machinery suffers from some numerical drawbacks which will be here discussed. It deserves to recall similar drawbacks had been previously found ([
11,
12]) although for the special value
in Hamburger case, exploring special regions of the moment space. Essentially, numerical troubles arise because the expected solution
is contaminated with a small wiggle that (a) moving to infinity, (b) is scaled in such a way that its contribution to the (
M + 1)-th order moment
is always
and (c) may become invisible to numerical methods of quadrature.
Now we provide some theoretical ground to justify above heuristics, which holds true in both Hamburger and Stieltjes case thanks to MaxEnt formalism. First of all, under the constraints , we prove the wiggle exists. At this purpose both the relationships for each and as have to be proved.
for each
. From Appendix we recalled if
exists with its
M-th moment
, MaxEnt
does not exist if
. Let us consider
where
varies continuously. Then
is monotonically decreasing ([
9], Equation (2.1)) with
as
. As a consequence no set
satisfies the constraints (
) since the monotonicity of
would require
. Let us consider
where
varies continuously. Here, for each
,
guarantees integrability, so that
may assume every real value. Collecting together the results about
, the set
satisfies the constraints
for each
. Equivalently, we can assert
are appointed to meet
, whilst
to meet
.
as
. Differentiating (
4) with respect to
and recalling the relationship ([
9], Equation (2.1))
one has
From Theorem 1 we proved, as , , so that and then too.
We are ready to prove the statement concerning the fact that
exhibits a small wiggle at
(analogously, in symmetric Hamburger case the wiggle is exhibited at
). At
so that
admits maximum value at
As increases we proved the relationships and , so that moves to infinity (from numerical evidence, as increases, too much slower than ). Since has finite moments for each , it follows the wiggle in a compact packet is scaled in such a way that its contribution to the (M + 1)-th order moment is always (whilst for all higher moments the contribution due to this maximum obviously grows without bound, as a consequence of Lyapunov’s inequality).
An additional complication comes from the fact that height and position of wiggle is extremely sensitive to the parameters and , so that it becomes progressively smaller and smaller until to be “invisible” if an unsuitable numerical method of quadrature is adopted. As a consequence the procedure becomes increasingly ill-conditioned to such a degree that numerical error precludes finding a suitable solution. As remedy, for instance, the quadrature on the unbounded domain has to be mapped onto finite interval, as well an adaptive quadrature is required. Since the wiggle moves along x-axis as increases, a fixed nodes quadrature formula could be unsuitable as the wiggle could become invisible for some values of .
Above remedies are just a numerical trick, not a reduction of Stieltjes or Hamburger problem into Hausdorff one. Indeed, all the subsequent numerical examples consider and use random variables X having unbounded support or .
As well the dual formulation, which evaluates
minimizing the potential function
avoids the computation of higher moments, as required by Newton-type methods by solving (
3).
The drawbacks just illustrated lead us to equip the stopping criterion (
7) based on entropy with a further one based on the moments, which allows us the relationship
holds true. That is,
(or involving the absolute error) for a proper
.
The following question arises: it is
, here identified with
and
is chosen so that stopping criteria (
7) and (
13) are verified, an acceptable approximation of underlying unknown density? Although the wiggle has non-physical meaning, nevertheless from the approximate density one like to calculate accurate and interesting quantities. We will resume the issue in the final part of the paper.
For practical purposes in both Stieltjes and Hamburger case
is calculated according to (
9)–(
11) uniquely by means of MaxEnt machinery following these two distinct steps
Before to illustrate some numerical examples that confirm the goodness of the proposed method, it is worth spend some words discussing the outlined procedure. The calculation of is obtained through an approximate procedure and hence has a limited range of applicability. The main problem is the presence of wiggles; at the end to contain their detrimental effect it is necessary that convergence of to be fast. For example, the value that precludes the existence of in the Stieltjes case, must be such that the difference be small. Larger values make the convergence of to slow, allowing the generation of a small wiggle at great distance. The latter may become invisible to numerical quadrature methods.
Below are some numerical examples which both take into account the above remarks about the difference and illustrate the theoretical and numerical aspects mentioned above.
Example 1. The Stieltjes case with and prescribed is considered. Now exists if and only if the inequality holds ([5], Theorem 2). The moment set represents an additional boundary in . If the moments satisfy the reverse inequality there is no pdf which maximizes the entropy. We consider the latter case taking and ; then is calculated by means of (
11)
, with . Values of entropy with increasing values of not reported here lead to conclusion that the entropy stabilizes rapidly as increases. This may be an evidence of high accuracy in the reconstruction. Taking , Equation (
11)
is satisfied starting from . Then , which is identified with , jointly with are displayed in Figure 1 (top). The difference between and is insignificant since was chosen to avoid the dentrimental effect of wiggle. In Figure 1 (bottom) the same , on a logarithmic scale and on extended x-axis scale, is reported to evidenciate the presence of small wiggle. The moments , satisfy , , respectively. It can be concluded satisfies all the expected theoretical properties and can be considered the "best" substitute of the missing . Example 2. Hamburger case with ; here , with the skewness = 0.5, are assumed. MaxEnt density does not exist if the first 3 moments are specified and the skewness is required to be non-zero. Then does not exists whilst (Normal distribution) exists, with entropy . Taking , Equation (
11)
is satisfied starting from . Then , which is identified with , jointly with are displayed in Figure 2 (top). In Figure 2 (bottom) the same is displayed but on a logarithmic scale and on extended x-axis scale, to highlight the presence of small wiggle. The moments , , satisfy , , , respectively. It can be concluded satisfies all the expected theoretical properties and can be considered the "best" substitute of the missing .
Remark 1. It is worth to note that the nonsymmetric Hamburger case with has been discussed in [13], pp. 413–415, Equation (12.32), but solely on the basis of a simple heuristic reasoning; they use a tricky problem to observe that even if the Lagrange multipliers cannot be chosen to satisfy the given constraints, the “maximum” entropy can be found and it is equal toconcluding that in this situation the entropy may only be ϵ-achievable. Just to give a simple example of it, but not a formal justification, the authors consider the case in which a Normal distribution be contaminated with a small “wiggle” at a very high value of x; consequently the moments of new distribution are almost the same as those of the non contaminated Normal, the biggest change being in the third moment (the new distribution is not any more symmetric). However, adding new wiggles in opportune positions to balance the changes caused by the original wiggle we can bring the first and the second moments back to their original values and also get any value of the third moment without reducing the entropy significantly below that of the associated non contaminated Normal (from this the conclusion about the ϵ-achievability of the entropy). Just above heuristic procedure is displayed in Figure 2, and interpreted saying may be identified with the Normal distribution on which some wiggles are superimposed. This result is a particular case of the more general result covered by this paper and coincides with the above (
9)–(
11)
when, in this case, is the density function of a Normal distribution. Lastly, all above heuristics agrees with the mathematical general result that two continuous density functions having the same first moments (including ) cross each other in at least points ([14], Vol.1, No. 140, p. 83). In our case and the Normal density plotted in Figure 2, share the first moments and they cross each other at three points as the inspection of the previous figure suggests. Example 3. Symmetric Hamburger case with , prescribed and MaxEnt density are considered. is the Normal distribution with and entropy . does not exists, being its existence condition not verified. A further even moment with increasing values is added and has to be calculated.
Taking , Equation (
11)
is satisfied starting from . Then , which is identified with , jointly with are displayed in Figure 3 (top). In Figure 3 (bottom) the same , in logarithmic scale and on extended x-axis scale, is reported to highlight the presence of two symmetric wiggles travelling in opposite direction and illustrated too in same Figure (bottom). The moments , satisfy , , respectively. In each of the previous three examples we have assumed that and differ from a small amount and this to avoid the detrimental effect due to the wiggle; consequently, the difference between and becomes insignificant too. As a result,
- 1.
The convergence of to is fast and avoids the formation of small evanescent wiggles at a great distance;
- 2.
The rise of numerical quadrature problems.
As a consequence the two densities and are almost superimposed precluding the possibility to evaluate the effect produced in from having discarded . It would be interesting to be able to assess how high values may affect the difference . The goal could be achieved by a suitable numerical quadrature method.
5. Discussion and Conclusions
In the present paper, we have discussed the case which arises when, in presence of a prefixed moment set representing the available information, the (reduced) moment problem admits solution but the MaxEnt density as a solution of the regularization problem does not exist. In the previous sections, we have given the conditions under which a solution of the Stieltjes and Hamburger (reduced) moment problems may be found in the genuine Jaynes’ spirit by finding the overall largest entropy distribution which is compatible with the available information and showing that this is the best approximant of the underlying true but unknown distribution. The substitute of the missing MaxEnt solution is found using solely the usual MaxEnt machinery.
Now we look at the issue from a different point of view. Suppose represent all and only the available information. Two cases 1. and 2. may present:
Only the first
M moments may be measured but additionally the
exists. In this situation the traditional MaxEnt machinery will produce the usual solution
which has a well known analytical form corresponding to the Jaynes’ non committal approximant (MaxEnt) of the underlying
f (see Equation (
2));
Only the first
M moments may be measured but additionally the
does not exist. Here any information about the analytical form of the substitute of the missing MaxEnt solution is lacking. If only the first
M moments may be measured, it is reasonable to assume the underlying
f admits the first
M moments solely. Then it can to be restated
to represent all and only the available information. Next, assuming
takes finite value, MaxEnt machinery may be invoked, from which
as above and the consequent Theorem 1. To find a genuine minimal committal approximant in the MaxEnt spirit of the underlying
f just
is taken, so that, from the monotonicity of
, the spurious information represented by
has a minimum effect on the approximant (in other terms, to guarantee to be minimal committal).
The solution we have proposed in this paper for case 2. offers an alternative and exhaustive answer to the common empirical “forced” practices consisting in
- (a)
Replacing an unbounded support with an arbitrarily large interval, or
- (b)
Neglecting the prescribed higher moment so that the reduced number of moments allows the existence of MaxEnt solution.
As we have widely said before (see Introduction), solutions like (a) and (b) imply a forced pseudo-solution of the original problem which conflicts with MaxEnt rationale.
The above conflict is not merely theoretical and it has some practical consequences. This leads us to distinguish theoretical and practical aspects of the procedure we proposed. MaxEnt technique is invoked because one reputes the found distribution to be “the best” and the obtained results are “the best”. Essentially this is the practitioner’s main concern. More specifically, since the MaxEnt distribution constrained by first M moments does not exist, we are inclined to turn to . Depending on whether is considered or not considered, or will be used to approximate the unknown underlying density f. It may happen that some summarizing quantities based on different approximations of f as or , remain unaltered as we illustrate in next few rows.
If
g is a bounded function of
X,
and
lead to similar values, as Pinsker’s inequality ([
15], p. 390) and (
11) yield,
As a consequence, although we settle for a density constrained by fewer moments, and then conceptually in contrast with the MaxEnt spirit, nevertheless the results remain unaltered.
The matter runs similarly whether quantiles have to be calculated. They may be configured as expected values of proper bounded functions: indeed, for fixed
x,
with
if
and
if
. Then, if
and
denote the distribution functions corresponding to
and
, respectively, we have in Stieltjes case (and mutatis mutandis equivalently holds for Hamburger case)
Again, although we settle for a density constrained by fewer moments, and this goes conceptually against the spirit of Jaynes, nevertheless the results concerning expected values of g remain unaltered. However, if g is an arbitrary unbounded function of X then the sequence of the above inequalities does not hold and the calculation of expected values of g could lead to different results, i.e., .
In conclusion: if the maximum entropy distribution does not exist, being guided by the spirit of maximum entropy could always turn out to be the best choice.