1. Introduction
When studying probability distributions, one of the challenging questions we arrive at comes from the classical moment problem. The question is whether or not a probability distribution is uniquely determined by the sequence of all moments, assuming they are finite. The answer can be given for the distribution itself; equivalently, for the associated random variable X; its distribution function F; the density ; or the bounded positive measure induced by F. Thus, if the answer is positive, we call the distribution (also ) M-determinate; otherwise, we call it M-indeterminate. (Here, ‘M’ stands for ‘Moment’.)
It is well known that if is M-indeterminate, then there are infinitely many absolutely continuous distributions, infinitely many purely discrete distributions, and infinitely many singular distributions, all having the same moments as .
It is important, from both the theoretical and applied points of view, to have criteria at hand allowing to specify/identify the determinacy or indeterminacy property of a distribution. The best is to work with conditions which are in the group ‘checkable conditions’; comments and references are given at the end of our paper. There is another group of ‘non-checkable conditions’. Here are the well-known classical necessary and sufficient conditions for the (in)determinacy of
in terms of the limits of the smallest eigenvalues of sequences of Hankel matrices. Our recent review paper [
1] describes the whole spectrum, called ‘a bunch’, of the fundamental results, old and recent. The reader will find in [
1] details about the great contributions of T. Stieltjes, H. Hamburger, N. Akhiezer, M. Krein, C. Berg, K. Schmüdgen, M. Putinar, B. Simon, and others. Their works are widely known.
Developments over the last few decades have shown the efficiency of involving the
Principle of Maximum Entropy, see, for example, [
2,
3,
4]. We also use the terms ‘maximum entropy approach’ and ‘maximum entropy method’. For ‘maximum entropy’, we write the traditional ‘MaxEnt’.
The idea of the MaxEnt method consists in selecting a distribution which possesses maximum uncertainty, and at the same time, fulfills the restrictions imposed by the known information.
In general, it is more delicate to deal with M-indeterminate distributions, since we need, for example, to know how to find, describe and work with an infinite family of distributions all having the same moments. In any case, MaxEnt may help to shed light on this ‘dark tunnel’.
In this paper, we follow the generally accepted terminology and notations as used in probability theory. We write for a random variable X whose distribution function is F, with being the density, and specify the range of values of X, the support of F, which is assumed to be unbounded. Only in this case can the ‘interesting’ property of M-indeterminacy appear. We work with the moment sequence , and if , this is a Hamburger case, while with it is a Stieltjes case.
For
X being an absolutely continuous random variable with strictly positive density
f, we are looking for conditions, or criteria, guaranteeing the M-determinacy or M-indeterminacy of
. We use the entropy (called also ‘differential entropy’), which is denoted by
and defined as follows:
The idea is very natural: We start with the n-truncated moment set and based on it, we find the MaxEnt approximant of f and study the limit of the entropy of as
There is a remarkable fact, namely, that there are only two possibilities for the ‘value’ of the limit ; either it is a finite number, or it is ‘equal’ to . Depending on this limit, we decide that f is M-determinate or M-indeterminate.
It is relevant to mention one of the results proved in ([
5], Theorem 1): if an absolutely continuous distribution
F with density
f is M-determinate, then the sequence of MaxEnt approximants is converging in entropy to
f. One of our goals in this paper is to involve additional arguments allowing to show that such a result on entropy convergence can be extended to the case of M-indeterminate distributions.
The remainder of the paper is organized as follows. In
Section 2, we recall briefly what we need about Hankel matrices and introduce the MaxEnt setup. In
Section 3, we calculate the entropy of densities with the given
n-truncated moment set, for fixed
n, and also for the entire moment sequence
In
Section 4, we provide an M-indeterminacy MaxEnt criterion in the Stieltjes case. In
Section 5, we present corollaries related to the M-indeterminacy in the Hamburger case. Discussed is the question: Among a family of infinitely many densities all with the same moments, which density has the largest entropy?
2. Basics of Hankel Matrices and the MaxEnt Setup
When we tell that with is a moment sequence, it always means that there is a probability measure which is ‘behind’. Thus, think of a random variable X defined in an underlying probability space , taking values in a set . If F is its distribution function, , then is a positive Borel measure induced by F. We write just
A basic assumption is that
for all
Thus, well defined are the moments
and also the moment sequence
If
, we say that
is a
Hamburger moment sequence, while for
,
is a
Stieltjes moment sequence.For any moment sequence
, we define a few infinite sequences of
Hankel matrices, namely,
and
, and their
determinants, as follows:
If , is the ‘basic’ Hankel matrix, for is the ‘shifted’ Hankel matrix: is based on the ‘shifted’ moment sequence generated by the measure with .
In what follows, we involve and use the entropy of the strictly positive density f under the constraint of knowing only the n-truncated moment set We will see that the MaxEnt formalism allows to study in parallel both the Hamburger and the Stieltjes cases; hence, we assume that the distributions and their densities have support or .
Let us consider the Stieltjes case. For a density f with n-truncated moment set there is a density, say satisfying two properties:
- (a)
The ‘first’ n moments of are exactly
- (b)
maximizes the Shannon entropy.
It is well known, see [
2], that
where
are the Lagrange multipliers satisfying the constraints
In this case, we use the simple notation
for the entropy of
and remember that
depends on the moments
. It is easy to see that
We would like the sequence of approximants
and the entropy sequence
to be well defined for any
It may happen, see ([
6], Theorem 1), that for given
f and
, the desired density
does not exist, in which case the quantity
is meaningless. However, in the cited paper, the following useful relation is established (the class
is defined at the end of this section):
, even if the MaxEnt approach does not apply. Since the entropy is monotone and non-increasing as
n increases, the latter equality enables us to simply set
, thus filling the ‘gap’ left by the non-existing densities
. This justifies the assumption made in the sequel, without loss of generality, that all entries of the monotone non-increasing entropy sequence
are well defined.
In the non-symmetric Hamburger case, once the
n-truncated moment set
for even
n is assigned, the positivity of the Hankel determinant
guarantees the existence of a MaxEnt solution, see ([
6], Appendix A). As a consequence, the entire entropy sequence
is defined. In the symmetric Hamburger problem, the MaxEnt density existence is guaranteed under conditions similar to those in the Stieltjes case.
Now suppose that
is a moment set for which we ‘keep fixed’ (unchanged) the moments
while we treat as ‘varying continuously’ the moment
. If letting
, then the
-truncated moment set can be written as
. Moreover, the existence conditions for a solution of the moment problem require the Hankel determinants to be positive. This is guaranteed by imposing
b to have a lower bound, say
, which is the unique real number satisfying the equation
As well, due to the MaxEnt machinery, see ([
6], Appendix A), in the Stieltjes case, the following value of
b has to be considered:
Notice that in general,
. Recall that we deal with the truncated
-moment set in which the parameter
b stands for the
th moment:
In the Stieltjes case, we introduce the following classes of densities:
Similar notions can be introduced also in the Hamburger case, just replacing with .
The class is a convex set for each n, and then is also convex. We know that in the M-indeterminate case, contains ‘infinitely many’ densities, all being solutions of the same moment problem.
For both the Hamburger and Stieltjes cases, we need to recall a few known facts which will be essentially used later.
Fact 1. We are going to work the moment sequence whose underlying density f has entropy such that either is finite, or . To be precise, distributions with are not allowed. The reason for this is that once is assigned, the ‘option’ is not feasible since it is well known in the MaxEnt setup that , where is finite because of Lyapunov’s inequality (Hamburger case) and is finite for every (Stieltjes case).
Fact 2. Once the moment set is given and is the corresponding MaxEnt density, the entropy sequence is monotone non-increasing, and its limit is either finite or
Fact 3. For consistency between the differential entropy of a continuous random variable and the entropy of its discretization, the differential entropy of any discrete measure which can be compared with Dirac’s deltas set is assumed to be
see ([
3], pp. 247–249).
Fact 4. If the density
f is bounded, this is sufficient to eliminate the option
. Indeed, suppose that
for all
x. It follows that
3. Entropy of Densities Which Are M-Indeterminate
The MaxEnt formalism allows to treat both Hamburger and Stieltjes cases in a similar way. For the sake of brevity, we confine ourselves mainly to discussions on the Stieltjes case. All arguments can then be easily extended to the Hamburger case. This possibility is one of the advantages of involving the MaxEnt machinery.
3.1. Entropy of Densities from the Class
We start with the formulation and the proof of the following result.
Theorem 1. Suppose that is the full moment sequence of a given density For fixed n, based on the n-truncated moment set we consider the MaxEnt
approximant of f, and let be the entropy of . Then, there are infinitely many densities whose entropy is spanning an interval, namely Proof. We provide arguments in both cases, Stieltjes and Hamburger.
(a)
Stieltjes case. Preliminarily, for fixed
let us consider
and the upper bound
of its
th order moment. It was mentioned that, in general,
is different from
. Our goal is to specify the range of values of the entropy
, where
g is an arbitrary density from the class
. For this, we introduce the following suitable subclass
Notice that we rely here on the specific truncated moment set , the boundary of the moment space. Equivalently, the elements of are MaxEnt densities which are constrained by ; they belong to and, primarily, they all have analytically tractable entropy. The latter property enables us to calculate the entropy of all by evaluating the entropy of all
Let us consider for b varying in the interval and calculate the values spanned by the entropy .
Subcase a1. If
, the right-end point, it is easy to verify that
has a Lagrange multiplier
and hence
coincides with
; hence,
Subcase a2. If
b is ‘close’ to the left-end point, i.e.,
, we look at the Hankel determinants
and
and see that either
or
. This implies that the underlying measure
is discrete; see, for example, ([
7], Theorem 1.3, p. 6). Therefore, the entropy quantity
is approaching
:
It remains to mention an essential property of the entropy
as a function of the variable
b. Since
see ([
2], Equation (2.73), p. 47), or the arguments below, we have that
is monotone increasing with respect to
.
(b)
Hamburger case. The arguments here are similar to those above. We need to replace
by the following one with analogous meaning of all notations:
Here,
and
are such that
In such a case, it is easy to see that
and hence, just as above, we conclude that
satisfies the entropy relation
.
Joining together (
2) and (
3) (with obvious extension to the Hamburger case) with
or
and referring to the monotone increasing of the entropy with respect to
b, we conclude that indeed there are infinitely many densities
and
whose entropy spans the interval in (
1), with this property holding for all
. Theorem 1 is proved. □
3.2. Entropy of Densities from the Class
Among the well-known properties of Shannon’s entropy, we use its concavity as a functional, which implies that the entropy of all densities can be calculated.
We start with the Stieltjes moment sequence and calculate the entropy sequence , which is monotone, non-increasing and convergent. Similarly, for a Hamburger moment sequence , we calculate the entropy sequence , being also monotone, non-increasing and convergent.
Let us show first that there exists only one density, say,
such that
has the largest entropy, i.e.,
Indeed, the set of solutions to the S-indeterminate moment problem includes infinitely many densities, previously grouped in the convex set . On the other hand, the continuous entropy functional is strictly concave and, over the convex set , attains its maximum value. Hence, we have that the optimization problem to maximize over has indeed a unique solution such that
Relying on Theorem 1, we are ready to calculate the entropy of all moment equivalent densities . We keep in mind, all densities in the class have support in the Stieltjes case and in the Hamburger case.
Theorem 2. Suppose that is the full Stieltjes moment sequence of a density f and it is known that f is M-indeterminate. We use and as before. Then, there are infinitely many densities whose entropy is spanning an interval, namely, Proof. Note first that each
satisfies
; hence, according to Theorem 1,
g has entropy
This implies that
which completes the proof. □
We use below, for example, S-determinate or H-determinate, meaning that a density is on or on , and it is M-determinate or H-determinate. This similarly applies for S-indeterminacy and H-indeterminacy.
In general, it is not easy to establish the S-determinacy, and hence S-indeterminacy, through known criteria based on necessary and sufficient conditions. The existence of the density with the largest entropy, see Theorem 1, indicates that there is some similarity between the M-determinate and M-indeterminate cases. Consequently, since initially a finite set of moments is involved, the technique of density reconstruction through the MaxEnt approach can be applied without distinguishing these two cases.
With , all as above, we give here some details.
First, if
f is S-determinate and H-determinate, the sequence of approximants
converges in entropy to a unique underlying density
f, see ([
5], Section 3), that is,
as
with
all being finite. However, from the used procedure, relying on the geometrical meaning of Theorem 2.19, p. 72 in [
7], it is immediate to deduce that the statement of convergence in entropy is equally extended to the case where
, from which
Second, if
f is S-indeterminate, the entropy sequence
is monotone non-increasing and hence convergent with lower bound
, i.e.
It is useful to mention that Theorem 2 and the comments completely agree with the rationale of the MaxEnt approach: when all known information has been taken into account, a system with maximum entropy is the most probable state because it is the system in which the least amount of information has been defined.
Moreover, Theorem 2 justifies the approach of reconstruction of the density f, starting from a finite set of moments and passing to the full moment sequence, regardless of the M-determinacy or M-indeterminacy of f. In any case, that issue is not really of great practical significance. In fact, a full set of moments will ‘never’ be available; hence, for practical purposes, we deal only with finite n, which is perhaps ‘big enough’. Nevertheless, is a valuable approximation of since both and have the same first n moments and This fact also corresponds well to the MaxEnt rationale. Thus, the question of moment (in)determinacy of the density f is not essential for the procedure we follow.
4. Stieltjes Case: MaxEnt Criterion for M-Indeterminacy
We deal with a random variable X on with finite all moments . Recall that We mentioned in the Introduction one fundamental fact: if the distribution F is M-indeterminate, then there are infinitely many distributions of any kind, all having the same moments as F.
Recall that a Stieltjes moment sequence can also be considered a Hamburger moment sequence, i.e., it is coming from a random variable in . We always have to make a distinction between M-determinacy and M-indeterminacy by specifying that it is in the sense of Stieltjes, or in the sense of Hamburger. We use below the obvious terms, S-determinate, S-indeterminate, H-determinate and H-indeterminate, in their short forms, S-det, S-indet, H-det and H-indet. Let us list the possibilities for the distribution F:
If F is S-indet, it is also H-indet. If F is H-det, it is also S-det.
If F is H-indet, then either F is also S-indet, or, it may look a little ‘surprising’, F is S-det.
Thus, we have three cases; they will be discussed below. Relying on the results in
Section 3, we provide now a MaxEnt criterion for M-indeterminacy in the Stieltjes case.
Theorem 3. (Main result.) Let f be a probability density with finite all moments. Denote by its full moment sequence and its nth truncated set. If is considered as a Stieltjes moment sequence, we write for the MaxEnt approximant of f based on Similarly, will stand for the MaxEnt approximant of f based on if considering as a Hamburger moment sequence. For the entropy, we use the notations and
The Stieltjes moment sequence corresponds to the moments of infinitely many distributions on ; equivalently, the distribution F is M-indeterminate, if and only if the following relation holds: Proof. First, we recall the well-known result according to which if
n is odd, the estimator
does not exist. Since the entropy is monotone and non-increasing as
n increases, it is proved in [
6] (Section 3.2) that the entropy quantity
can be associated with
in the sense that
. Thus, the sequence
will be well defined for each
n, filling up the ‘initial’ gap left by the odd moments. Furthermore, the inequality
is meaningful for any
n since both
and
are based on the same constraints, the moment set
, whilst
has an additional constraint, namely, the support is
.
Now we consider the three possibilities mentioned above. In brackets, we write what is F first, and what is second.
Case 1. [
F is S-indet and H-indet] We refer to relation (
4) from which it follows that
holds.
Case 2. [
F is H-indet and S-det] We recall that the unique solution, a measure, with positive support is a Nevanlinna extremal solution whose spectrum contains 0 and is a discrete unbounded subset of
see ([
8], Remark 2.2.2, p. 178). Then, as quoted before,
is finite and
Case 3. [F is H-det and S-det] Clearly, one solution solely supported on exists. As a consequence, for the limit we have that either L is finite or L is ‘equal’ to . If L is finite, the distribution F is absolutely continuous with either bounded or unbounded density. If , F is either absolutely continuous with unbounded density, or it is discrete.
It remains to show that the converse statements, call them Case 1c, Case 2c, and Case 3c, are also true. We show this by contradiction.
Indeed, in Case 1c, if holds true, then both cases ‘H-indet with S-det’ and ‘H-det with S-det’ are not possible. This is because they respectively require both and . These arguments show that F is H-indet and S-indet. The arguments to prove Case 2c and Case 3c are similar. □
It is worth mentioning that the criterion for M-indeterminacy established in Theorem 3, the Stieltjes case, cannot be extended to the Hamburger case. Indeed, from both Case 2 [H-indet with S-det] and Case 3 [H-det with S-det], the condition ‘finite lower bound ’ does not distinguish the H-indeterminate case from the H-determinate case. Nevertheless, from Theorem 3, some useful corollaries concerning Stieltjes or Hamburger easily follow.
Corollary 1. A necessary condition for the distribution F to be S-indeterminate with H-indeterminacy is for both quantities and to be finite.
Corollary 2. A sufficient condition for the distribution F to be H-determinate is that the quantity is ‘equal’ to .
Notice that in the Stieltjes case, Theorem 3 provides also a sufficient condition to guarantee the existence of a density, which is equivalent to the absolute continuity property of the distribution F. Similarly, such a condition can be extended to the Hamburger case.
Corollary 3. Let be a strictly positive definite Stieltjes moment sequence which corresponds to the moments of exactly one distribution F. If is finite, then F is absolutely continuous with either bounded or unbounded density.
5. Comments on M-Indeterminate Distributions on
Here, we cite a result from ([
9], Theorem 1 and Corollary 1, pp. 100–101); see also ([
10], Examples 11.12 and 11.13) for a general family of distributions.
General Statement. Suppose that F is a distribution function on with moment sequence (Hamburger case). Then, if F is M-indeterminate, symmetric and non-symmetric solutions exist.
Besides the above sources, we can also refer to the notion
Stieltjes class,
, introduced for any M-indeterminate distribution, see [
11]. Recall that
where
is the density of the M-indeterminate distribution
F, and
, called a ‘perturbation function’, is a sign function with norm
, satisfying the ‘vanishing moments’ property,
Another related recent work is [
12].
It turns out that the MaxEnt technique enables us to make a further refinement of what we know about the symmetric solutions, also of the measures, which are M-indeterminate.
Theorem 4. Suppose that F is an arbitrary distribution on with finite moments and moment sequence (Hamburger case) and that F is M-indeterminate. Then, the density , see Section 3.2, with the largest entropy, is symmetric. Proof. Consider an arbitrary non-symmetric . It is easy to verify that is such that . Moreover, g and have the same entropy, i.e., . Consider the densities and for which the entropies and are maximal. They are both in the set . Combining the above general statement with the uniqueness of the MaxEnt density , it follows that ; hence, is symmetric. □
6. Brief Conclusions
In this paper, we establish a new criterion for the M-indeterminacy of a probability density on the positive half-line (Stieltjes case) by involving the MaxEnt approach. Interesting corollaries are derived for probability densities on the whole real line (Hamburger case). The obtained results are new and they can be considered a valuable addition to the results based on two groups of conditions called ‘checkable’ or ‘uncheckable’ for either the M-determinacy or M-indeterminacy of distributions.
The recent review paper [
1] contains a comprehensive description of significant results based on ‘uncheckable conditions’, including two illustrations of how to use this kind of condition as an indication for a specific property of a distribution in terms of its moments. However, from the applied point of view, most useful are the results involving ‘checkable conditions’. The reader is referred to the following sources: [
10,
13,
14]. The property ‘M-indeterminacy’, besides its non-triviality as a mathematical phenomenon, arises in important applied areas. Among them are atmospheric studies, gravity theory and quantum mechanics, see, for example, [
15,
16,
17]. The involvement of the MaxEnt technique may lead to challenging theoretical problems; however, the answers, when available, would shed additional light on the analysis of applied problems.