Stochastic Order and Generalized Weighted Mean Invariance

Abstract

In this paper, we present order invariance theoretical results for weighted quasi-arithmetic means of a monotonic series of numbers. The quasi-arithmetic mean, or Kolmogorov–Nagumo mean, generalizes the classical mean and appears in many disciplines, from information theory to physics, from economics to traffic flow. Stochastic orders are defined on weights (or equivalently, discrete probability distributions). They were introduced to study risk in economics and decision theory, and recently have found utility in Monte Carlo techniques and in image processing. We show in this paper that, if two distributions of weights are ordered under first stochastic order, then for any monotonic series of numbers their weighted quasi-arithmetic means share the same order. This means for instance that arithmetic and harmonic mean for two different distributions of weights always have to be aligned if the weights are stochastically ordered, this is, either both means increase or both decrease. We explore the invariance properties when convex (concave) functions define both the quasi-arithmetic mean and the series of numbers, we show its relationship with increasing concave order and increasing convex order, and we observe the important role played by a new defined mirror property of stochastic orders. We also give some applications to entropy and cross-entropy and present an example of multiple importance sampling Monte Carlo technique that illustrates the usefulness and transversality of our approach. Invariance theorems are useful when a system is represented by a set of quasi-arithmetic means and we want to change the distribution of weights so that all means evolve in the same direction.

1. Introduction and Motivation

Stochastic orders [1,2], are orders defined in probability theory and statistics, to quantify the concept of one random variable being bigger or smaller than another one. Discrete probability distributions, also called probability mass functions (pmf), are sequences of n-tuples of non-negative values that add up to 1, and can be thus interpreted in several ways, for instance, as weights in computation of moments of the discrete random variable described by the pmf, or as equivalence classes of compositional data [3]. Stochastic orders have found application in decision and risk theory [4], and in economics in general, among many other fields [2]. Some stochastic orders have been defined based on order invariance: two pmf’s are ordered when the arithmetic means of any increasing sequence of real numbers weighted with the corresponding pmf’s are ordered in the same direction. This raises the question whether this invariance might also hold for other kind of means beyond the arithmetic mean.

The quasi-arithmetic means, also called Kolmogorov or Kolmogorov–Nagumo means, are ubiquitous in many branches of science [5]. They have the expression $g^{-1}\left({\sum }_{k=1}^M{\alpha }_{k}g\left(b_k\right)\right)$, where $g\left(x\right)$ is a real-valued strictly monotonous function, ${\left\{b_k\right\}}_{k=1}^M$ a sequence of reals, and ${\left\{{\alpha }_k\right\}}_{k=1}^M$ a set of weights with ${\sum }_{k=1}^M{\alpha }_k=1$. This family of means comprises the usual means: arithmetic $g\left(x\right)=x$, ${\sum }_{k=1}^M{\alpha }_k{b}_k$, harmonic $g\left(x\right)=1/x$, ${\left({\sum }_{k=1}^M\frac{{\alpha }_k}{b_k}\right)}^{-1}$, power mean $g\left(x\right)=x^p$, ${\left({\sum }_{k=1}^M{\alpha }_k{b}_k^p\right)}^{\frac{1}{p}}$. For a long time, economists have discussed the best mean for a problem [6]. Harmonic mean is used for the price earning ratio, and power means are used to represent the aggregate labor demand and its corresponding wage [7], and the constant elasticity of substitution (CES) [8]. Yoshida [9,10] has studied the invariance under quasi-arithmetic means with function $g\left(x\right)$ increasing and for utility functions. In information theory, Alfred Rényi [11] defined axiomatically the entropy of a probability distribution as a Kolmogorov mean of the information $1/log{p}_k$ conveyed by result k with probability $p_k$, and recently, Americo et al. [12] defined conditional entropy based on quasi-arithmetic means. In physics, the equivalent spring constant of springs combined in series is obtained as the harmonic mean of the individual spring constants, and in parallel, as their arithmetic mean [13], while the equivalent resistance of resistors combined in parallel is obtained by the harmonic mean of the individual resistances, and in series by their arithmetic mean [14]. In traffic flow [15], arithmetic and harmonic mean of speed distribution are used. In [16] both geometric and harmonic mean are used in addition to arithmetic mean to improve noise source maps.

In our recent work on inequalities for generalized, quasi-arithmetic weighted means [17], we found some invariance properties, that depended on the particular relationship considered between the sequences of weights. These relationships between weights define first stochastic order, and likelihood ratio order. Their application to multiple importance sampling (MIS), a Monte Carlo technique, has been presented in [18], its application to cross entropy in [19], and in [20] applications to image processing, traffic flow and income distribution have been shown. In [21], the invariance results on products of distributions of independent scalar r.v.’s [22] was generalized to any joint distribution of a 2-dimensional r.v.

In this paper, we show that the order invariance is a necessary and sufficient condition for first stochastic order, and that it holds under any quasi-arithmetic mean. We also study invariance under the second stochastic order, likelihood ratio, hazard-rate, and increasing convex stochastic orders. The fact that the invariance results hold for both increasing and decreasing monotonic functions allows us to use both utilities and liabilities, represented by efficiencies and expected error, respectively, to look for an optimal solution, where in liabilities we look for minimum expected error, while in utilities for maximum efficiency.

The rest of the paper is organized as follows. In Section 2, we introduce the stochastic order; in Section 3, the arithmetic mean and its relationship with stochastic order. In Section 4, we present the invariance theorems; in Section 5, we discuss the invariance for concave (convex) functions; in Section 6, its application to stochastic orders; in Section 7, we present an example based on the linear combination of Monte Carlo estimators. Finally, conclusions and future work are given in Section 8.

2. Stochastic Orders

Stochastic orders are pre-orders (i.e, binary relationships holding symmetric and transitive properties) defined on probability distributions with finite support. Note that equivalently, one can think of sequences (i.e., ordered sets) of non-negative weights/values that sum up to one. Observe that any sequence $\left\{{\alpha }_k\right\}$ of M positive numbers such that ${\sum }_{k=1}^M{\alpha }_k=1$ can be considered a probability distribution. It can be seen too as an element of the (M-1)-Simplex, i.e., $\left\{{\alpha }_k\right\}\in {\Delta }_{M-1}$. While several interpretations hold and hence increase the range of applicability, in the remaining of this paper, we will talk of sequence without any loss of generality.

Notation. 

We use the symbols $\prec ,\succ$ to represent orders between two sequences $\left\{{\alpha }_k\right\}$ and $\left\{{\alpha }_k^{\prime }\right\}$ of size M, e.g., and we write $\left\{{\alpha }_k\right\}\succ \left\{{\alpha }_k^{\prime }\right\}$ or equivalently $\left\{{\alpha }_k^{\prime }\right\}\prec \left\{{\alpha }_k\right\}$. We will denote the elements of the sequences without the curly brackets, e.g., the first element of the sequence $\left\{{\alpha }_k\right\}$ is denoted as ${\alpha }_1$, while the last one is ${\alpha }_M$. Moreover, the first and last elements of the sequence receive special attention, since when ${\alpha }_1\le {\alpha }_1^{\prime }$, we can write this order as $\left\{{\alpha }_k\right\}{\succ }_F\left\{{\alpha }_k^{\prime }\right\}$ (where ${\succ }_F$ stands for first ) and whenever ${\alpha }_M\ge {\alpha }_M^{\prime }$, we can write $\left\{{\alpha }_k\right\}{\succ }_L\left\{{\alpha }_k^{\prime }\right\}$ (where ${\succ }_L$ stands for last ). The case with both ${\alpha }_1\le {\alpha }_1^{\prime }$ and ${\alpha }_M\ge {\alpha }_M^{\prime }$, we can denote as $\left\{{\alpha }_k\right\}{\succ }_{FL}\left\{{\alpha }_k^{\prime }\right\}$. The orders ${\succ }_F,{\succ }_L,{\succ }_{FL}$ are superorders of first stochastic dominance order, $\left\{{\alpha }_k\right\}{\succ }_{FSD}\left\{{\alpha }_k^{\prime }\right\}$, that will be studied in Section 6. We denote as $\left\{{\alpha }_{M-k+1}\right\}$ the sequence with the same elements of $\left\{{\alpha }_k\right\}$ but in reversed order.

Example 1

(Toy example). Given sequences $\left\{{\alpha }_k\right\}=\{0.5,0.1,0.2,0.2\}$, and $\left\{{\alpha }_k^{\prime }\right\}=$$\{0.4,0.1,0.2,0.3\}$, we have both $\left\{{\alpha }_k\right\}{\prec }_F\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k\right\}{\prec }_L\left\{{\alpha }_k^{\prime }\right\}$, and thus $\left\{{\alpha }_k\right\}{\prec }_{FL}\left\{{\alpha }_k^{\prime }\right\}$. On the other hand for sequences $\left\{{\alpha }_k\right\}$ and $\left\{{\alpha }_k^{″}\right\}=\{0.4,0.3,0.2,0.1\}$, we have $\left\{{\alpha }_k\right\}{\prec }_F\left\{{\alpha }_k^{″}\right\}$, $\left\{{\alpha }_k\right\}{\nprec }_L\left\{{\alpha }_k^{″}\right\}$, and thus $\left\{{\alpha }_k\right\}{\nprec }_{FL}\left\{{\alpha }_k^{″}\right\}$.

Property 1

(Mirror property). One desirable property of stochastic orders is such that when reversing the ordering of the sequence elements, the stochastic order should be reversed as well, i.e., if $\left\{{\alpha }_k\right\}\succ \left\{{\alpha }_k^{\prime }\right\}$, then $\left\{{\alpha }_{M-k+1}^{\prime }\right\}\succ \left\{{\alpha }_{M-k+1}\right\}$. This is similar to the invariance of physical laws to the right and left hand. We call this property the mirror property.

Definition 1

(Mirror property). We say that a stochastic order has the mirror property if

$$\left\{{\alpha }_k^{\prime }\right\}\prec \left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_{M-k+1}\right\}\prec \left\{{\alpha }_{M-k+1}^{\prime }\right\}.$$

(1)

Observe that the simple orders defined before, ${\succ }_F$, ${\succ }_L$, do not hold this property, but ${\succ }_{FL}$ holds it. We will see in Section 6 that usual stochastic orders do hold the mirror property. However, an order that is insensitive to the permutation of the elements of a sequence, like majorization or Lorentz order, does not hold the mirror property.

3. Quasi-Arithmetic Mean

Stochastic orders are usually defined by invariance to arithmetic mean [1,2] (see Section 6), and we want to investigate in this paper invariance to more general means. We define here the kind of means we are interested in.

Definition 2

(Quasi-arithmetic or Kolmogorov or Kolmogorov–Nagumo mean). A quasi-arithmetic weighted mean (or Kolmogorov mean) $\mathcal{M}(\left\{b_k\right\},\left\{{\alpha }_k\right\})$ of a sequence of real numbers ${\left\{b_k\right\}}_{k=1}^M$, is of the form $g^{-1}\left({\sum }_{k=1}^M{\alpha }_{k}g\left(b_k\right))\right)$, where $g\left(x\right)$ is a real valued, invertible strictly monotonic function, with inverse function $g^{-1}\left(x\right)$ and ${\left\{{\alpha }_k\right\}}_{k=1}^M$ are positive weights such that ${\sum }_{k=1}^M{\alpha }_k=1$.

Examples of such mean are arithmetic weighted mean ($g\left(x\right)=x$), harmonic weighted mean ($g\left(x\right)=1/x$), geometric weighted mean ($g\left(x\right)=logx$) and in general, the weighted power means ($g\left(x\right)=x^r$).

Given a distribution $\left\{p_k\right\}$, Shannon entropy, $\mathcal{H}\left(\left\{p_k\right\}\right)=-{\sum }_k{p}_{k}log{p}_k$, and Rényi entropy, ${\mathcal{R}}_{\beta }\left(\left\{p_k\right\}\right)=\frac{1}{1-\beta }log{\sum }_k{p}_k^{\beta }$, can be considered as quasi-arithmetic means of the sequence $\left\{\frac{1}{log{p}_k}\right\}$ with weights $\left\{p_k\right\}$: Shannon entropy with $g\left(x\right)=x$ (arithmetic mean or expected value), and Rényi entropy with $g\left(x\right)=2^{(\beta -1)x}$ [11]. Tsallis entropy, ${\mathcal{S}}_q\left(\left\{p_k\right\}\right)=\frac{1-{\sum }_{i=1}^M{p}_i^q}{q-1}$, can be considered the weighted arithmetic mean of the sequence $\left\{\frac{1}{{ln}_q{p}_k}\right\}$ with weights $\left\{p_k\right\}$, where ${ln}_{q}x=\frac{x^{1-q}-1}{1-q}$ is the q-logarithm function [23]. Without loss of generality, we consider from now on that $\{b_k=f\left(k\right)\}$ where $f\left(x\right)$ is a real valued function.

Lemma 1.

Consider the sequences of M positive weights ${\left\{{\alpha }_k\right\}}_{k=1}^M$ and ${\left\{{\alpha }_k^{\prime }\right\}}_{k=1}^M$, ${\sum }_{k=1}^M{\alpha }_k=1,{\sum }_{k=1}^M{\alpha }_k^{\prime }=1$, $g\left(x\right)$ a strictly monotonic function. The following conditions (a), (b), (a’), (b’), (c) and (d) are equivalent.

(a) for $g\left(x\right)$ increasing, for any increasing function $f\left(x\right)$ the following inequality holds:

$$\sum _{k=1}^M{\alpha }_k^{\prime }g\left(f\left(k\right)\right)\le \sum _{k=1}^M{\alpha }_{k}g\left(f\left(k\right)\right)$$

(2)

(b) for $g\left(x\right)$ increasing, for any increasing function $f\left(x\right)$ the following inequality holds:

$$\sum _{k=1}^M{\alpha }_{M-k+1}g\left(f\left(k\right)\right)\le \sum _{k=1}^M{\alpha }_{M-k+1}^{\prime }g\left(f\left(k\right)\right)$$

(3)

(a’) for $g\left(x\right)$ decreasing, for any increasing function $f\left(x\right)$ the following inequality holds:

$$\sum _{k=1}^M{\alpha }_{k}g\left(f\left(k\right)\right)\le \sum _{k=1}^M{\alpha }_k^{\prime }g\left(f\left(k\right)\right)$$

(4)

(b’) for $g\left(x\right)$ decreasing, for any increasing function $f\left(x\right)$ the following inequality holds:

$$\sum _{k=1}^M{\alpha }_{M-k+1}^{\prime }g\left(f\left(k\right)\right)\le \sum _{k=1}^M{\alpha }_{M-k+1}g\left(f\left(k\right)\right)$$

(5)

(c) the following inequalities hold:

$$\begin{array}{ccc}\hfill {\alpha }_1^{\prime }& \ge & {\alpha }_1\hfill \\ \hfill {\alpha }_1^{\prime }+{\alpha }_2^{\prime }& \ge & {\alpha }_1+{\alpha }_2\hfill \\ & ⋮& \hfill \\ \hfill {\alpha }_1^{\prime }+\dots +{\alpha }_{M-1}^{\prime }& \ge & {\alpha }_1+\dots +{\alpha }_{M-1}\hfill \\ \hfill {\alpha }_1^{\prime }+\dots +{\alpha }_{M-1}^{\prime }+{\alpha }_M^{\prime }& =& {\alpha }_1+\dots +{\alpha }_{M-1}+{\alpha }_M\hfill \end{array}$$

(6)

(d) the following inequalities hold:

$$\begin{array}{ccc}\hfill {\alpha }_M^{\prime }& \le & {\alpha }_M\hfill \\ \hfill {\alpha }_M^{\prime }+{\alpha }_{M-1}^{\prime }& \le & {\alpha }_M+{\alpha }_{M-1}\hfill \\ \hfill ⋮\\ \hfill {\alpha }_M^{\prime }+\dots +{\alpha }_2^{\prime }& \le & {\alpha }_M+\dots +{\alpha }_2\hfill \\ \hfill {\alpha }_M^{\prime }+{\alpha }_{M-1}^{\prime }+\dots +{\alpha }_1^{\prime }& =& {\alpha }_M+{\alpha }_{M-1}+\dots +{\alpha }_1\hfill \end{array}$$

(7)

If $f\left(x\right)$ is decreasing, the inequalities in a) through b’) are reversed.

Proof. 

Indirect or partial proofs can be found in [17,20]. We provide a complete proof in the Appendix A. □

Note. 

Observe that in Lemma 1, it is sufficient to consider the monotonicity of the sequences ${\left\{f\left(k\right)\right\}}_{k=1}^M$. Furthermore, Lemma 1 can be extended to any real sequences $\left\{y_k\right\}$ and $\left\{x_k\right\}$, such that ${\sum }_{k=1}^M{y}_k={\sum }_{k=1}^M{x}_k$. It is enough to observe that the order of all the inequalities are unchanged by adding a positive constant, so that $\left\{y_k\right\}$ and $\left\{x_k\right\}$ can be made positive, and also are unchanged by the multiplication of a positive constant, so that the resulting $\left\{y_k\right\}$ and $\left\{x_k\right\}$ sequences can be normalized.

Theorem 1.

Given a mean $\mathcal{M}$ with strictly monotonic function $g\left(x\right)$ and two distributions $\left\{{\alpha }_k^{\prime }\right\},\left\{{\alpha }_k\right\}$, the following propositions are equivalent:

(a) for all increasing functions $f\left(x\right)$

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\le \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$$

(8)

(b) for all increasing functions $f\left(x\right)$

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_{M-k+1}\right\})\le \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_{M-k+1}^{\prime }\right\})$$

(9)

(a’) for all decreasing functions $f\left(x\right)$

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\ge \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$$

(10)

(b’) for all decreasing functions $f\left(x\right)$

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_{M-k+1}\right\})\ge \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_{M-k+1}^{\prime }\right\})$$

(11)

(c) Condition (c) of Lemma 1 holds

Proof. 

It is a direct consequence of Lemma 1 and the definition of quasi-arithmetic mean, observing that the inverse $g^{-1}\left(x\right)$ of a strictly monotonic increasing (respectively decreasing) function $g\left(x\right)$ is also increasing (respectively decreasing). □

4. Invariance

Theorem 2

(Invariance). Given two distributions $\left\{{\alpha }_k^{\prime }\right\},\left\{{\alpha }_k\right\}$, and two quasi-arithmetic means $\mathcal{M}$, ${\mathcal{M}}^{🟉}$, the following propositions are equivalent:

(a) for all increasing functions $f\left(x\right)$

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\le \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$$

(12)

(b) for all increasing functions $f\left(x\right)$

$${\mathcal{M}}^{🟉}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\le {\mathcal{M}}^{🟉}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$$

(13)

(a’) for all decreasing functions $f\left(x\right)$

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\ge \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$$

(14)

(b’) for all decreasing functions $f\left(x\right)$

$${\mathcal{M}}^{🟉}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\ge {\mathcal{M}}^{🟉}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$$

(15)

Proof. 

It is a direct consequence of the observation that conditions (c) and (d) in Lemma 1 do not depend on any particular $g\left(x\right)$ function considered, and thus the order of inequalities does not change with the mean considered as long as $\left\{{\alpha }_k\right\}$ and $\left\{{\alpha }_k^{\prime }\right\}$ are kept fixed. □

The following properties relate stochastic order with the quasi-arithmetic mean. Let $\mathcal{I}$ be the set of monotonous functions, ${\mathcal{I}}^{<}$ the set of increasing functions, ${\mathcal{I}}^{>}$ the set of decreasing functions.

Definition 3

(preserve mean order property). We say that a stochastic order preserves mean order for a given mean $\mathcal{M}$ and a set $I^{<}\subset {\mathcal{I}}^{<}$ of increasing functions when, for all functions $f\left(x\right)\in I^{<}$ and any distributions $\left\{{\alpha }_k^{\prime }\right\},\left\{{\alpha }_k\right\}$,

$$\left\{{\alpha }_k\right\}\prec \left\{{\alpha }_k^{\prime }\right\}\Rightarrow \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\le \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\}).$$

(16)

Definition 4

(preserve inverse mean order property). We say that a stochastic order preserves inverse mean order for a given mean $\mathcal{M}$ and a set $I^{>}\subset {\mathcal{I}}^{>}$ of decreasing functions when, for all functions $f\left(x\right)\in I^{>}$ and any distributions $\left\{{\alpha }_k^{\prime }\right\},\left\{{\alpha }_k\right\}$,

$$\left\{{\alpha }_k\right\}\prec \left\{{\alpha }_k^{\prime }\right\}\Rightarrow \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\ge \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\}).$$

(17)

Theorem 2 together with the preserve mean order properties allows us to state the following invariance property:

Theorem 3

(preserve mean order invariance). Given a stochastic order that preserves mean order (respectively preserves inverse mean order) for a given mean and for ${\mathcal{I}}^{<}$ (respectively for ${\mathcal{I}}^{>}$), then for any mean it preserves both mean order for ${\mathcal{I}}^{<}$ and inverse mean order for ${\mathcal{I}}^{>}$. In other words, the preserve mean order properties are invariant with respect to the mean considered.

Observe that from Lemma 1 and Theorems 1 and 3, we have that a necessary and sufficient condition for an order to preserve mean order for ${\mathcal{I}}^{<}$ (or preserve inverse mean order for ${\mathcal{I}}^{>}$) is the holding of Equation (6), independently of the mean considered. We will see in Section 6 that this corresponds to first stochastic dominance order.

5. Concavity and Convexity

Let us consider now ${\mathcal{I}}_v^{<}\subset {\mathcal{I}}^{<}$, the set of all increasing concave functions, ${\mathcal{I}}_x^{<}\subset {\mathcal{I}}^{<}$, the set of all increasing convex functions, ${\mathcal{I}}_v^{>}\subset {\mathcal{I}}^{>}$, the set of all decreasing concave functions, and ${\mathcal{I}}_x^{>}\subset {\mathcal{I}}^{>}$, the set of all decreasing convex functions. The following theorem relates the preserve mean order properties with the mirror property.

Theorem 4.

If an order holds the mirror property, then holding preserve mean order for ${\mathcal{I}}_v^{<}$ (respectively ${\mathcal{I}}_x^{<}$) implies holding preserve inverse mean order for ${\mathcal{I}}_v^{>}$ (respectively ${\mathcal{I}}_x^{>}$) and viceversa.

Proof. 

Suppose $\left\{{\alpha }_k^{\prime }\right\}\prec \left\{{\alpha }_k\right\}$ and $f\left(x\right)$ decreasing and concave (respectively convex). Then, by the mirror property $\left\{{\alpha }_{M-k+1}\right\}\prec \left\{{\alpha }_{M-k+1}^{\prime }\right\}$, and by the hypothesis of the theorem

$$\begin{array}{ccc}\hfill \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})& =& \mathcal{M}(\left\{f(M-k+1)\right\},\left\{{\alpha }_{M-k+1}\right\})=\mathcal{M}(\left\{f\left(m\left(k\right)\right)\right\},\left\{{\alpha }_{M-k+1}\right\})\hfill \\ \hfill & \le & \mathcal{M}(\left\{f\left(m\left(k\right)\right)\right\},\left\{{\alpha }_{M-k+1}^{\prime }\right\})=\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\}),\hfill \end{array}$$

(18)

where $m\left(x\right)=M-x+1$, and because if $f\left(x\right)$ is decreasing and concave (respectively convex) then $f\left(m\right(x\left)\right)$ is increasing and concave (respectively convex). □

The following result is necessary to prove Lemma 2.

Theorem 5.

Given $f\left(x\right)$ and weights $\left\{{\alpha }_k^{\prime }\right\},\left\{{\alpha }_k\right\}$, if inequality $\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})\le \mathcal{M}(\left\{f\left(k\right)\right\},$$\left\{{\alpha }_k\right\})$ holds for any $g\left(x\right)$ strictly increasing and convex (respectively concave) then it holds for any $g\left(x\right)$ strictly decreasing and concave (respectively convex). If it holds for any $g\left(x\right)$ strictly decreasing and convex (respectively concave) then it holds for any $g\left(x\right)$ strictly increasing and concave (respectively convex).

Proof. 

Consider the quasi-arithmetic mean with Kolmogorov function $g^{🟉}=-g\left(x\right)$ (with inverse $g^{🟉-1}=g^{-1}(-x)$). When $g\left(x\right)$ is increasing, $g^{🟉}\left(x\right)$ is decreasing and viceversa. When $g\left(x\right)$ is convex, $g^{🟉}\left(x\right)$ is concave and viceversa. We have

$$\begin{array}{ccc}\hfill g^{🟉-1}\left(\sum _i{\alpha }_i^{\prime }{g}^{🟉}\left(f\left(i\right)\right)\right)& =& g^{-1}\left(-\left(\sum _i{\alpha }_i^{\prime }(-g\left(f\left(i\right)\right))\right)\right)=g^{-1}\left(\sum _i{\alpha }_i^{\prime }g\left(f\left(i\right)\right)\right)\hfill \\ \hfill & \le & g^{-1}\left(\sum _i{\alpha }_{i}g\left(f\left(i\right)\right)\right)=g^{-1}\left(-\left(\sum _i{\alpha }_i(-g\left(f\left(i\right)\right))\right)\right)=g^{🟉-1}\left(\sum _i{\alpha }_i{g}^{🟉}\left(f\left(i\right)\right)\right)\hfill \end{array}$$

(19)

□

The following Lemma is needed to prove how the invariance properties for one mean extends to other means.

Lemma 2.

Given two distributions $\left\{{\alpha }_k^{\prime }\right\},\left\{{\alpha }_k\right\}$, and a quasi-arithmetic mean $\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$ with function $g\left(x\right)$. Consider the following Equations:

$$\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k^{\prime }\right\})\le \mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\}),$$

(20)

and

$$\begin{array}{c}\hfill \mathcal{M}\left(\left\{f\left(k\right)\right\},\left\{{\alpha }_{M-k+1}\right\}\right)\le \mathcal{M}\left(\left\{f\left(k\right)\right\},\left\{{\alpha }_{M-k+1}^{\prime }\right\}\right).\end{array}$$

(21)

Then, for each line in Table 1, and for $g\left(x\right),f\left(x\right)$ filling the conditions in first and second column in Table 1 and holding Equations (20) and (21), Equations (20) and (21) hold too for $g^{🟉}\left(x\right),f^{🟉}\left(x\right)$ filling the conditions in columns three and four.

Table 1. For each line, for $g\left(x\right),f\left(x\right)$ filling the conditions in first and second column, then Equations (20) and (21) hold for $g^{🟉},f^{🟉}$ filling the conditions in columns three and four. By changing $f^{🟉}$ from increasing to decreasing, the reverse of Equations (20) and (21) hold. ICX: convex and increasing, ICV: concave and increasing, DCX: convex and decreasing, DCV: concave and decreasing.

	$\mathit{g}$	$\mathit{f}$	${\mathit{g}}^{🟉}$	${\mathit{f}}^{🟉}$
1	ICX	ICV	ICV	ICV
1’	"	"	DCX	ICV
2	DCX	ICX	DCV	ICX
2’	"	"	ICX	ICX
3	ICV	ICX	ICX	ICX
3’	"	"	DCV	ICX
4	DCV	ICV	DCX	ICV
4’	"	"	ICV	ICV
5	ICX	ICV	DCV	ICX
5’	"	"	ICX	ICX
6	DCX	ICX	ICV	ICV
6’	"	"	DCX	ICV
7	ICV	ICX	DCX	ICV
7’	"	"	ICV	ICV
8	DCV	ICV	ICX	ICX
8’	"	"	DCV	ICX

Additionally, for each line in Table 1, changing $f^{🟉}\left(x\right)$ in column four from increasing to decreasing, Equations (20) and (21) hold with the inequalities reversed.

Proof. 

The proof of lines 1–8 in Table 1 is in the Appendix B. Lines 1’-8’ in Table 1 are a direct consequence of Theorem 5 applied to lines 1–8. □

Theorem 6.

Given a quasi-arithmetic mean $\mathcal{M}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$ with function $g\left(x\right)$, we have that if an order holds the mirror property then for each line in Table 1, and for $g\left(x\right)$ filling the condition in the first column in Table 1 and preserving order for $f\left(x\right)$ in the second column in Table 1, the mean for $g^{🟉}\left(x\right)$ filling the condition in column three in Table 1 preserves order for $f^{🟉}\left(x\right)$ filling the condition in column four in Table 1.

Proof. 

It is enough to apply Lemma 2, taking into account that if Equation (20) holds then Equation (21) holds too by the mirror property. □

Corollary 1.

Consider the weighted arithmetic mean $\mathcal{A}(\left\{f\left(k\right)\right\},\left\{{\alpha }_k\right\})$. Given an order that holds the mirror property and preserves the order for mean $\mathcal{A}$, then

(a) If order is preserved for mean $\mathcal{A}$ and for ${\mathcal{I}}_v^{<}$ then order is preserved for any mean with concave-increasing/convex-decreasing (respectively concave-decreasing/convex-increasing) function $g\left(x\right)$ and for ${\mathcal{I}}_v^{<}$ (respectively for ${\mathcal{I}}_x^{<}$).

(b) If order is preserved for mean $\mathcal{A}$ and for ${\mathcal{I}}_x^{<}$, then order is preserved for any mean with convex-increasing/concave-decreasing (respectively convex-decreasing/concave-increasing) function $g\left(x\right)$ and for ${\mathcal{I}}_x^{<}$ (respectively for ${\mathcal{I}}_v^{<}$).

Proof. 

Arithmetic mean is a quasi-arithmetic mean with increasing function $g\left(x\right)=x$, which is both concave-increasing and convex-increasing, and thus Table 1 collapses to Table 2.

Table 2. For each line, $g\left(x\right)=x$, $f\left(x\right)$ filling the conditions in second column and Equations (20) and (21) holding, then Equations (20) and (21) hold too for $g^{🟉},f^{🟉}$ filling the conditions in columns three and four. ICX: convex and increasing, ICV: concave and increasing, DCX: convex and decreasing, DCV: concave and decreasing.

$\mathit{f}$	${\mathit{g}}^{🟉}$	${\mathit{f}}^{🟉}$
ICV	ICV	ICV
ICV	DCX	ICV
ICX	ICX	ICX
ICX	DCV	ICX
ICV	DCV	ICX
ICV	ICX	ICX
ICX	DCX	ICV
ICX	ICV	ICV

□

Functions $x^p$, for $p\ge 1$ are convex-increasing and with $p\le 0$ are convex-decreasing over ${\mathcal{R}}_{++}$, $e^x$ convex-increasing over $\mathcal{R}$, while $logx$, $x^p$, for $0<p\le 1$ are concave-increasing over ${\mathcal{R}}_{++}$. Affine functions are both concave and convex over $\mathcal{R}$. If $g\left(x\right)$ is convex, $-g\left(x\right)$ is concave and viceversa, and the composition of concave-increasing and concave is concave, and convex-increasing and convex is convex. We will see in the next section that preserving the order for mean $\mathcal{A}$ for ${\mathcal{I}}_v^{<}$ is defined as second-order stochastic dominance or increasing concave order, and preserving the order for mean $\mathcal{A}$ for ${\mathcal{I}}_x^{<}$ is defined as increase convex order stochastic dominance. Both orders hold the mirror property and thus Corollary 1 applies to both.

6. Application to Stochastic Orders and Cross-Entropy

6.1. First-Order Stochastic Dominance

Originating in the economics risk literature [24], first-order stochastic dominance [1,2] FSD, between two probability distributions, $\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k\right\}$, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}$, is defined as:

Definition 5.

$\left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}$⇔ for any increasing function $f\left(x\right)$,

$$E{\left[f\left(k\right)\right]}_{{\alpha }_k^{\prime }}\le E{\left[f\left(k\right)\right]}_{{\alpha }_k}.$$

Remember that the expected value is the arithmetic weighted mean.

A necessary and sufficient condition for first-order stochastic dominance is defined by the condition $c)$ of Lemma 1, which is equivalent to condition $d)$ of Lemma 1, thus $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_{M-k+1}\right\}{\prec }_{FSD}\left\{{\alpha }_{M-k+1}^{\prime }\right\}$, i.e., the mirror property holds. From the definition of FSD, and from Theorem 3, we can redefine FSD, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}$ as there exists a mean $\mathcal{M}$ such that, for any increasing function $f\left(x\right)$, Equation (16) holds. This is, the definition of FSD is independent of the mean considered, while the original definition relies on the expected value (arithmetic mean). The mean considered can be arithmetic, harmonic, geometric or any other quasi-arithmetic mean.

Let us consider now a strictly monotonous function $g\left(x\right)$, and define a generalized cross-entropy $\mathcal{GCE}(\left\{p_i\right\},\left\{q_i\right\})=g^{-1}\left({\sum }_i{p}_{i}g(-log{q}_i)\right)$. Observe that it is a quasi-arithmetic mean, and for $g\left(x\right)=x$, we get the cross-entropy $CE(\left\{p_i\right\},\left\{q_i\right\})=-\sum p_{i}log{q}_i$. Other functions that generalize cross entropy have been defined in the context of training deep neural networks [25]. We can state the following theorem:

Theorem 7.

Given distributions $\left\{p_i^{\prime }\right\},\left\{p_i\right\},\left\{q_i\right\}$, increasing, and $\left\{p_k^{\prime }\right\}{\prec }_{FSD}\left\{p_k\right\}$, then for any mean $\mathcal{GCE}$ with function $g\left(x\right)$, $\mathcal{GCE}(\left\{p_i\right\},\left\{q_i\right\})\le \mathcal{GCE}(\left\{p_i^{\prime }\right\},\left\{q_i\right\})$.

Proof. 

Observe first that $f\left(k\right)=-log{q}_k$ is a decreasing sequence. From the hypothesis, condition $c)$ of Lemma 1 holds, thus we can then apply Theorem 1 to the mean with function $g\left(x\right)$ and for $\left\{f\right(k\left)\right\}$ decreasing. □

The following result relates the entropies of two distributions with the first stochastic order.

Theorem 8.

Given $\left\{p_k\right\},\left\{q_k\right\}$ increasing, and $\left\{q_k\right\}{\prec }_{FSD}\left\{p_k\right\}$, then $\mathcal{H}\left(\left\{p_k\right\}\right)\le \mathcal{H}\left(\left\{q_k\right\}\right)$, where $\mathcal{H}$ stands for Shannon entropy.

Proof. 

The Kullback–Leibler distance is always positive [26], ${\sum }_i{p}_{i}log\frac{p_i}{q_i}\ge 0$, and thus we have that $\mathcal{H}\left(\left\{p_k\right\}\right)=-{\sum }_i{p}_{i}log{p}_i\le -\sum p_{i}log{q}_i$. Applying Theorem 7 for $g\left(x\right)=x$, $-\sum p_{i}log{q}_i\le -\sum q_{i}log{q}_i=\mathcal{H}\left(\left\{q_k\right\}\right).$ □

6.2. Second-Order Stochastic Dominance and Increase Convex Ordering

Definition 6.

Second-order stochastic dominance between two probability distributions, $\left\{{\alpha }_k\right\}$, $\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{SSD}\left\{{\alpha }_k\right\}$, occurs when for any increasing concave function $f\left(x\right)$,

$$E{\left[f\left(k\right)\right]}_{{\alpha }_k^{\prime }}\le E{\left[f\left(k\right)\right]}_{{\alpha }_k}.$$

Trivially

$$\left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_k^{\prime }\right\}{\prec }_{SSD}\left\{{\alpha }_k\right\}.$$

Let us consider the cumulative distribution function, $F_i={\alpha }_1+{\alpha }_2+\dots +{\alpha }_i$, and the survival function ${\overline{F}}_i=1-F_i={\alpha }_M+{\alpha }_{M-1}+\dots +{\alpha }_{i+1}$. A necessary and sufficient condition for second-order stochastic dominance [1,2] is the following:

$$\begin{array}{ccc}\hfill F_1^{\prime }& \ge & F_1\hfill \\ \hfill F_1^{\prime }+F_2^{\prime }& \ge & F_1+F_2\hfill \\ & ⋮& \hfill \\ \hfill F_1^{\prime }+\dots +F_{M-1}^{\prime }& \ge & F_1+\dots +F_{M-1}\hfill \\ \hfill F_1^{\prime }+\dots +F_{M-1}^{\prime }+F_M^{\prime }& \ge & F_1+\dots +F_{M-1}+F_M,\hfill \end{array}$$

(22)

or equivalently,

$$\begin{array}{ccc}\hfill {\overline{F}}_1^{\prime }& \le & {\overline{F}}_1\hfill \\ \hfill {\overline{F}}_1^{\prime }+{\overline{F}}_2^{\prime }& \le & {\overline{F}}_1+{\overline{F}}_2\hfill \\ & ⋮& \hfill \\ \hfill {\overline{F}}_1^{\prime }+\dots +{\overline{F}}_{M-1}^{\prime }& \le & {\overline{F}}_1+\dots +{\overline{F}}_{M-1}\hfill \\ \hfill {\overline{F}}_1^{\prime }+\dots +{\overline{F}}_{M-1}^{\prime }+{\overline{F}}_M^{\prime }& \le & {\overline{F}}_1+\dots +{\overline{F}}_{M-1}+{\overline{F}}_M.\hfill \end{array}$$

(23)

Let us see first that $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{SSD}\left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_{M-k+1}\right\}{\prec }_{SSD}\left\{{\alpha }_{M-k+1}^{\prime }\right\}$, i.e., SSD holds the mirror property. Effectively, define $G_i={\alpha }_M+{\alpha }_{M-1}+\dots +{\alpha }_{i+1}$, and the survival function ${\overline{G}}_i=1-G_i={\alpha }_1+{\alpha }_2+\dots +{\alpha }_i$. However, $G_i={\overline{F}}_i$, ${\overline{G}}_i=F_i$, $G_i^{\prime }={{\overline{F}}^{\prime }}_i$, ${{\overline{G}}^{\prime }}_i=F_i^{\prime }$, and substituting in Equation (22) or Equation (23), we obtain the desired result.

From Corollary 1, second-order stochastic dominance preserves mean order for all means $\mathcal{M}$ defined by a concave-increasing/convex-decreasing (respectively concave-decreasing/convex-increasing) function $g\left(x\right)$, and for the set of all concave-increasing functions ${\mathcal{I}}_v^{<}$ (respectively convex-increasing functions ${\mathcal{I}}_x^{<}$). For instance, order is preserved for the geometric mean, or any mean with $g\left(x\right)=x^p$, $0<p\le 1$ and for ${\mathcal{I}}_v^{<}$, or for or any mean with $g\left(x\right)=x^p$ with $p<0$, as the harmonic mean, for ${\mathcal{I}}_x^{<}$. In particular, we can state the following theorem about cross-entropy of two distributions,

Theorem 9.

Given distributions $\left\{p_i^{\prime }\right\},\left\{p_i\right\},\{q_i=f\left(i\right)\}$, $f\left(x\right)$ concave-increasing, and $\left\{p_i^{\prime }\right\}{\prec }_{SSD}\left\{p_i\right\}$, then $CE(\left\{p_i\right\},\left\{q_i\right\})\le CE(\left\{p_i^{\prime }\right\},\left\{q_i\right\})$.

Proof. 

We have that $CE(\left\{p_i\right\},\left\{q_i\right\})=-log\mathcal{G}(\left\{q_i\right\},\left\{p_i\right\})=-log\left({\Pi }_i{q}_i^{p_i}\right)=-log(exp({\sum }_i$$p_{i}log{q}_i\left)\right)$, where $\mathcal{G}$ stands for geometric mean. The geometric mean is a quasi-arithmetic mean with function $g\left(x\right)=logx$, concave-increasing. Using Definition 6 and applying Corollary 1(a), we obtain the inequality $\mathcal{G}(\left\{q_i\right\},\left\{p_i^{\prime }\right\})\le \mathcal{G}(\left\{q_i\right\},\left\{p_i\right\})$ and we apply the function $-logx$ to both members of this inequality. □

When we consider $f\left(x\right)$, a convex instead of a concave function, we talk of increasing convex order, ICX. This is,

Definition 7.

Second-order stochastic dominance between two probability distributions, $\left\{{\alpha }_k\right\}$, $\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{ICX}\left\{{\alpha }_k\right\}$, occurs when for any increasing convex function $f\left(x\right)$,

$$E{\left[f\left(k\right)\right]}_{{\alpha }_k^{\prime }}\le E{\left[f\left(k\right)\right]}_{{\alpha }_k}.$$

$\left\{{\alpha }_k\right\}$ is greater in increasing convex order than $\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{ICX}\left\{{\alpha }_k\right\}$, if and only if the following inequalities hold:

$$\begin{array}{ccc}\hfill F_M^{\prime }& \ge & F_M\hfill \\ \hfill F_M^{\prime }+F_{M-1}^{\prime }& \ge & F_M+F_{M-1}\hfill \\ & ⋮& \hfill \\ \hfill F_M^{\prime }+\dots +F_2^{\prime }& \ge & F_M+\dots +F_2\hfill \\ \hfill F_M^{\prime }+\dots +F_2^{\prime }+F_1^{\prime }& \ge & F_M+\dots +F_2+F_1\hfill \end{array}$$

(24)

or equivalently,

$$\begin{array}{ccc}\hfill {\overline{F}}_M^{\prime }& \le & {\overline{F}}_M\hfill \\ \hfill {\overline{F}}_M^{\prime }+{\overline{F}}_{M-1}^{\prime }& \le & {\overline{F}}_M+{\overline{F}}_{M-1}\hfill \\ & ⋮& \hfill \\ \hfill {\overline{F}}_M^{\prime }+\dots +{\overline{F}}_2^{\prime }& \le & {\overline{F}}_M+\dots +{\overline{F}}_2\hfill \\ \hfill {\overline{F}}_M^{\prime }+\dots +{\overline{F}}_2^{\prime }+{\overline{F}}_1^{\prime }& \le & {\overline{F}}_M+\dots +{\overline{F}}_2+{\overline{F}}_1\hfill \end{array}$$

(25)

Trivially

$$\left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_k^{\prime }\right\}{\prec }_{ICX}\left\{{\alpha }_k\right\}.$$

Mirror property is also immediate. From Corollary 1, second-order stochastic dominance preserves mean order for all means $\mathcal{M}$ defined by a convex-increasing/concave-decreasing (respectively convex-decreasing/concave-increasing) function $g\left(x\right)$ and for the set of all convex-increasing functions ${\mathcal{I}}_x^{<}$ (respectively concave-increasing functions ${\mathcal{I}}_v^{<}$). For instance, any mean with $g\left(x\right)=x^p$, $p\ge 1$ as the weighted quadratic mean with ${\mathcal{I}}_x^{<}$, or $g\left(x\right)=x^p$, $p<0$ as the harmonic mean with ${\mathcal{I}}_v^{<}$, or the mean with $g\left(x\right)=-logx$ with ${\mathcal{I}}_v^{<}$. In particular, we can state the following result,

Theorem 10.

Given distributions $\left\{p_i^{\prime }\right\},\left\{p_i\right\},\{q_i=f\left(i\right)\}$, $f\left(x\right)$ concave-increasing, and $\left\{p_i^{\prime }\right\}{\prec }_{ICX}\left\{p_i\right\}$, then $CE(\left\{p_i\right\},\left\{q_i\right\})\le CE(\left\{p_i^{\prime }\right\},\left\{q_i\right\})$.

Proof. 

Consider the quasi-arithmetic mean ${\mathcal{G}}^{\prime }(\left\{q_i\right\},\left\{p_i\right\})$ with $g\left(x\right)=-logx$, convex-decreasing. We have that $CE(\left\{p_i\right\},\left\{q_i\right\})=-log{\mathcal{G}}^{\prime }(\left\{q_i\right\},\left\{p_i\right\})$. Using Definition 7 and applying Corollary 1(b), we have that ${\mathcal{G}}^{\prime }(\left\{q_i\right\},\left\{p_i^{\prime }\right\})\le {\mathcal{G}}^{\prime }(\left\{q_i\right\},\left\{p_i\right\})$, and we apply to each member of the inequality the function $-logx$. □

6.3. Likelihood Ratio Dominance

Definition 8.

Likelihood ratio dominance, LR, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{LR}\left\{{\alpha }_k\right\}$, is defined as

$$\left\{{\alpha }_k^{\prime }\right\}{\prec }_{LR}\left\{{\alpha }_k\right\}\iff \forall (i,j),1\le i,j\le M,i\le j\Rightarrow {\alpha }_i^{\prime }/{\alpha }_j^{\prime }\ge {\alpha }_i/{\alpha }_j.$$

(26)

As ${\alpha }_i^{\prime }/{\alpha }_j^{\prime }\ge {\alpha }_i/{\alpha }_j\Rightarrow {\alpha }_j/{\alpha }_i\ge {\alpha }_j^{\prime }/{\alpha }_i^{\prime }$, we have that LR holds the mirror property.

It can be shown that LR order implies FSD order,

and then LR order holds Theorems 3 and 7 and Equation (16) holds for any mean $\mathcal{M}$,

Theorem 11.

Likelihood-ratio order implies first stochastic order, i.e.,

$$\left\{{\alpha }_k^{\prime }\right\}{\prec }_{LR}\left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}.$$

(27)

Proof. 

There are indirect proofs in the literature [1,17]. A direct proof is in the Appendix C. □

As the condition for LR order, Equation (26), is easy to check, this order comes very handy in proving sufficient condition for FSD order. Additionally, for the uniform distribution $\{1/N\}$ and any increasing distribution $\left\{{\alpha }_k\right\}$, we have that $\{1/M\}{\prec }_{LR}\left\{{\alpha }_k\right\}$, while for any decreasing distribution $\left\{{\alpha }_k\right\}$ we have that $\left\{{\alpha }_k\right\}{\prec }_{LR}\{1/M\}$.

Consider Shannon entropy, $\mathcal{H}\left(\left\{p_k\right\}\right)$, and Rényi entropy ${\mathcal{R}}_{\beta }\left(\left\{p_k\right\}\right)$ of a distribution $\left\{p_k\right\}$, that as seen in Section 3 are quasi-arithmetic means of the sequence $\{1/log{p}_k\}$ with weights $\left\{p_k\right\}$, and with $g\left(x\right)=x$ and $g\left(x\right)=2^{(\beta -1)x}$, respectively. Without loss of generality, we can consider $\left\{p_k\right\}$ in increasing order, then the sequence $\{1/log{p}_k\}$ will be decreasing. We have that $\{1/M\}\prec \left\{p_k\right\}$, and then we can apply Theorem 3, and obtain for Shannon entropy

$$\mathcal{H}\left(\left\{p_k\right\}\right)\le \mathcal{A}\left(\{1/log{p}_k\}\right),$$

where $\mathcal{A}\left(\{1/log{p}_k\}\right)$ is the arithmetic mean of $\{1/log{p}_k\}$, and for Rényi entropy, observing that $g^{-1}\left(x\right)=\frac{1}{1-\beta }logx$,

$${\mathcal{R}}_{\beta }\left(\left\{p_k\right\}\right)\le \frac{1}{1-\beta }log\sum _k\frac{1}{M}{p}_k^{(\beta -1)}=\frac{1}{1-\beta }log\mathcal{A}(\left\{p_k^{(\beta -1)}\right\}.$$

For Tsallis entropy, which can be considered the weighted arithmetic mean of the sequence $\{1/{ln}_q{p}_k\}$ with weights $\left\{p_k\right\}$, as for $\left\{p_k\right\}$ in increasing order the sequence $\{1/{ln}_q{p}_k\}$ will be decreasing (the derivative of ${ln}_{q}x$ is positive), we have similarly to Shannon entropy,

$${\mathcal{S}}_q\left(\left\{p_k\right\}\right)\le \mathcal{A}\left(\{1/{ln}_q{p}_k\}\right).$$

We can also state the following result for the generalized cross entropy $\mathcal{GCE}$. We say that $\left\{p_i\right\}$ is comonotonic [27,28] with $\left\{q_i\right\}$ when, for all $1\le i,j\le M$, $q_i<q_j\Rightarrow p_i\le p_j$.

Theorem 12.

Given distributions $\left\{p_i\right\},\left\{q_i\right\}$ comonotonic, then $\mathcal{GCE}(\left\{p_i\right\},\left\{q_i\right\})\le \mathcal{GCE}(\{1/M\},$$\left\{q_i\right\})$.

Proof. 

Without loss of generality, we can reorder $\left\{p_i\right\},\left\{q_i\right\}$ so that both are increasing. We know that $\{1/M\}{\prec }_{LR}\left\{p_k\right\}$ and thus $\{1/M\}{\prec }_{FSD}\left\{p_k\right\}$. From the definition of ${\prec }_{FSD}$, preserve mean order holds for increasing functions and for arithmetic mean, thus we can then apply Theorem 3 for $\{-log{q}_i\}$ decreasing. □

From Theorem 12, when $\left\{p_i\right\},\left\{q_i\right\}$ are comonotonic, then

$$CE(\left\{p_i\right\},\left\{q_i\right\})\le \frac{1}{M}\sum _{k=1}^M-log{q}_k.$$

(28)

6.4. Hazard Rate

Definition 9.

A probability distribution $\left\{{\alpha }_k\right\}$ is greater in hazard rate, HR, to $\left\{{\alpha }_k^{\prime }\right\}$, $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{HR}\left\{{\alpha }_k\right\}$, if and only if for all $j\ge i$, the following condition is filled

$$\frac{{{\overline{F}}^{\prime }}_i}{{\overline{F}}_i}\ge \frac{{{\overline{F}}^{\prime }}_j}{{\overline{F}}_j}.$$

Observe that this can be written as

$$\frac{{\overline{F}}_j}{{{\overline{F}}^{\prime }}_j}\ge \frac{{\overline{F}}_i}{{{\overline{F}}^{\prime }}_i},$$

or

$$\frac{1-F_j}{1-{F^{\prime }}_j}\ge \frac{1-F_i}{1-{F^{\prime }}_i},$$

If we consider now the sequences written in inverse order, i.e., ${\alpha }_{M-i+1},{\alpha }_{M-i+1}^{\prime }$, it is clear that if $i\le j$ then $M-j+1\le M-i+1$, and from $1-{\overline{F}}_j={\overline{F}}_{M-j+1}$, we get

$$\frac{{\overline{F}}_{M-j+1}}{{{\overline{F}}^{\prime }}_{M-j+1}}\ge \frac{{\overline{F}}_{M-i+1}}{{{\overline{F}}^{\prime }}_{M-i+1}},$$

which means $\left\{{\alpha }_{M-k+1}\right\}{\prec }_{HR}\left\{{\alpha }_{M-k+1}^{\prime }\right\}$, and HR order holds the mirror property.

It can be shown that $\left\{{\alpha }_k^{\prime }\right\}{\prec }_{HR}\left\{{\alpha }_k\right\}\Rightarrow \left\{{\alpha }_k^{\prime }\right\}{\prec }_{FSD}\left\{{\alpha }_k\right\}$ and then HR order holds Theorems 3 and 7, and Equation (16) holds for any mean $\mathcal{M}$.

7. Example: Linear Combination of Monte Carlo Techniques

When we want to estimate an integral $\mu =\int h\left(x\right)dx$ using MIS (multiple importance sampling) Monte Carlo methods, we have several choices of techniques, each of them with a given pdf $\left\{p_k\left(x\right)\right\}$, $1\le k\le n$, which provide the primary estimators $\{I_{k,1}=\frac{h\left(x\right)}{p_k\left(x\right)}\}$. If for $h\left(x\right)>0$, we have that $p_k\left(x\right)>0$, then the technique is unbiased. We are interested in optimal ways of combining the techniques. One option is linearly combining the different estimators $\left\{I_{i,n_i}\right\}$, ${\sum }_{i=1}^M{w}_i{I}_{i,n_i}$, with weights $\left\{w_k\right\}$ and sampling proportions $\{n_i={\alpha }_{i}N\}$, with $N={\sum }_{i=1}^M{n}_i$. If all techniques are unbiased, the resulting combination is also unbiased. The variance is given by

$$V_N=\sum _{i=1}^M{w}_i^2\frac{v_i}{n_i}=\frac{1}{N}\sum _{i=1}^M{w}_i^2\frac{v_i}{{\alpha }_i}=\frac{1}{N}V,$$

(29)

where $v_i$ are the variances for the primary estimators of each technique and V is the variance for the primary MIS estimator.

The optimal combination of weights, i.e. the one that leads to minimum variance, has been studied in [18,29,30].

The variance value will depend on two sets of weights, $\left\{w_i\right\}$ and $\left\{{\alpha }_i\right\}$, but there are cases where we can reduce it to a single set of weights. We present the following examples, where variances are taken for $N=1$:

• when ${\alpha }_i=w_i$, then $V={\sum }_{i=1}^M{\alpha }_i{v}_i$, the weighted arithmetic mean ($g\left(x\right)=x$) of $\left\{v_i\right\}$.
• when the sampling proportions are fixed, the optimal variance is given by $\mathcal{H}\left(\left\{\frac{v_k}{{\alpha }_k}\right\}\right)$, the weighted harmonic mean ($g\left(x\right)=1/x$) of $\left\{v_i\right\}$.
• when weights $\left\{w_i\right\}$ are fixed, the optimal variance is given by ${\left({\sum }_{k=1}^M{w}_k{\sqrt{v}}_k\right)}^2$, which is the weighted power mean of $\left\{v_i\right\}$ with exponent $r=1/2$ ($g\left(x\right)=x^{1/2}$).

Observe that the variance in these three cases is a quasi-arithmetic mean.

Let us now order $\left\{v_k\right\}$ in increasing order, and let us take ${\alpha }_k^{\prime }=h\left(v_k\right)$, $h\left(x\right)$ decreasing, and ${\alpha }_k=1/M$. We have that ${\alpha }_k^{\prime }=h\left(v_k\right){\prec }_{LR}\{{\alpha }_k=1/M\}$ and thus, by Theorem 11, ${\alpha }_k^{\prime }=h\left(v_k\right){\prec }_{FSD}\{{\alpha }_k=1/M\}$, and we can apply Theorem 2 with $f\left(k\right)=v_k$ to the quasi-arithmetic means above. Thus, the variance is less when taking sampling proportions or coefficients decreasing in $v_k$ than for equal sampling or equal weighting.

The same would be the case when considering a different measure of error, whenever the error of the combined techniques can also be expressed as a weighted quasi-arithmetic mean of the values of this measure for all techniques. For instance, suppose we use as measure of error the standard deviation, ${\sigma }_i^2=v_i$, ${\sigma }^2=V$, the above cases become

• when ${\alpha }_i=w_i$, then $\sigma =\sqrt{V}=\sqrt{{\sum }_{i=1}^M{\alpha }_i{\sigma }_i^2}$, the weighted root mean square or quadratic mean of $\left\{{\sigma }_i\right\}$, or weighted power mean with $r=2$ ($g\left(x\right)=x^2$).
• when the weights $\left\{w_i\right\}$ are fixed, the optimal standard deviation is given by $\sigma =\sqrt{V}=\sqrt{H\left(\left\{\frac{{\sigma }_k^2}{{\alpha }_k}\right\}\right)}$, the power mean of $\left\{{\sigma }_i\right\}$ with $r=-2$ ($g\left(x\right)=x^{-2}$).
• when the sampling proportions are fixed, the optimal standard deviation is given by $\sigma =\sqrt{V}={\sum }_{k=1}^M{w}_k{\sigma }_k$, the arithmetic mean of $\left\{{\sigma }_i\right\}$ ($g\left(x\right)=x$).

For more details see [18].

Liabilities vs. Utilities

From the above example, we can study the relationship between utilities, which we try to maximize, and liabilities, that we try to minimize. As a liability can always be considered as the inverse of a utility, we can see from Theorem 1 that establishing invariance properties on the order between means of utilities is equivalent to doing this with liabilities, except that the order is inverted. We can define for instance the efficiency $E=1/V$ as the inverse of the variance, and obtain it as a weighted mean of individual technique’s efficiencies, $e_i=1/v_i$. Consider, for instance, the first case in the example above, where $V={\sum }_{i=1}^M{\alpha }_i{v}_i$. We have

$$E=\frac{1}{V}=\frac{1}{{\sum }_{i=1}^M{\alpha }_i{v}_i}=\frac{1}{{\sum }_{i=1}^M{\alpha }_i{e}_i^{-1}}=\mathcal{H}\left(\left\{\frac{e_i}{{\alpha }_i}\right\}\right),$$

where $\mathcal{H}$ denotes the weighted harmonic mean.

Considering now the second case

$$E=\frac{1}{V}=\frac{1}{\mathcal{H}\left(\left\{\frac{v_k}{{\alpha }_k}\right\}\right)}=\sum _{i=1}^M{\alpha }_i{v}_i^{-1}=\sum _{i=1}^M{\alpha }_i{e}_i,$$

or weighted arithmetic mean. For the third case,

$$E=\frac{1}{V}=\frac{1}{{\left({\sum }_{k=1}^M{w}_k{\sqrt{v}}_k\right)}^2}={\left(\sum _{k=1}^M{w}_k{e}_k^{-1/2}\right)}^{-2},$$

which is the weighted power mean with $r=-1/2$.

8. Conclusions and Future Work

We have presented in this paper the relationship between stochastic orders and quasi-arithmetic means. We have proved several ordering invariance theorems, that show that given two distributions under a certain stochastic order, the ordering of the means is preserved for any quasi-arithmetic mean we might consider, this is, not only for the arithmetic mean (or expected value). We have shown how the results apply to first order, second order, likelihood ratio, hazard-rate, and increasing convex stochastic orders, and its application to cross-entropy. We have also presented an application example based on the linear combination of Monte Carlo estimators, and shown that the invariance allows costs or liabilities to be considered as the symmetric case of utilities.

In the future, we want to generalize our results to spatial weight matrices [31]. The rows in a spatial weight matrix are weights that give the influence of n entities over each other. Different weighted means such as arithmetic, harmonic, or geometric [32] can be used to compute this influence. We can thus apply our invariance results to each row. We will also investigate which of our results for Shannon entropy extend to Tsallis entropy too. Both Shannon and Tsallis entropy are weighted arithmetic means [23], and given $\left\{p_k\right\}$ monotonic, both $\{1/log{p}_k\}$ and $\{1/{ln}_q{p}_k\}$ are monotonic too. Finally, we will investigate the invariance of the different stochastic orders under the operations of compositional data [3].