On Representations of Divergence Measures and Related Quantities in Exponential Families

Abstract

Within exponential families, which may consist of multi-parameter and multivariate distributions, a variety of divergence measures, such as the Kullback–Leibler divergence, the Cressie–Read divergence, the Rényi divergence, and the Hellinger metric, can be explicitly expressed in terms of the respective cumulant function and mean value function. Moreover, the same applies to related entropy and affinity measures. We compile representations scattered in the literature and present a unified approach to the derivation in exponential families. As a statistical application, we highlight their use in the construction of confidence regions in a multi-sample setup.

1. Introduction

There is a broad literature on divergence and distance measures for probability distributions, e.g., on the Kullback–Leibler divergence, the Cressie–Read divergence, the Rényi divergence, and Phi divergences as a general family, as well as on associated measures of entropy and affinity. For definitions and details, we refer to [1]. These measures have been extensively used in statistical inference. Excellent monographs on this topic were provided by Liese and Vajda [2], Vajda [3], Pardo [1], and Liese and Miescke [4].

Within an exponential family as defined in Section 2, which may consist of multi-parameter and multivariate distributions, several divergence measures and related quantities are seen to have nice explicit representations in terms of the respective cumulant function and mean value function. These representations are contained in different sources. Our focus is on a unifying presentation of main quantities, while not aiming at an exhaustive account. As an application, we derive confidence regions for the parameters of exponential distributions based on different divergences in a simple multi-sample setup.

For the use of the aforementioned measures of divergence, entropy, and affinity, we refer to the textbooks [1,2,3,4] and exemplarily to [5,6,7,8,9,10] for statistical applications, including the construction of test procedures as well as methods based on dual representations of divergences, and to [11] for a classification problem.

2. Exponential Families

Let $\Theta \ne \varnothing$ be a parameter set, $\mu$ be a $\sigma$-finite measure on the measurable space $(\mathcal{X},\mathcal{B})$, and $\mathcal{P}=\{P_{\vartheta }:\vartheta \in \Theta \}$ be an exponential family (EF) of distributions on $(\mathcal{X},\mathcal{B})$ with $\mu$-density

$$f_{\vartheta }\left(x\right)=C\left(\vartheta \right)exp\left\{\sum _{j=1}^k{Z}_j\left(\vartheta \right)T_j\left(x\right)\right\}h\left(x\right),x\in \mathcal{X},$$

(1)

of $P_{\vartheta }$ for $\vartheta \in \Theta$, where $C,Z_1,\dots ,Z_k:\Theta \to \mathbb{R}$ are real-valued functions on $\Theta$ and $h,T_1,\dots ,T_k:(\mathcal{X},\mathcal{B})\to ({\mathbb{R}}^1,{\mathbb{B}}^1)$ are real-valued Borel-measurable functions with $h\ge 0$. Usually, $\mu$ is either the counting measure on the power set of $\mathcal{X}$ (for a family of discrete distributions) or the Lebesgue measure on the Borel sets of $\mathcal{X}$ (in the continuous case). Without loss of generality and for a simple notation, we assume that $h>0$ (the set $\{x\in \mathcal{X}:h(x)=0\}$ is a null set for all $P\in \mathcal{P}$). Let $\nu$ denote the $\sigma$-finite measure with $\mu$-density h.

We assume that representation (1) is minimal in the sense that the number k of summands in the exponent cannot be reduced. This property is equivalent to $Z_1,\dots ,Z_k$ being affinely independent mappings and $T_1,\dots ,T_k$ being $\nu$-affinely independent mappings; see, e.g., [12] (Cor. 8.1). Here, $\nu$-affine independence means affine independence on the complement of every null set of $\nu$.

To obtain simple formulas for divergence measures in the following section, it is convenient to use the natural parameter space

$${\Xi }^{\ast }=\left\{\mathit{\zeta }\in {\mathbb{R}}^k:\int e^{{\mathit{\zeta }}^t\mathit{T}}hd\mu <\infty \right\}$$

and the (minimal) canonical representation $\{P_{\mathit{\zeta }}^{\ast }:\mathit{\zeta }\in \mathit{Z}(\Theta )\}$ of $\mathcal{P}$ with $\mu$-density

$$f_{\mathit{\zeta }}^{\ast }\left(x\right)=C^{\ast }\left(\mathit{\zeta }\right)e^{{\mathit{\zeta }}^t\mathit{T}\left(x\right)}h\left(x\right),x\in \mathcal{X},$$

(2)

of $P_{\mathit{\zeta }}^{\ast }$ and normalizing constant $C^{\ast }\left(\mathit{\zeta }\right)$ for $\mathit{\zeta }={({\zeta }_1,\dots ,{\zeta }_k)}^t\in \mathit{Z}(\Theta )\subset {\Xi }^{\ast }$, where $\mathit{Z}={(Z_1,\dots ,Z_k)}^t$ denotes the (column) vector of the mappings $Z_1,\dots ,Z_k$ and $\mathit{T}={(T_1,\dots ,T_k)}^t$ denotes the (column) vector of the statistics $T_1,\dots ,T_k$. For simplicity, we assume that $\mathcal{P}$ is regular, i.e., we have that $\mathit{Z}(\Theta )={\Xi }^{\ast }$ ($\mathcal{P}$ is full) and that ${\Xi }^{\ast }$ is open; see [13]. In particular, this guarantees that $\mathit{T}$ is minimal sufficient and complete for $\mathcal{P}$; see, e.g., [14] (pp. 25–27).

The cumulant function

$$\kappa \left(\mathit{\zeta }\right)=-ln\left(C^{\ast }\left(\mathit{\zeta }\right)\right),\mathit{\zeta }\in {\Xi }^{\ast },$$

associated with $\mathcal{P}$ is strictly convex and infinitely often differentiable on the convex set ${\Xi }^{\ast }$; see [13] (Theorem 1.13 and Theorem 2.2). It is well-known that the Hessian matrix of $\kappa$ at $\mathit{\zeta }$ coincides with the covariance matrix of $\mathit{T}$ under $P_{\mathit{\zeta }}^{\ast }$ and that it is also equal to the Fisher information matrix $\mathbf{I}\left(\mathit{\zeta }\right)$ at $\mathit{\zeta }$. Moreover, by introducing the mean value function

$$\pi \left(\mathit{\zeta }\right)=E_{\mathit{\zeta }}\left[\mathit{T}\right],\mathit{\zeta }\in {\Xi }^{\ast },$$

(3)

we have the useful relation

$$\pi =\nabla \kappa ,$$

(4)

where $\nabla \kappa$ denotes the gradient of $\kappa$; see [13] (Cor. 2.3). $\pi$ is a bijective mapping from ${\Xi }^{\ast }$ to the interior of the convex support of ${\nu }^{\mathit{T}}$, i.e., the closed convex hull of the support of ${\nu }^{\mathit{T}}$; see [13] (p. 2 and Theorem 3.6).

Finally, note that representation (2) can be rewritten as

$$f_{\mathit{\zeta }}^{\ast }\left(x\right)=e^{{\mathit{\zeta }}^t\mathit{T}\left(x\right)-\kappa \left(\mathit{\zeta }\right)}h\left(x\right),x\in \mathcal{X},$$

(5)

for $\mathit{\zeta }\in {\Xi }^{\ast }$.

3. Divergence Measures

Divergence measures may be applied, for instance, to quantify the “disparity” of a distribution to some reference distribution or to measure the “distance” between two distributions within some family in a certain sense. If the distributions in the family are dominated by a $\sigma$-finite measure, various divergence measures have been introduced by means of the corresponding densities. In parametric statistical inference, they serve to construct statistical tests or confidence regions for underlying parameters; see, e.g., [1].

Definition 1.

Let $\mathcal{F}$ be a set of distributions on $(\mathcal{X},\mathcal{B})$. A mapping $D:\mathcal{F}\times \mathcal{F}\to \mathbb{R}$ is called a divergence (or divergence measure) if:

(i) 

$D(P,Q)\ge 0$ for all $P,Q\in \mathcal{F}$ and $D(P,Q)=0\iff P=Q$ (positive definiteness).

If additionally

(ii) 

$D(P,Q)=D(Q,P)$ for all $P,Q\in \mathcal{F}$ (symmetry) is valid, D is called a distance (or distance measure or semi-metric). If D then moreover meets

(iii) 

$D(P_1,P_2)\le D(P_1,Q)+D(Q,P_2)$ for all $P_1,P_2,Q\in \mathcal{F}$ (triangle inequality), D is said to be a metric.

Some important examples are the Kullback–Leibler divergence (KL-divergence):

$$D_{KL}(P_1,P_2)=\int f_{1}ln\left(\frac{f_1}{f_2}\right)d\mu ,$$

the Jeffrey distance:

$$D_J(P_1,P_2)=D_{KL}(P_1,P_2)+D_{KL}(P_2,P_1)$$

as a symmetrized version, the Rényi divergence:

$$D_{R_q}(P_1,P_2)=\frac{1}{q(q-1)}ln\left(\int f_1^q{f}_2^{1-q}d\mu \right),q\in \mathbb{R}\setminus \{0,1\},$$

(6)

along with the related Bhattacharyya distance $D_B(P_1,P_2)=D_{R_{1/2}}(P_1,P_2)/4$, the Cressie–Read divergence (CR-divergence):

$$D_{C{R}_q}(P_1,P_2)=\frac{1}{q(q-1)}\int f_1\left[{\left(\frac{f_1}{f_2}\right)}^{q-1}-1\right]d\mu ,q\in \mathbb{R}\setminus \{0,1\},$$

(7)

which is the same as the Chernoff $\alpha$-divergence up to a parameter transformation, the related Matusita distance $D_M(P_1,P_2)=D_{C{R}_{1/2}}(P_1,P_2)/2$, and the Hellinger metric:

$$D_H(P_1,P_2)={\left[\int {\left(\sqrt{f_1}-\sqrt{f_2}\right)}^{2}d\mu \right]}^{1/2}$$

(8)

for distributions $P_1,P_2\in \mathcal{F}$ with $\mu$-densities $f_1,f_2$, provided that the integrals are well-defined and finite.

$D_{KL}$, $D_{R_q}$, and $D_{C{R}_q}$ for $q\in \mathbb{R}\setminus \{0,1\}$ are divergences, and $D_J$, $D_{R_{1/2}}$, $D_B$, $D_{C{R}_{1/2}}$, and $D_M(=D_H^2)$, since they moreover satisfy symmetry, are distances on $\mathcal{F}\times \mathcal{F}$. $D_H$ is known to be a metric on $\mathcal{F}\times \mathcal{F}$.

In parametric models, it is convenient to use the parameters as arguments and briefly write, e.g.,

$$D_{KL}({\vartheta }_1,{\vartheta }_2)\mathrm{for}{D}_{KL}(P_{{\vartheta }_1},P_{{\vartheta }_2}),{\vartheta }_1,{\vartheta }_2\in \Theta ,$$

if the parameter $\vartheta \in \Theta$ is identifiable, i.e., if the mapping $\vartheta \mapsto P_{\vartheta }$ is one-to-one on $\Theta$. This property is met for the EF $\mathcal{P}$ in Section 2 with minimal canonical representation (5); see, e.g., [13] (Theorem 1.13(iv)).

It is known from different sources in the literature that the EF structure admits simple formulas for the above divergence measures in terms of the corresponding cumulant function and/or mean value function. For the KL-divergence, we refer to [15] (Cor. 3.2) and [13] (pp. 174–178), and for the Jeffrey distance also to [16].

Theorem 1.

Let $\mathcal{P}$ be as in Section 2 with minimal canonical representation (5). Then, for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$, we have

$$\begin{array}{ccc}\hfill D_{KL}(\mathit{\zeta },\mathit{\eta })& =& \kappa \left(\mathit{\eta }\right)-\kappa \left(\mathit{\zeta }\right)+{(\mathit{\zeta }-\mathit{\eta })}^t\pi \left(\mathit{\zeta }\right)\hfill \\ \hfill and{D}_J(\mathit{\zeta },\mathit{\eta })& =& {(\mathit{\zeta }-\mathit{\eta })}^t(\pi \left(\mathit{\zeta }\right)-\pi \left(\mathit{\eta }\right)).\hfill \end{array}$$

(9)

Proof. 

By using Formulas (3) and (5), we obtain for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ that

$$\begin{array}{ccc}\hfill D_{KL}(\mathit{\zeta },\mathit{\eta })& =& \int \left[ln\left(f_{\mathit{\zeta }}^{\ast }\right)-ln\left(f_{\mathit{\eta }}^{\ast }\right)\right]f_{\mathit{\zeta }}^{\ast }d\mu \hfill \\ \hfill & =& \int \left[{(\mathit{\zeta }-\mathit{\eta })}^t\mathit{T}-\kappa \left(\mathit{\zeta }\right)+\kappa \left(\mathit{\eta }\right)\right]f_{\mathit{\zeta }}^{\ast }d\mu \hfill \\ \hfill & =& \kappa \left(\mathit{\eta }\right)-\kappa \left(\mathit{\zeta }\right)+{(\mathit{\zeta }-\mathit{\eta })}^t\pi \left(\mathit{\zeta }\right).\hfill \end{array}$$

From this, the representation of $D_J$ is obvious. □

As a consequence of Theorem 1, $D_{KL}$ and $D_J$ are infinitely often differentiable on ${\Xi }^{\ast }\times {\Xi }^{\ast }$, and the derivatives are easily obtained by making use of the EF properties. For example, by using Formula (4), we find $\nabla D_{KL}(\mathit{\zeta },·)=\pi (·)-\pi \left(\mathit{\zeta }\right)$ and that the Hessian matrix of $D_{KL}(\mathit{\zeta },·)$ at $\mathit{\eta }$ is the Fisher information matrix $\mathbf{I}\left(\mathit{\eta }\right)$, where $\mathit{\zeta }\in {\Xi }^{\ast }$ is considered to be fixed.

Moreover, we obtain from Theorem 1 that the reverse KL-divergence $D_{KL}^{\ast }(\mathit{\zeta },\mathit{\eta })=D_{KL}(\mathit{\eta },\mathit{\zeta })$ for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ is nothing but the Bregman divergence associated with the cumulant function $\kappa$; see, e.g., [1,11,17]. As an obvious consequence of Theorem 1, other symmetrizations of the KL-divergence may be expressed in terms of $\kappa$ and $\pi$ as well, such as the so-called resistor-average distance (cf. [18])

$$\begin{array}{ccc}\hfill D_{RA}(\mathit{\zeta },\mathit{\eta })& =& 2{\left(\frac{1}{D_{KL}(\mathit{\zeta },\mathit{\eta })}+\frac{1}{D_{KL}(\mathit{\eta },\mathit{\zeta })}\right)}^{-1}\hfill \\ \hfill & =& \frac{2D_{KL}(\mathit{\zeta },\mathit{\eta })D_{KL}(\mathit{\eta },\mathit{\zeta })}{D_J(\mathit{\zeta },\mathit{\eta })},\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast },\mathit{\zeta }\ne \mathit{\eta },\hfill \end{array}$$

(10)

with $D_{RA}(\mathit{\zeta },\mathit{\zeta })=0$, $\mathit{\zeta }\in {\Xi }^{\ast }$, or the distance

$$D_{GA}(\mathit{\zeta },\mathit{\eta })={\left[D_{KL}(\mathit{\zeta },\mathit{\eta })D_{KL}(\mathit{\eta },\mathit{\zeta })\right]}^{1/2},\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast },$$

(11)

obtained by taking the harmonic and geometric mean of $D_{KL}$ and $D_{KL}^{\ast }$; see [19].

Remark 1.

Formula (9) can be used to derive the test statistic

$$\Lambda \left(x\right)=-2ln\left(\frac{{sup}_{\mathit{\zeta }\in {\Xi }_0}{f}_{\mathit{\zeta }}^{\ast }\left(x\right)}{{sup}_{\mathit{\zeta }\in {\Xi }^{\ast }}{f}_{\mathit{\zeta }}^{\ast }\left(x\right)}\right),x\in \mathcal{X},$$

of the likelihood-ratio test for the test problem

$$H_0:\mathit{\zeta }\in {\Xi }_{0}against{H}_1:\mathit{\zeta }\in {\Xi }^{\ast }\setminus {\Xi }_0,$$

where $\varnothing \ne {\Xi }_0⊊{\Xi }^{\ast }$. If the maximum likelihood estimators (MLEs) $\widehat{\mathit{\zeta }}=\widehat{\mathit{\zeta }}\left(x\right)$ and ${\widehat{\mathit{\zeta }}}_0={\widehat{\mathit{\zeta }}}_0\left(x\right)$ of ζ in ${\Xi }^{\ast }$ and ${\Xi }_0$ (based on x) both exist, we have:

$$\begin{array}{ccc}\hfill \Lambda & =& 2\left[ln\left(f_{\widehat{\mathit{\zeta }}}^{\ast }\right)-ln\left(f_{{\widehat{\mathit{\zeta }}}_0}^{\ast }\right)\right]\hfill \\ \hfill & =& 2\left[\kappa \left({\widehat{\mathit{\zeta }}}_0\right)-\kappa \left(\widehat{\mathit{\zeta }}\right)+{(\widehat{\mathit{\zeta }}-{\widehat{\mathit{\zeta }}}_0)}^t\mathit{T}\right]\hfill \\ \hfill & =& 2D_{KL}(\widehat{\mathit{\zeta }},{\widehat{\mathit{\zeta }}}_0)\hfill \end{array}$$

by using that the unrestricted MLE fulfils $\pi \left(\widehat{\mathit{\zeta }}\right)=\mathit{T}$; see, e.g., [12] (p. 190) and [13] (Theorem 5.5). In particular, when testing a simple null hypothesis with ${\Xi }_0=\left\{\mathit{\eta }\right\}$ for some fixed $\mathit{\eta }\in {\Xi }^{\ast }$, we have $\Lambda =2D_{KL}(\widehat{\mathit{\zeta }},\mathit{\eta })$.

Convenient representations within EFs of the divergences in Formulas (6)–(8) can also be found in the literature; we refer to [2] (Prop. 2.22) for $D_{R_q}$, $D_H$, and $D_M$, to [20] for $D_B$, and to [9] for $D_{R_q}$. The formulas may all be obtained by computing the quantity

$$A_q(P_1,P_2)=\int f_1^q{f}_2^{1-q}d\mu ,q\in \mathbb{R}\setminus \{0,1\}.$$

(12)

For $q\in (0,1)$, we have the following identity (cf. [21]).

Lemma 1.

Let $\mathcal{P}$ be as in Section 2 with minimal canonical representation (5). Then, for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ and $q\in (0,1)$, we have:

$$A_q(\mathit{\zeta },\mathit{\eta })=exp\left\{\kappa (q\mathit{\zeta }+(1-q\left)\mathit{\eta }\right)-\left[q\kappa \right(\mathit{\zeta })+(1-q\left)\kappa \right(\mathit{\eta }\left)\right]\right\}.$$

Proof. 

Let $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ and $q\in (0,1)$. Then,

$$\begin{array}{ccc}\hfill A_q(\mathit{\zeta },\mathit{\eta })& =& \int {\left(f_{\mathit{\zeta }}^{\ast }\right)}^q{\left(f_{\mathit{\eta }}^{\ast }\right)}^{1-q}d\mu \hfill \\ \hfill & =& \int exp\left\{{(q\mathit{\zeta }+(1-q)\mathit{\eta })}^t\mathit{T}-[q\kappa \left(\mathit{\zeta }\right)+(1-q)\kappa \left(\mathit{\eta }\right)]\right\}hd\mu \hfill \\ \hfill & =& exp\left\{\kappa (q\mathit{\zeta }+(1-q\left)\mathit{\eta }\right)-\left[q\kappa \right(\mathit{\zeta })+(1-q\left)\kappa \right(\mathit{\eta }\left)\right]\right\},\hfill \end{array}$$

where the convexity of ${\Xi }^{\ast }$ ensures that $\kappa (q\mathit{\zeta }+(1-q\left)\mathit{\eta }\right)$ is defined. □

Remark 2.

For arbitrary divergence measures, several transformations and skewed versions as well as symmetrization methods, such as the Jensen–Shannon symmetrization, are studied in [19]. Applied to the KL-divergence, the skew Jensen–Shannon divergence is introduced as

$$D_{J{S}_q}(P_1,P_2)=q{D}_{KL}(P_1,q{P}_1+(1-q)P_2)+(1-q)D_{KL}(P_2,q{P}_1+(1-q)P_2)$$

for $P_1,P_2\in \mathcal{P}$ and $q\in (0,1)$, which includes the Jensen–Shannon distance for $q=1/2$ (the distance $D_{J{S}_{1/2}}^{1/2}$ even forms a metric). Note that, for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$, the density $q{f}_{\mathit{\zeta }}^{\ast }+(1-q)f_{\mathit{\eta }}^{\ast }$ of the mixture $q{P}_{\mathit{\zeta }}^{\ast }+(1-q)P_{\mathit{\eta }}^{\ast }$ does not belong to $\mathcal{P}$, in general, such that the identity in Theorem 1 for the KL-divergence is not applicable, here.

However, from the proof of Lemma 1, it is obvious that

$$\frac{1}{A_q(\mathit{\zeta },\mathit{\eta })}{\left(f_{\mathit{\zeta }}^{\ast }\right)}^q{\left(f_{\mathit{\eta }}^{\ast }\right)}^{1-q}=f_{q\mathit{\zeta }+(1-q)\mathit{\eta }}^{\ast },\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast },q\in (0,1),$$

i.e., the EF $\mathcal{P}$ is closed when forming normalized weighted geometric means of the densities. This finding is utilized in [19] to introduce another version of the skew Jensen–Shannon divergence based on the KL-divergence, where the weighted arithmetic mean of the densities is replaced by the normalized weighted geometric mean. The skew geometric Jensen–Shannon divergence thus obtained is given by

$$\begin{array}{c}\hfill D_{GJ{S}_q}(\mathit{\zeta },\mathit{\eta })=q{D}_{KL}(\mathit{\zeta },q\mathit{\zeta }+(1-q)\mathit{\eta })+(1-q)D_{KL}(\mathit{\eta },q\mathit{\zeta }+(1-q)\mathit{\eta }),\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast },\end{array}$$

for $q\in (0,1)$. By using Theorem 1, we find

$$\begin{array}{ccc}\hfill D_{GJ{S}_q}(\mathit{\zeta },\mathit{\eta })& =& q\left[\kappa \left(q\mathit{\zeta }+(1-q)\mathit{\eta }\right)-\kappa \left(\mathit{\zeta }\right)+(1-q){(\mathit{\zeta }-\mathit{\eta })}^t\pi \left(\mathit{\zeta }\right)\right]\hfill \\ & & +(1-q)\left[\kappa \left(q\mathit{\zeta }+(1-q)\mathit{\eta }\right)-\kappa \left(\mathit{\eta }\right)+q{(\mathit{\eta }-\mathit{\zeta })}^t\pi \left(\mathit{\eta }\right)\right]\hfill \\ \hfill & =& \kappa \left(q\mathit{\zeta }+(1-q)\mathit{\eta }\right)\hfill \\ & & -\left[q\kappa \left(\mathit{\zeta }\right)+(1-q)\kappa \left(\mathit{\eta }\right)\right]+q(1-q){(\mathit{\zeta }-\mathit{\eta })}^t\left[\pi \left(\mathit{\zeta }\right)-\pi \left(\mathit{\eta }\right)\right]\hfill \\ \hfill & =& ln\left(A_q(\mathit{\zeta },\mathit{\eta })\right)+q(1-q)D_J(\mathit{\zeta },\mathit{\eta }),\hfill \end{array}$$

(13)

for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ and $q\in (0,1)$.

In particular, setting $q=1/2$ gives the geometric Jensen–Shannon distance:

$$D_{GJS}(\mathit{\zeta },\mathit{\eta })=\kappa \left(\frac{\mathit{\zeta }+\mathit{\eta }}{2}\right)-\frac{\kappa \left(\mathit{\zeta }\right)+\kappa \left(\mathit{\eta }\right)}{2}+\frac{{(\mathit{\zeta }-\mathit{\eta })}^t\left[\pi \left(\mathit{\zeta }\right)-\pi \left(\mathit{\eta }\right)\right]}{4},\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }.$$

For more details and properties as well as related divergence measures, we refer to [19,22].

Formulas for $D_{R_q}$, $D_{C{R}_q}$, and $D_H$ are readily deduced from Lemma 1.

Theorem 2.

Let $\mathcal{P}$ be as in Section 2 with minimal canonical representation (5). Then, for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ and $q\in (0,1)$, we have

$$\begin{array}{ccc}\hfill D_{R_q}(\mathit{\zeta },\mathit{\eta })& =& \frac{1}{q(q-1)}\left[\kappa (q\mathit{\zeta }+(1-q\left)\mathit{\eta }\right)-\left[q\kappa \right(\mathit{\zeta })+(1-q\left)\kappa \right(\mathit{\eta }\left)\right]\right],\hfill \\ \hfill D_{C{R}_q}(\mathit{\zeta },\mathit{\eta })& =& \frac{1}{q(q-1)}\left[exp\left\{\kappa (q\mathit{\zeta }+(1-q\left)\mathit{\eta }\right)-\left[q\kappa \right(\mathit{\zeta })+(1-q\left)\kappa \right(\mathit{\eta }\left)\right]\right\}-1\right],\hfill \\ \hfill and{D}_H(\mathit{\zeta },\mathit{\eta })& =& {\left(2-2exp\left\{\kappa \left(\frac{\mathit{\zeta }+\mathit{\eta }}{2}\right)-\frac{\kappa \left(\mathit{\zeta }\right)+\kappa \left(\mathit{\eta }\right)}{2}\right\}\right)}^{1/2}.\hfill \end{array}$$

Proof. 

Since

$$D_{R_q}=\frac{ln\left(A_q\right)}{q(q-1)},D_{C{R}_q}=\frac{A_q-1}{q(q-1)},\mathrm{and}{D}_H={\left(2-2A_{1/2}\right)}^{1/2},$$

the assertions are directly obtained from Lemma 1. □

It is well-known that

$$\underset{q\to 1}{lim}{D}_{R_q}(P_1,P_2)=D_{KL}(P_1,P_2)\mathrm{and}\underset{q\to 0}{lim}{D}_{R_q}(P_1,P_2)=D_{KL}(P_2,P_1),$$

such that Formula (9) results from the representation of the Rényi divergence in Theorem 2 by sending q to 1.

The Sharma–Mittal divergence (see [1]) is closely related to the Rényi divergence as well and, by Theorem 2, a representation in EFs is available.

Moreover, representations within EFs for so-called local divergences can be derived as, e.g., the Cressie–Read local divergence, which results from the CR-divergence by multiplying the integrand with some kernel density function; see [23].

Remark 3.

Inspecting the proof of Theorem 2, $D_{R_q}$ and $D_{C{R}_q}$ are seen to be strictly decreasing functions of $A_q$ for $q\in (0,1)$; for $q=1/2$, this is also true for $D_H$. From an inferential point of view, this finding yields that, for fixed $q\in (0,1)$, test statistics and pivot statistics based on these divergence measures will lead to the same test and confidence region, respectively. This is not the case within some divergence families such as $D_{R_q}$, $q\in (0,1)$, where different values of q correspond to different tests and confidence regions, in general.

A more general form of the Hellinger metric is given by

$$D_{H,m}(P_1,P_2)={\left(\int |f_1^{1/m}-f_2^{1/m}{|}^{m}d\mu \right)}^{1/m}$$

for $m\in \mathbb{N}$, where $D_{H,2}=D_H$; see Formula (8). For $m\in 2\mathbb{N}$, i.e., if m is even, the binomial theorem then yields

$$\begin{array}{ccc}\hfill {\left[D_{H,m}(P_1,P_2)\right]}^m& =& \int {\left(f_1^{1/m}-f_2^{1/m}\right)}^{m}d\mu \hfill \\ \hfill & =& \sum _{k=0}^m{(-1)}^k\left(\genfrac{}{}{0pt}{}{m}{k}\right)\int f_1^{k/m}{f}_2^{(m-k)/m}d\mu \hfill \\ \hfill & =& \sum _{k=0}^m{(-1)}^k\left(\genfrac{}{}{0pt}{}{m}{k}\right)A_{k/m}(P_1,P_2),\hfill \end{array}$$

and inserting for $A_{k/m}$, $k=1,1,\dots ,m-1$, according to Lemma 1 along with $A_0\equiv 1\equiv A_1$ gives a formula for $D_{H,m}$ in terms of the cumulant function of the EF $\mathcal{P}$ in Section 2. This representation is stated in [16].

Note that the representation for $A_q$ in Lemma 1 (and thus the formulas for $D_{R_q}$ and $D_{C{R}_q}$ in Theorem 2) are also valid for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ and $q\in \mathbb{R}\setminus [0,1]$ as long as $q\mathit{\zeta }+(1-q)\mathit{\eta }\in {\Xi }^{\ast }$ is true. This can be used, e.g., to find formulas for $D_{C{R}_2}$ and $D_{C{R}_{-1}}$, which coincide with the Pearson ${\chi }^2$-divergence

$$\begin{array}{ccc}\hfill D_{{\chi }^2}(\mathit{\zeta },\mathit{\eta })& =& \frac{1}{2}\int \frac{{(f_{\mathit{\zeta }}^{\ast }-f_{\mathit{\eta }}^{\ast })}^2}{f_{\mathit{\eta }}^{\ast }}d\mu =\frac{1}{2}\left[A_2(\mathit{\zeta },\mathit{\eta })-1\right]\hfill \\ \hfill & =& \frac{1}{2}\left[exp\left\{\kappa \left(2\mathit{\zeta }-\mathit{\eta }\right)-2\kappa \left(\mathit{\zeta }\right)+\kappa \left(\mathit{\eta }\right)\right\}-1\right]\hfill \end{array}$$

for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ with $2\mathit{\zeta }-\mathit{\eta }\in {\Xi }^{\ast }$ and the reverse Pearson ${\chi }^2$-divergence (or Neyman ${\chi }^2$-divergence) $D_{{\chi }^2}^{\ast }(\mathit{\zeta },\mathit{\eta })=D_{{\chi }^2}(\mathit{\eta },\mathit{\zeta })$ for $\mathit{\zeta },\mathit{\eta }\in {\Xi }^{\ast }$ with $2\mathit{\eta }-\mathit{\zeta }\in {\Xi }^{\ast }$. Here, the restrictions on the parameters are obsolete if ${\Xi }^{\ast }={\mathbb{R}}^k$ for some $k\in \mathbb{N}$, which is the case for the EF of Poisson distributions and for any EF of discrete distributions with finite support such as binomial or multinomial distributions (with $n\in \mathbb{N}$ fixed). Moreover, quantities similar to $A_q$ such as $\int f_{\mathit{\zeta }}^{\ast }{\left(f_{\mathit{\eta }}^{\ast }\right)}^{\gamma }d\mu$ for $\gamma >0$ arise in the so-called $\gamma$-divergence, for which some representations can also be obtained; see [24] (Section 4).

Remark 4.

If the assumption of the EF $\mathcal{P}$ to be regular is weakened to $\mathcal{P}$ being steep, Lemma 1 and Theorem 2 remain true; moreover, the formulas in Theorem 1 are valid for ζ lying in the interior of ${\Xi }^{\ast }$. Steep EFs are full EFs in which boundary points of ${\Xi }^{\ast }$ that belong to ${\Xi }^{\ast }$ satisfy a certain property. A prominent example is provided by the full EF of inverse normal distributions. For details, see, e.g., [13].

The quantity $A_q$ in Formula (12) is the two-dimensional case of the weighted Matusita affinity

$${\rho }_{w_1,\dots ,w_n}(P_1,\dots ,P_n)=\int {\left(\prod _{i=1}^n{f}_i\right)}^{w_i}d\mu$$

(14)

for distributions $P_1,\dots ,P_n$ with $\mu$-densities $f_1,\dots ,f_n$, weights $w_1,\dots ,w_n>0$ satisfying ${\sum }_{i=1}^n{w}_i=1$, and $n\ge 2$; see [4] (p. 49) and [6]. ${\rho }_{w_1,\dots ,w_n}$, in turn, is a generalization of the Matusita affinity

$${\rho }_n(P_1,\dots ,P_n)=\int {\left(\prod _{i=1}^n{f}_i\right)}^{1/n}d\mu$$

introduced in [25,26]. Along the lines of the proof of Lemma 1, we find the representation

$${\rho }_{w_1,\dots ,w_n}({\mathit{\zeta }}^{\left(1\right)},\dots ,{\mathit{\zeta }}^{\left(n\right)})=exp\left\{\kappa \left(\sum _{i=1}^n{w}_i{\mathit{\zeta }}^{\left(i\right)}\right)-\sum _{i=1}^n{w}_i\kappa \left({\mathit{\zeta }}^{\left(i\right)}\right)\right\},{\mathit{\zeta }}^{\left(1\right)},\dots ,{\mathit{\zeta }}^{\left(n\right)}\in {\Xi }^{\ast },$$

for the EF $\mathcal{P}$ in Section 2; cf. [27]. In [4], the quantity in Formula (14) is termed Hellinger transform, and a representation within EFs is stated in Example 1.88.

${\rho }_{w_1,\dots ,w_n}$ can be used, for instance, as the basis of a homogeneity test (with null hypothesis $H_0:{\mathit{\zeta }}^{\left(1\right)}=\dots ={\mathit{\zeta }}^{\left(n\right)}$) or in discriminant problems.

For a representation of an extension of the Jeffrey distance to more than two distributions in an EF, the so-called Toussaint divergence, along with statistical applications, we refer to [8].

4. Entropy Measures

The literature on entropy measures, their applications, and their relations to divergence measures is broad. We focus on some selected results and state several simple representations of entropy measures within EFs.

Let the EF in Section 2 be given with $h\equiv 1$, which is the case, e.g., for the one-parameter EFs of geometric distributions and exponential distributions as well as for the two-parameter EF of univariate normal distributions. Formula (5) then yields that

$$\int {\left(f_{\mathit{\zeta }}^{\ast }\right)}^{r}d\mu =\int e^{r{\mathit{\zeta }}^t\mathit{T}-r\kappa \left(\mathit{\zeta }\right)}d\mu =e^{\kappa \left(r\mathit{\zeta }\right)-r\kappa \left(\mathit{\zeta }\right)}=J_r\left(\mathit{\zeta }\right),\mathrm{say},$$

for $r>0$ and $\mathit{\zeta }\in {\Xi }^{\ast }$ with $r\mathit{\zeta }\in {\Xi }^{\ast }$. Note that the latter condition is not that restrictive, since the natural parameter space of a regular EF is usually a cartesian product of the form $A_1\times \dots \times A_k$ with $A_i\in \{\mathbb{R},(-\infty ,0),(0,\infty )\}$ for $1\le i\le k$.

The Taneja entropy is then obtained as

$$\begin{array}{ccc}\hfill H_T\left(\mathit{\zeta }\right)& =& -2^{r-1}\int {\left(f_{\mathit{\zeta }}^{\ast }\right)}^{r}ln\left(f_{\mathit{\zeta }}^{\ast }\right)d\mu =-2^{r-1}\left({\mathit{\zeta }}^t\int \mathit{T}{e}^{r{\mathit{\zeta }}^t\mathit{T}-r\kappa \left(\mathit{\zeta }\right)}d\mu -\kappa \left(\mathit{\zeta }\right)J_r\left(\mathit{\zeta }\right)\right)\hfill \\ \hfill & =& -2^{r-1}{J}_r\left(\mathit{\zeta }\right)\left({\mathit{\zeta }}^t\int \mathit{T}{f}_{r\mathit{\zeta }}^{\ast }d\mu -\kappa \left(\mathit{\zeta }\right)\right)=-2^{r-1}{e}^{\kappa \left(r\mathit{\zeta }\right)-r\kappa \left(\mathit{\zeta }\right)}\left({\mathit{\zeta }}^t\pi \left(r\mathit{\zeta }\right)-\kappa \left(\mathit{\zeta }\right)\right)\hfill \end{array}$$

for $r>0$ and $\mathit{\zeta }\in {\Xi }^{\ast }$ with $r\mathit{\zeta }\in {\Xi }^{\ast }$, which includes the Shannon entropy

$$H_S\left(\mathit{\zeta }\right)=-\int f_{\mathit{\zeta }}^{\ast }ln\left(f_{\mathit{\zeta }}^{\ast }\right)d\mu =\kappa \left(\mathit{\zeta }\right)-{\mathit{\zeta }}^t\pi \left(\mathit{\zeta }\right),\mathit{\zeta }\in {\Xi }^{\ast },$$

by setting $r=1$; see [7,28].

Several other important entropy measures are functions of $J_r$ and therefore admit respective representations in terms of the cumulant function of the EF. Two examples are provided by the Rényi entropy and the Havrda–Charvát entropy (or Tsallis entropy), which are given by

$$\begin{array}{ccc}\hfill H_{R_r}\left(\mathit{\zeta }\right)& =& \frac{1}{1-r}ln\left(J_r\left(\mathit{\zeta }\right)\right)=\frac{\kappa \left(r\mathit{\zeta }\right)-r\kappa \left(\mathit{\zeta }\right)}{1-r},r>0,r\ne 1,\hfill \\ \hfill \mathrm{and}{H}_{H{C}_r}\left(\mathit{\zeta }\right)& =& \frac{1}{1-r}\left(J_r\left(\mathit{\zeta }\right)-1\right)=\frac{1}{1-r}\left(e^{\kappa \left(r\mathit{\zeta }\right)-r\kappa \left(\mathit{\zeta }\right)}-1\right),r>0,r\ne 1,\hfill \end{array}$$

for $\mathit{\zeta }\in {\Xi }^{\ast }$ with $r\mathit{\zeta }\in {\Xi }^{\ast }$; for the definitions, see, e.g., [1]. More generally, the Sharma–Mittal entropy is seen to be

$$\begin{array}{ccc}\hfill H_{S{M}_{r,s}}\left(\mathit{\zeta }\right)& =& \frac{1}{1-s}\left[{\left(J_r\left(\mathit{\zeta }\right)\right)}^{\frac{1-s}{1-r}}-1\right]\hfill \\ \hfill & =& \frac{1}{1-s}\left[{\left(e^{\kappa \left(r\mathit{\zeta }\right)-r\kappa \left(\mathit{\zeta }\right)}\right)}^{\frac{1-s}{1-r}}-1\right],r>0,r\ne 1,s\in \mathbb{R},s\ne 1,\hfill \end{array}$$

for $\mathit{\zeta }\in {\Xi }^{\ast }$ with $r\mathit{\zeta }\in {\Xi }^{\ast }$, which yields the representation for $H_S$ as $r=s\to 1$, for $H_{R_r}$ as $s\to 1$, and for $H_{H{C}_r}$ as $s\to r$; see [29].

If the assumption $h\equiv 1$ is not met, the calculus of the entropies becomes more involved. The Shannon entropy, for instance, is then given by

$$H_S\left(\mathit{\zeta }\right)=\kappa \left(\mathit{\zeta }\right)-{\mathit{\zeta }}^t\pi \left(\mathit{\zeta }\right)+E_{\mathit{\zeta }}[ln\left(h\right)],\mathit{\zeta }\in {\Xi }^{\ast },$$

where the additional additive term $E_{\mathit{\zeta }}[ln\left(h\right)]$, as it is the mean of $ln\left(h\right)$ under $P_{\mathit{\zeta }}^{\ast }$, will also depend on $\mathit{\zeta }$, in general; see, e.g., [17]. Since

$$\int {\left(f_{\mathit{\zeta }}^{\ast }\right)}^{r}d\mu =e^{\kappa \left(r\mathit{\zeta }\right)-r\kappa \left(\mathit{\zeta }\right)}{E}_{r\mathit{\zeta }}\left[h^{r-1}\right]$$

for $r>0$ and $\mathit{\zeta }\in {\Xi }^{\ast }$ with $r\mathit{\zeta }\in {\Xi }^{\ast }$ (cf. [29]), more complicated expressions result for other entropies and require to compute respective moments of h. Of course, we arrive at the same expressions as for the case $h\equiv 1$ if the entropies are introduced with respect to the dominating measure $\nu$, which is neither a counting nor a Lebesgue measure, in general; see Section 2. However, in contrast to divergence measures, entropies usually depend on the dominating measure, such that the resulting entropy values of the distributions will be different.

Representations of Rényi and Shannon entropies for various multivariate distributions including several EFs can be found in [30].

5. Application

As aforementioned, applications of divergence measures in statistical inference have been extensively discussed; see the references in the introduction. As an example, we make use of the representations of the symmetric divergences (distances) in Section 3 to construct confidence regions that are different from the standard rectangles for exponential parameters in a multi-sample situation.

Let $n_1,\dots ,n_k\in \mathbb{N}$ and $X_{ij}$, $1\le i\le k$, $1\le j\le n_i$, be independent random variables, where $X_{i1},\dots ,X_{i{n}_i}$ follow an exponential distribution with (unknown) mean $1/{\alpha }_i$ for $1\le i\le k$. The overall joint distribution $P_{\mathit{\alpha }}$, say, has the density function

$$f_{\mathit{\alpha }}\left(\mathit{x}\right)=e^{{\mathit{\alpha }}^t\mathit{T}\left(\mathit{x}\right)-\kappa \left(\mathit{\alpha }\right)},$$

(15)

with the k-dimensional statistic

$$\mathit{T}\left(\mathit{x}\right)=-{(x_{1•},\dots ,x_{k•})}^t,\mathrm{where}{x}_{i•}=\sum _{j=1}^{n_i}{x}_{ij},1\le i\le k,$$

for $\mathit{x}=(x_{11},\dots ,x_{1n_1},\dots ,x_{k1},\dots ,x_{k{n}_k})\in {(0,\infty )}^n$, the cumulant function

$$\kappa \left(\mathit{\alpha }\right)=-\sum _{i=1}^k{n}_{i}ln\left({\alpha }_i\right),\mathit{\alpha }={({\alpha }_1,\dots ,{\alpha }_k)}^t\in {(0,\infty )}^k,$$

and $n={\sum }_{i=1}^k{n}_i$. It is easily verified that $\mathcal{P}=\{P_{\mathit{\alpha }}:\mathit{\alpha }\in {(0,\infty )}^k\}$ forms a regular EF with minimal canonical representation (15). The corresponding mean value function is given by

$$\pi \left(\mathit{\alpha }\right)=-{\left(\frac{n_1}{{\alpha }_1},\dots ,\frac{n_k}{{\alpha }_k}\right)}^t,\mathit{\alpha }={({\alpha }_1,\dots ,{\alpha }_k)}^t\in {(0,\infty )}^k.$$

To construct confidence regions for $\mathit{\alpha }$ based on the Jeffrey distance $D_J$, the resistor-average distance $D_{RA}$, the distance $D_{GA}$, the Hellinger metric $D_H$, and the geometric Jensen–Shannon distance $D_{GJS}$, we first compute the KL-divergence $D_{KL}$ and the affinity $A_{1/2}$. Note that, by Remark 3, constructing a confidence region based on $D_H$ is equivalent to constructing a confidence region based on either $A_{1/2}$, $D_{R_{1/2}}$, or $D_{C{R}_{1/2}}$.

For $\mathit{\alpha }={({\alpha }_1,\dots ,{\alpha }_k)}^t,\mathit{\beta }={({\beta }_1,\dots ,{\beta }_k)}^t\in {(0,\infty )}^k$, we obtain from Theorem 1 that

$$\begin{array}{ccc}\hfill D_{KL}(\mathit{\alpha },\mathit{\beta })& =& -\sum _{i=1}^k{n}_{i}ln\left({\beta }_i\right)+\sum _{i=1}^k{n}_{i}ln\left({\alpha }_i\right)-\sum _{i=1}^k\frac{n_i}{{\alpha }_i}({\alpha }_i-{\beta }_i)\hfill \\ \hfill & =& \sum _{i=1}^k{n}_i\left[\frac{{\beta }_i}{{\alpha }_i}-ln\left(\frac{{\beta }_i}{{\alpha }_i}\right)-1\right],\hfill \end{array}$$

such that

$$\begin{array}{ccc}\hfill D_J(\mathit{\alpha },\mathit{\beta })& =& D_{KL}(\mathit{\alpha },\mathit{\beta })+D_{KL}(\mathit{\beta },\mathit{\alpha })\hfill \\ \hfill & =& \sum _{i=1}^k{n}_i\left(\frac{{\alpha }_i}{{\beta }_i}+\frac{{\beta }_i}{{\alpha }_i}-2\right).\hfill \end{array}$$

$D_{RA}$ and $D_{GA}$ are then computed by inserting for $D_{KL}$ and $D_J$ in Formulas (10) and (11). Applying Lemma 1 yields

$$\begin{array}{ccc}\hfill A_{1/2}(\mathit{\alpha },\mathit{\beta })& =& \left[\prod _{i=1}^k{\left(\frac{{\alpha }_i+{\beta }_i}{2}\right)}^{-n_i}\right]\left(\prod _{i=1}^k{\alpha }_i^{n_i/2}\right)\left(\prod _{i=1}^k{\beta }_i^{n_i/2}\right)\hfill \\ \hfill & =& \prod _{i=1}^k{\left[\frac{1}{2}\left(\sqrt{\frac{{\alpha }_i}{{\beta }_i}}+\sqrt{\frac{{\beta }_i}{{\alpha }_i}}\right)\right]}^{-n_i},\hfill \end{array}$$

which gives $D_H(\mathit{\alpha },\mathit{\beta })={[2-2A_{1/2}(\mathit{\alpha },\mathit{\beta })]}^{1/2}$ by inserting, and, by using Formula (13), also leads to

$$\begin{array}{ccc}\hfill D_{GJS}(\mathit{\alpha },\mathit{\beta })& =& ln\left(A_{1/2}(\mathit{\alpha },\mathit{\beta })\right)+\frac{D_J(\mathit{\alpha },\mathit{\beta })}{4}\hfill \\ \hfill & =& \frac{1}{4}\sum _{i=1}^k{n}_i\left[\frac{{\alpha }_i}{{\beta }_i}+\frac{{\beta }_i}{{\alpha }_i}-4ln\left(\frac{1}{2}\left(\sqrt{\frac{{\alpha }_i}{{\beta }_i}}+\sqrt{\frac{{\beta }_i}{{\alpha }_i}}\right)\right)-2\right].\hfill \end{array}$$

The MLE $\widehat{\mathit{\alpha }}={({\widehat{\alpha }}_1,\dots ,{\widehat{\alpha }}_k)}^t$ of $\mathit{\alpha }$ based on $\mathit{X}=(X_{11},\dots ,X_{1n_1},\dots ,X_{k1},\dots ,X_{k{n}_k})$, is given by

$$\widehat{\mathit{\alpha }}={\left(\frac{n_1}{X_{1•}},\dots ,\frac{n_k}{X_{k•}}\right)}^t,$$

where ${\widehat{\alpha }}_1,\dots ,{\widehat{\alpha }}_k$ are independent. By inserting the random distances $D_J(\widehat{\mathit{\alpha }},\mathit{\alpha })$, $D_{RA}(\widehat{\mathit{\alpha }},\mathit{\alpha })$, $D_{GA}(\widehat{\mathit{\alpha }},\mathit{\alpha })$, $D_H(\widehat{\mathit{\alpha }},\mathit{\alpha })$, and $D_{GJS}(\widehat{\mathit{\alpha }},\mathit{\alpha })$ turn out to depend on $\mathit{X}$ only through the vector ${({\alpha }_1/{\widehat{\alpha }}_1,\dots ,{\alpha }_k/{\widehat{\alpha }}_k)}^t$ of component-wise ratios, where ${\alpha }_i/{\widehat{\alpha }}_i$ has a gamma distribution with shape parameter $n_i$, scale parameter $1/n_i$, and mean 1 for $1\le i\le k$. Since these ratios are moreover independent, the above random distances form pivot statistics with distributions free of $\mathit{\alpha }$.

Now, confidence regions for $\mathit{\alpha }$ with confidence level $p\in (0,1)$ are given by

$$\begin{array}{ccc}\hfill C_{•}& =& \left\{\mathit{\alpha }\in {(0,\infty )}^k:D_{•}(\widehat{\mathit{\alpha }},\mathit{\alpha })\le c_{•}\left(p\right)\right\},\hfill \end{array}$$

where $c_{•}\left(p\right)$ denotes the p-quantile of $D_{•}(\widehat{\mathit{\alpha }},\mathit{\alpha })$ for $•=J,RA,GA,H,GJS$, numerical values of which can readily be obtained via Monte Carlo simulation by sampling from gamma distributions.

Confidence regions for the mean vector $\mathit{m}={(1/{\alpha }_1,\dots ,1/{\alpha }_k)}^t$ with confidence level $p\in (0,1)$ are then given by

$$\begin{array}{ccc}\hfill {\tilde{C}}_{•}& =& \left\{{\left(\frac{1}{{\alpha }_1},\dots ,\frac{1}{{\alpha }_k}\right)}^t\in {(0,\infty )}^k:{({\alpha }_1,\dots ,{\alpha }_k)}^t\in C_{•}\right\}\hfill \end{array}$$

for $•=J,RA,GA,H,GJS$.

In Figure 1 and Figure 2, realizations of ${\tilde{C}}_J$, ${\tilde{C}}_{RA}$, ${\tilde{C}}_{GA}$, ${\tilde{C}}_H$, and ${\tilde{C}}_{GJS}$ are depicted for the two-sample case ($k=2$) and some sample sizes $n_1,n_2$ and values of $\widehat{\mathit{\alpha }}={({\widehat{\alpha }}_1,{\widehat{\alpha }}_2)}^t$, where the confidence level is chosen as $p=90\%$. Additionally, realizations of the standard confidence region

$$R=\left[\frac{2n_1}{{\widehat{\alpha }}_1{\chi }_{1-q}^2\left(2n_1\right)},\frac{2n_1}{{\widehat{\alpha }}_1{\chi }_q^2\left(2n_1\right)}\right]\times \left[\frac{2n_2}{{\widehat{\alpha }}_2{\chi }_{1-q}^2\left(2n_2\right)},\frac{2n_2}{{\widehat{\alpha }}_2{\chi }_q^2\left(2n_2\right)}\right]$$

with a confidence level of 90% for $\mathit{m}={(m_1,m_2)}^t$ are shown in the figures, where $q=(1-\sqrt{0.9})/2$ and ${\chi }_{\gamma }^2\left(v\right)$ denotes the $\gamma$-quantile of the chi-square distribution with v degrees of freedom.

Figure 1. Illustration of the confidence regions ${\tilde{C}}_J$ (solid light grey line), ${\tilde{C}}_{RA}$ (solid dark grey line), ${\tilde{C}}_{GA}$ (solid black line), ${\tilde{C}}_H$ (dashed black line), ${\tilde{C}}_{GJS}$ (dotted black line), and R (rectangle) for the mean vector $\mathit{m}={(m_1,m_2)}^t$ with level 90% and sample sizes $n_1,n_2$ based on a realization $\widehat{\mathit{\alpha }}={(0.0045,0.0055)}^t$, respectively $\widehat{\mathit{m}}={(222.2,181.8)}^t$ of the MLE (circle).

Figure 2. Illustration of the confidence regions ${\tilde{C}}_J$ (solid light grey line), ${\tilde{C}}_{RA}$ (solid dark grey line), ${\tilde{C}}_{GA}$ (solid black line), ${\tilde{C}}_H$ (dashed black line), ${\tilde{C}}_{GJS}$ (dotted black line), and R (rectangle) for the mean vector $\mathit{m}={(m_1,m_2)}^t$ with level 90% and sample sizes $n_1,n_2$ based on a realization $\widehat{\mathit{\alpha }}={(0.003,0.007)}^t$, respectively $\widehat{\mathit{m}}={(333.3,142.9)}^t$ of the MLE (circle).

It is found that over the sample sizes and realizations of $\widehat{\mathit{\alpha }}$ considered, the confidence regions ${\tilde{C}}_J$, ${\tilde{C}}_{RA}$, ${\tilde{C}}_{GA}$, ${\tilde{C}}_H$, and ${\tilde{C}}_{GJS}$ are similarly shaped but do not coincide as the plots for different sample sizes show. In terms of (observed) area, all divergence-based confidence regions perform considerably better than the standard rectangle. This finding, however, depends on the parameter of interest, which here is the vector of exponential means; for the divergence-based confidence regions and the standard rectangle for $\mathit{\alpha }$ itself, the contrary assertion is true. Although the divergence-based confidence regions have a smaller area than the standard rectangle, this is not at the cost of large projection lengths with respect to the $m_1$- and $m_2$-axes, which serve as further characteristics for comparing confidence regions. Monte Carlo simulations may moreover be applied to compute the expected area and projection lengths as well as the coverage probabilities of false parameters for a more rigorous comparison of the performance of the confidence regions, which is beyond the scope of this article.