*1.1. Exponential Families*

Let (X , Σ) be a measurable space, and consider a regular minimal exponential family [1] E of probability measures *P<sup>θ</sup>* all dominated by a base measure *μ* (*P<sup>θ</sup> μ*):

$$\mathcal{E} = \{ P\_{\theta} \; : \; \theta \in \Theta \}. \tag{1}$$

The Radon–Nikodym derivatives or densities of the probability measures *P<sup>θ</sup>* with respect to *μ* can be written canonically as

$$p\_{\theta}(\mathbf{x}) = \frac{\mathbf{d}P\_{\theta}}{\mathbf{d}\mu}(\mathbf{x}) = \exp\left(\theta^{\top}t(\mathbf{x}) - F(\theta) + k(\mathbf{x})\right),\tag{2}$$

where *θ* denotes the natural parameter, *t*(*x*) the sufficient statistic [1–4], and *F*(*θ*) the lognormalizer [1] (or cumulant function). The optional auxiliary term *k*(*x*) allows us to change the base measure *μ* into the measure *ν* such that <sup>d</sup>*<sup>ν</sup>* <sup>d</sup>*<sup>μ</sup>* (*x*) = *<sup>e</sup>k*(*x*). The order *<sup>D</sup>* of the family is the dimension of the natural parameter space Θ:

$$\Theta = \left\{ \theta \in \mathbb{R}^D \, : \, \int\_X \exp\left(\theta^\top t(\mathbf{x}) + k(\mathbf{x})\right) \mathrm{d}\mu(\mathbf{x}) < \infty \right\},\tag{3}$$

where R denotes the set of reals. The sufficient statistic *t*(*x*)=(*t*1(*x*), ... , *tD*(*x*)) is a vector of *D* functions. The sufficient statistic *t*(*x*) is said to be minimal when the *D* + 1 functions 1, *t*1(*x*), ..., *tD*(*x*) are linearly independent [1]. The sufficient statistics *t*(*x*) are such that the probability Pr[*X*|*θ*] = Pr[*X*|*t*(*X*)]. That is, all information necessary for the statistical inference of parameter *θ* is contained in *t*(*X*). Exponential families are characterized as

**Citation:** Nielsen, F. Statistical Divergences between Densities of Truncated Exponential Families with Nested Supports: Duo Bregman and Duo Jensen Divergences. *Entropy* **2022**, *24*, 421. https://doi.org/ 10.3390/e24030421

Academic Editors: Karagrigoriou Alexandros and Makrides Andreas

Received: 2 March 2022 Accepted: 16 March 2022 Published: 17 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

families of parametric distributions with finite-dimensional sufficient statistics [1]. Exponential families {*pλ*} include among others the exponential, normal, gamma/beta, inverse gamma, inverse Gaussian, and Wishart distributions once a reparameterization *θ* = *θ*(*λ*) of the parametric distributions {*pλ*} is performed to reveal their natural parameters [1].

When the sufficient statistic *t*(*x*) is *x*, these exponential families [1] are called natural exponential families or tilted exponential families [5] in the literature. Indeed, the distributions *P<sup>θ</sup>* of the exponential family E can be interpreted as distributions obtained by tilting the base measure *μ* [6]. In this paper, we consider either discrete exponential families like the family of Poisson distributions (univariate distributions of order *D* = 1 with respect to the counting measure) or continuous exponential families like the family of normal distributions (univariate distributions of order *D* = 2 with respect to the Lebesgue measure). The Radon–Nikodym derivative of a discrete exponential family is a probability mass function (pmf), and the Radon–Nikodym derivative of a continuous exponential family is a probability density function (pdf). The support of a pmf *p*(*x*) is supp(*p*) = {*<sup>x</sup>* <sup>∈</sup> <sup>Z</sup> : *<sup>p</sup>*(*x*) <sup>&</sup>gt; <sup>0</sup>} (where <sup>Z</sup> denotes the set of integers) and the support of <sup>a</sup> *<sup>d</sup>*-variate pdf *<sup>p</sup>*(*x*) is supp(*p*) = {*<sup>x</sup>* <sup>∈</sup> <sup>R</sup>*<sup>d</sup>* : *<sup>p</sup>*(*x*) <sup>&</sup>gt; <sup>0</sup>}. The Poisson distributions have support <sup>N</sup> ∪ {0} where <sup>N</sup> denotes the set of natural numbers {1, 2, ... , }. Densities of an exponential family all have coinciding support [1].
