*1.3. Kullback–Leibler Divergence Between Exponential Family Distributions*

For two *σ*-finite probability measures *P* and *Q* on (X , Σ) such that *P* is dominated by *Q* (*P* -*Q*), the Kullback–Leibler divergence between *P* and *Q* is defined by

$$D\_{\rm KL}[P:Q] = \int\_{\mathcal{X}} \log \frac{\mathbf{d}P}{\mathbf{d}Q} \, \mathrm{d}P = E\_P \left[ \log \frac{\mathbf{d}P}{\mathbf{d}Q} \right],\tag{7}$$

where *EP*[*X*] denotes the expectation of a random variable *X* ∼ *P* [10]. When *P* - *Q*, we set *D*KL[*P* : *Q*]=+∞. Gibbs' inequality [11] *D*KL[*P* : *Q*] ≥ 0 shows that the Kullback–Leibler divergence (KLD for short) is always non-negative. The proof of Gibbs' inequality relies on Jensen's inequality and holds for the wide class of *f*-divergences [12] induced by convex generators *f*(*u*):

$$I\_f[P:\mathbb{Q}] = \int\_{\mathcal{X}} f\left(\frac{\mathrm{d}\mathcal{Q}}{\mathrm{d}P}\right) \mathrm{d}P \ge f\left(\int\_{\mathcal{X}} \frac{\mathrm{d}\mathcal{Q}}{\mathrm{d}P} \mathrm{d}P\right) \ge f(1). \tag{8}$$

The KLD is an *f*-divergence obtained for the convex generator *f*(*u*) = − log *u*.
