Average Entropy of Gaussian Mixtures

Abstract

We calculate the average differential entropy of a q-component Gaussian mixture in ${\mathbb{R}}^n$. For simplicity, all components have covariance matrix ${\sigma }^2\mathbf{1}$, while the means ${\left\{{\mathbf{W}}_i\right\}}_{i=1}^q$ are i.i.d. Gaussian vectors with zero mean and covariance $s^2\mathbf{1}$. We obtain a series expansion in $\mu =s^2/{\sigma }^2$ for the average differential entropy up to order $\mathcal{O}\left({\mu }^2\right)$, and we provide a recipe to calculate higher-order terms. Our result provides an analytic approximation with a quantifiable order of magnitude for the error, which is not achieved in previous literature.

1. Introduction

1.1. Gaussian Mixtures

A Gaussian mixture probability density on ${\mathbb{R}}^n$ is a function of the following form:

$$f\left(x\right)=\sum _{i=1}^q{p}_i{g}_{w_i,K_i}\left(x\right),$$

(1)

where q is some integer, $p_i$ are probabilities, and $g_{w,K}$ stands for the Gaussian distribution $g_{w,K}\left(x\right)={\left(2\pi \right)}^{-{\displaystyle \frac{n}{2}}}{(detK)}^{-{\displaystyle \frac{1}{2}}}exp[-{\displaystyle \frac{1}{2}}{(x-w)}^T{K}^{-1}(x-w)]$, with $x,w\in {\mathbb{R}}^n$. In words, the density $f\left(x\right)$ is built as a weighted average of q different n-dimensional Gaussian distributions. A mixture like this occurs when one stochastic process, with probability mass function $p_1,\dots ,p_q$, determines which distribution is chosen for x. Gaussian mixtures are widely used in various areas for their simplicity, as well as their wide range of applicability by virtue of being global (smooth) function approximators. Most recently, in astrophysics, Gaussian mixture models were used to study the kinematics of dark matter [1], as well as predicting the spectrum of quasars [2]. Furthermore, Gaussian mixtures are also widely used in the intersection between statistical physics and machine learning, i.e., diffusion models [3,4]. Diffusion models became popular for their wide range of applications, including generative tasks and image restoration, for example, see [5,6] for Gaussian mixture diffusion models. In wireless communications, Gaussian mixtures were recently utilized to estimate the channel trajectories of moving mobile terminals [7], while in cybernetic security, they played a role in defending against the Byzantine attack [8], and earlier for authentication in wireless networks [9]. In bioinformatics, they are used as an alternative to clustering algorithms for analyzing gene expression data [10,11]. They also find application in thermodynamics, e.g., a computationally efficient Monte Carlo method was devised in [12] to study the properties of nonadiabatic systems. In [13], many of the above-mentioned disciplines are combined, where the authors evaluate probabilities in deep generative models.

1.2. Related Work

Analytically computing or estimating the differential entropy of a Gaussian mixture is a difficult problem. The differential entropy of a continuous variable $X\in \mathcal{X}$, X∼f is defined as $h\left(X\right)=-{\int }_{\mathcal{X}}f\left(x\right)lnf\left(x\right)\mathrm{d}x$ [14]. In the special case of a single Gaussian component ($q=1$), the integral can be computed analytically, yielding ${\displaystyle \frac{n}{2}}ln2\pi e+{\displaystyle \frac{1}{2}}lndetK$. In [15], the authors study the case of $q=2$, $n=1$, and $w_1=-w_2$, where a numerical approximation for this case is obtained. It is a known fact that given the second moment of a random variable, the distribution with maximum entropy is the Gaussian distribution [14]. That is, we can easily obtain a loose upper bound for the differential entropy of a Gaussian mixture given by [16]:

$$h\left(X\right)\le {\displaystyle \frac{n}{2}}ln2\pi e+{\displaystyle \frac{1}{2}}ln\det (\Sigma ),$$

(2)

where $\Sigma$ is given by

$$\Sigma =\sum _{i=1}^q{p}_i(w_i{w}_i^{\mathrm{T}}+K_i)-\left(\sum _{i=1}^q{p}_i{w}_i\right)\left(\sum _{j=1}^q{p}_j{w}_j^{\mathrm{T}}\right).$$

(3)

A sequence of maximum entropy upper bounds for the differential entropy was obtained in [17], most notably the Laplacian upper bound, which is tighter than the Gaussian upper abound in some cases. However, it has no closed-form expression. Another way of approximating the differential entropy is by replacing the density inside the logarithm by a single Gaussian $\overline{f}$ with covariance and mean identical to the mixture. This leads to an exact expression in terms of the relative entropy,

$$h\left(X\right)={\displaystyle \frac{n}{2}}ln2\pi e+{\displaystyle \frac{1}{2}}ln\det (\Sigma )-D\left(f\right||\overline{f}).$$

(4)

One can then find approximations to the relative entropy, as in [18,19]. Although this method is efficient, it also does not a have closed-form expression and only provides an upper bound to $h\left(X\right)$. Other Monte Carlo sampling methods give guaranteed convergence. However, they become computationally demanding as the number of samples grows. An approximation for $h\left(X\right)$ was obtained in [20] by performing a Taylor series expansion of the logarithm term. In order to avoid the need to include higher-order terms, a splitting of Gaussian components belonging to the density outside the logarithm is applied. The idea is to split components with high variance and replace them with Gaussian mixtures with components of lower variance. This is because the higher the variance of the components the higher the number of terms needed in the Taylor expansion to achieve a certain level of accuracy. Of course, this process is not exact and produces an error depending on how many Gaussian components are included in the split. The splitting method is not directly comparable with our work, as we do not make a distinction between variances of the different components in the mixture; instead, our expansion parameter is how spread the components of the mixture are relative to their shared variance value. However, one can obtain a basic upper bound on $h\left(X\right)$, as shown in [20]:

$$h\left(X\right)\le \sum _{i=1}^q{p}_i\left(-ln{p}_i+{\displaystyle \frac{1}{2}}ln{\left(2\pi e\right)}^n\det \left(K_i\right)\right).$$

(5)

As mentioned in [20], the upper bound (5) is significantly tighter than the Gaussian bound (2); it is exact in the case of a single Gaussian; and it is arbitrarily close to the real value of $h\left(X\right)$ when the support shared between the components of the mixture is negligible. A refinement of this bound was also introduced by means of merging different clusters in the mixture. However, we are concerned with the average over all possible mixtures, and therefore, this family of bounds is less comparable with our result. In [21], the differential entropy is estimated for mixture distributions using pair-wise distances between the components of the mixture. It is shown that the estimator is a lower bound when the Chernoff $\alpha$-divergence is used, and an upper bound when the Kullback–Leibler divergence is used. From the results of [21], it is difficult to obtain tight bounds on the average entropy, which is the focus of our paper.

1.3. Contributions

We develop an analytical estimation method for the average Gaussian mixture entropy problem in the special case of equal weights ${\displaystyle \frac{1}{q}}$ and equal covariance matrix ${\sigma }^2\mathbf{1}$. Our method postulates that the displacements $w_i$ themselves are stochastic, i.i.d. with Gaussian distribution $g_{0,s^2\mathbf{1}}$. We compute $h\left(X\right)$ averaged over the displacements $w_i$, resulting in the conditional entropy $h\left(X\right|W)$. We mention that the quantity $h\left(X\right|W)$, for exactly this setting with equal weights, spherical covariance, and Gaussian displacements, plays a role in the detectability of digital watermarks (see Section 2.2).

We work in the regime $s<\sigma$. Our method uses the fraction $\mu ={\displaystyle \frac{s^2}{{\sigma }^2}}$ as the small parameter for a power expansion. We show results up to and including order $\mathcal{O}\left({\mu }^2\right)$. Our result is novel since most available estimators of $h\left(X\right)$ in the literature have no closed-form expressions, and usually rely on upper bounds rather than an explicit calculation of $h\left(X\right)$. In [20], although a Taylor series approximation is employed, its accuracy is dependent on the splitting mentioned in Section 1.2. Furthermore, it is difficult to estimate the order of magnitude for the error in a general expansion around the means. In contrast, our set-up does not need the splitting method, and for our result, it is possible to quantify the order of magnitude for the error since our power series is in $\mu$.

2. Deriving the Series Expansion

2.1. Notation

Boldface lowercase letters denote n-dimensional vectors (e.g., $\mathbf{a}$); the inner product between two n-dimensional vectors $\mathbf{a}$ and $\mathbf{b}$ is denoted by $\mathbf{a}·\mathbf{b}$; A hat denotes an array in q dimensions, i.e., indexed with the number of the Gaussian components; for lowercase letters, $\left(\widehat{·}\right)$ denotes a q-dimensional vector (e.g., $\widehat{a}$); for uppercase letters, a $q\times q$ matrix (e.g., $\widehat{A}$). ${\widehat{a}}^{\mathrm{T}}\widehat{b}$ denotes the q-dimensional inner product. ⊗ denotes the Kronecker product.

2.2. Starting Point

Consider a random variable $\mathbf{X}\in {\mathbb{R}}^n$ whose probability density function (pdf) is a Gaussian mixture with equal weights. The Gaussian pdfs all have covariance matrix ${\sigma }^2\mathbf{1}$, and they are centered on points ${\mathbf{w}}_1,\dots ,{\mathbf{w}}_q\in {\mathbb{R}}^n$. We write $\widehat{\mathbf{w}}=({\mathbf{w}}_1,\dots ,{\mathbf{w}}_q)$.

$$f_{\mathbf{X}|\widehat{\mathbf{W}}}\left(\mathbf{x}\right|\widehat{\mathbf{w}})={\displaystyle \frac{1}{q}}\sum _{j=1}^q{g}_{{\mathbf{w}}_j,{\sigma }^2}\left(\mathbf{x}\right)={\left(2\pi {\sigma }^2\right)}^{-{\displaystyle \frac{n}{2}}}{\displaystyle \frac{1}{q}}\sum _{j=1}^{q}exp\left\{-{\displaystyle \frac{{(\mathbf{x}-{\mathbf{w}}_j)}^2}{2{\sigma }^2}}\right\}.$$

(6)

We note that a simplification is possible when $n>q$, i.e., the dimension of the space is higher than the number of components in the mixture. For all configurations $\widehat{\mathbf{w}}$, it is possible to find a rotation such that after rotation ${\mathbf{w}}_j={\sum }_{\alpha =1}^q{w}_{j\alpha }{\mathbf{e}}_{\alpha }$, where the ${\mathbf{e}}_{\alpha }$ are the basis vectors in ${\mathbb{R}}^n$. In other words, for all ${\mathbf{w}}_j$, the vector components in the dimensions beyond q vanish. The density (6) simplifies to

$$f_{\mathbf{X}|\widehat{\mathbf{W}}}\left(\mathbf{x}\right|\widehat{\mathbf{w}})\to exp\left\{-{\displaystyle \frac{1}{2{\sigma }^2}}\sum _{\alpha =q+1}^n{x}_{\alpha }^2\right\}{\left(2\pi {\sigma }^2\right)}^{-{\displaystyle \frac{n}{2}}}{\displaystyle \frac{1}{q}}\sum _{j=1}^{q}exp\left\{-{\displaystyle \frac{{\sum }_{\alpha =1}^q{(x_{\alpha }-w_{j\alpha })}^2}{2{\sigma }^2}}\right\}.$$

(7)

The first exponent is trivially dealt with when the $ln{f}_{\mathbf{X}|\widehat{\mathbf{W}}}\left(\mathbf{x}\right|\widehat{\mathbf{w}})$ is integrated, yielding a constant contribution ${\displaystyle \frac{n-q}{2}}$ to the entropy. Hence, we only focus on the case $n\le q$ in our calculations. In general, analytically approximating the differential entropy $h\left(\mathbf{X}\right|\widehat{\mathbf{W}}=\widehat{\mathbf{w}})$ for a given $\widehat{\mathbf{w}}$ is a difficult problem. In this paper, we study a problem that is slightly easier: finding analytic approximations for $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})={\mathbb{E}}_{\widehat{\mathbf{w}}}h\left(\mathbf{X}\right|\widehat{\mathbf{W}}=\widehat{\mathbf{w}})$ in the case where the offsets ${\mathbf{w}}_j$ are i.i.d. Gaussian-distributed with zero mean and covariance matrix $s^2\mathbf{1}$.

$$f_{\widehat{\mathbf{W}}}\left(\widehat{\mathbf{w}}\right)=\prod _{j=1}^q{g}_{0,s^2\mathbf{1}}\left({\mathbf{w}}_j\right)={\left(2\pi s^2\right)}^{-{\displaystyle \frac{nq}{2}}}exp\left\{-\sum _{j=1}^q{\displaystyle \frac{{\mathbf{w}}_j^2}{2s^2}}\right\}.$$

(8)

We note that the quantity $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$, under precisely these conditions, occurs in the field of spread-spectrum watermarking [22,23]. In this setting, the $\widehat{\mathbf{W}}$ is a table of q random watermarking sequences. A row index $K\in \{1,\dots ,q\}$ is chosen at random, and the K’th row of $\widehat{\mathbf{W}}$ is additively inserted into some data $\mathbf{D}$, which are modeled as Gaussian, and then attackers apply noise in order to wipe out the watermark. The $\mathbf{X}$ plays the role of the watermarked data after this attack. A watermark detector tries to determine the index K based on X and $\widehat{\mathbf{W}}$ (and optionally the unwatermarked original data $\mathbf{D}$). In the analysis of the detection efficiency, an important figure of merit is the mutual information $I(K;\mathbf{X}\widehat{\mathbf{W}})$ or $I(K;\mathbf{D}\mathbf{X}\widehat{\mathbf{W}})$, both of which after some rewriting involve precisely the conditional entropy $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$.

We take $s^2<{\sigma }^2$ and develop a power series in the parameter $\mu =s^2/{\sigma }^2$. In the following sections, we obtain our results by following these steps:

• We first obtain a more compact form of $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ in Corollary 1 via a change in variables in order to write $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ in terms of the more familiar expectation with regard to a multivariate Gaussian density.
• We next perform another change or variables to obtain a diagonal form in the expectation with regard to the multivariate Gaussian. This is important since our new variables will be independent, and many expressions will be simplified if our variables are not correlated.
• We evaluate the leading terms to $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$, as shown in Theorem 2, where we obtain an expression that separates the leading contributions to $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ and the quantity S to be defined. The reason this is important is that the leading contributions contain terms that will make the Taylor series diverge, and therefore, we evaluate them analytically before performing the Taylor series expansion. We also include other terms that do not cause any problems in the limit of small $\mu$ for convenience. The remaining expression S is then safe to expand for small $\mu$.
• We perform a third change in variables to simplify S, and we obtain a Taylor series for $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ up to order $\mathcal{O}\left({\mu }^2\right)$ in Theorem 3 via a brute-force approach. The change in variables is necessary for making the analytical expression tractable.
• Finally, we provide a determinant-based approach to evaluate the power series for S, and we obtain a result for $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ up to order $\mathcal{O}\left(\mu \right)$.

2.3. First Change in Variables

We observe that inside the logarithm, the variables ${\mathbf{w}}_j$ occur only in the combination ${\mathbf{w}}_j-\mathbf{x}$. This allows us to introduce shifted variables ${\mathbf{z}}_j={\mathbf{w}}_j-\mathbf{x}$ and then analytically carry out the integration over $\mathbf{x}$.

Lemma 1. 

The differential entropy $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ is given by

$$\begin{array}{c}\hfill h\left(\mathbf{X}\right|\widehat{\mathbf{W}})={\displaystyle \frac{n}{2}}ln2\pi {\sigma }^2-{\left({\displaystyle \frac{\mu }{q}}\right)}^{{\displaystyle \frac{n}{2}}}{\left(2\pi \right)}^{-{\displaystyle \frac{qn}{2}}}{\displaystyle \frac{1}{q}}\sum _{j=1}^q\int \mathrm{d}{\mathbf{z}}_1\cdots \mathrm{d}{\mathbf{z}}_q{e}^{-{\displaystyle \frac{1}{2}}{\sum }_a{\mathbf{z}}_a^2+{\displaystyle \frac{1}{2q}}{\left({\sum }_a{\mathbf{z}}_a\right)}^2-{\displaystyle \frac{\mu }{2}}{\mathbf{z}}_j^2}ln{\displaystyle \frac{1}{q}}\sum _{k=1}^q{e}^{-{\displaystyle \frac{\mu }{2}}{\mathbf{z}}_k^2}.\end{array}$$

(9)

Proof. 

See Appendix A. □

Next, we can get rid of the ${\sum }_j$ summation, using permutation symmetry.

Lemma 2. 

Let $F({\mathbf{z}}_1,\dots ,{\mathbf{z}}_q)$ be any permutation invariant function; then, we have

$$\begin{array}{cc}\hfill & \int exp\left\{-{\displaystyle \frac{1}{2}}\sum _j{\mathbf{z}}_j^2+{\displaystyle \frac{1}{2q}}\sum _{ij}{\mathbf{z}}_i·{\mathbf{z}}_j\right\}{\displaystyle \frac{1}{q}}\sum _{k}exp\left\{-{\displaystyle \frac{\mu }{2}}{\mathbf{z}}_k^2\right\}F({\mathbf{z}}_1,\dots ,{\mathbf{z}}_q)\mathrm{d}{\mathbf{z}}_1\dots \mathrm{d}{\mathbf{z}}_q\hfill \\ \hfill & =\int exp\left\{-{\displaystyle \frac{1}{2}}{\widehat{\mathbf{z}}}^{\mathrm{T}}{\widehat{M}}^{\left(q\right)}\widehat{\mathbf{z}}\right\}F({\mathbf{z}}_1,\dots ,{\mathbf{z}}_q)\mathrm{d}{\mathbf{z}}_1\dots \mathrm{d}{\mathbf{z}}_q,\hfill \end{array}$$

(10)

where the matrix elements ${\left\{M_{ij}^{\left(k\right)}\right\}}_{k=1}^q$ are given by

$$M_{ij}^{\left(k\right)}={\delta }_{ij}-{\displaystyle \frac{1}{q}}+\mu {\delta }_{ik}{\delta }_{jk}.$$

(11)

Proof. 

See Appendix B. □

We now rewrite the differential entropy more compactly in the following corollary.

Corollary 1. 

Let the function F be given by

$$F({\mathbf{z}}_1,\dots ,{\mathbf{z}}_q)=ln{q}^{-1}\sum _{l=1}^{q}exp\left\{-{\displaystyle \frac{\mu }{2}}{\mathbf{z}}_l·{\mathbf{z}}_l\right\},$$

(12)

then, the differential entropy $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ can be written as

$$h\left(\mathbf{X}\right|\widehat{\mathbf{W}})={\displaystyle \frac{n}{2}}ln2\pi {\sigma }^2-{\left({\displaystyle \frac{\mu }{q}}\right)}^{{\displaystyle \frac{n}{2}}}{\left(2\pi \right)}^{-{\displaystyle \frac{qn}{2}}}\int exp\left\{-{\displaystyle \frac{1}{2}}{\widehat{\mathbf{z}}}^{\mathrm{T}}{\widehat{M}}^{\left(q\right)}\widehat{\mathbf{z}}\right\}F({\mathbf{z}}_1,\dots ,{\mathbf{z}}_q)\mathrm{d}{\mathbf{z}}_1\dots \mathrm{d}{\mathbf{z}}_q.$$

(13)

Proof. 

It follows directly from Lemmas 1 and 2, where we note that Equation (12) is permutation invariant. □

2.4. Diagonalization of ${\widehat{M}}^{\left(q\right)}$ and Second Change in Variables

We would like to obtain an expression for the expectation term in Equation (13); however, we first need to switch to a diagonal form that removes correlation between the integration variables, i.e., we would like to find the eigenvectors and eigenvalues of the matrix ${\widehat{M}}^{\left(q\right)}$. We first notice that adding the identity matrix, while it shifts the eigenvalues by 1, does not change the eigenvectors of a matrix. Hence, we diagonalize the matrix ${\widehat{C}}^{\left(q\right)}$, as shown in Theorem 1, and our result follows directly in Corollary 2.

Theorem 1. 

Let $\widehat{J}$ be the all-one matrix. Let ${\widehat{e}}^{\left(k\right)}$ denote the k-th standard basis vector. Then, the matrix ${\widehat{C}}^{\left(q\right)}$ given by

$${\widehat{C}}^{\left(q\right)}=-q^{-1}\widehat{J}+\mu \mathrm{diag}\left({\widehat{e}}^{\left(q\right)}\right)=-{\displaystyle \frac{1}{q}}\left(\begin{array}{cccc}1& 1& \dots & 1\\ 1& 1& \dots & 1\\ ⋮& ⋮& \ddots & ⋮\\ 1& 1& \dots & 1-q\mu \end{array}\right),$$

(14)

has an orthogonal matrix ${\widehat{\Lambda }}^{\left(q\right)}$ of eigenvectors given by

$${\widehat{\Lambda }}^{\left(q\right)}=({\widehat{\alpha }}_1,{\widehat{\alpha }}_2,\dots ,{\widehat{\alpha }}_{q-2},{\widehat{\beta }}_1,{\widehat{\beta }}_2),$$

(15)

where ${\left\{{\widehat{\alpha }}_i\right\}}_{i=1}^{q-2}$ are eigenvectors with eigenvalue 0, whereas ${\widehat{\beta }}_1$ and ${\widehat{\beta }}_2$ have eigenvalues ${\lambda }_1$ and ${\lambda }_2$, respectively, and read

$${\lambda }_j={\displaystyle \frac{1}{2}}\left((\mu -1)+{(-1)}^{j+1}\sqrt{{\mu }^2-(4q^{-1}-2)\mu +1}\right),j\in \{1,2\}.$$

(16)

Furthermore, the elements of the eigenvectors are given by

$$\begin{array}{cc}\hfill & {\alpha }_{i,k}={\displaystyle \frac{1}{\sqrt{i(i+1)}}}\left\{\begin{array}{cc}0\hfill & ,k>i+1\hfill \\ 1\hfill & ,k<i+1\hfill \\ -i\hfill & ,k=i+1\hfill \end{array}\right.,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & {\beta }_{i,k}={\displaystyle \frac{1}{\sqrt{q-1+{[1-q({\lambda }_i+1)]}^2}}}\left\{\begin{array}{cc}1\hfill & ,k\ne q\hfill \\ 1-q({\lambda }_i+1)\hfill & ,k=q\hfill \end{array}\right..\hfill \end{array}$$

(18)

Proof. 

See Appendix C. □

Note that in matrix notation, we can write

$${\widehat{M}}^{\left(q\right)}=\widehat{I}-q^{-1}\widehat{J}+\mu \mathrm{diag}\left({\widehat{e}}^{\left(q\right)}\right)=\widehat{I}+{\widehat{C}}^{\left(q\right)},$$

(19)

where $\widehat{I}$ denotes the identity matrix, and we have the following corollary:

Corollary 2. 

The matrix ${\widehat{M}}^{\left(q\right)}$ given by Equation (19) is diagonalizable by the matrix of eigenvectors ${\widehat{\Lambda }}^{\left(q\right)}$ given in Theorem 1, and we have

$$\begin{array}{c}\hfill {\left({\widehat{\Lambda }}^{\left(q\right)}\right)}^{\mathrm{T}}{\widehat{M}}^{\left(q\right)}{\widehat{\Lambda }}^{\left(q\right)}=\left(\begin{array}{ccccc}1& 0& \dots & 0& 0\\ 0& 1& \dots & 0& 0\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ 0& 0& \dots & m_1& 0\\ 0& 0& \dots & 0& m_2\end{array}\right)\equiv {\widehat{D}}^{\left(q\right)},\end{array}$$

(20)

where $m_1$ and $m_2$ are defined as

$$m_j\equiv {\lambda }_j+1,j\in \{1,2\}.$$

(21)

Proof. 

Follows from Theorem 1. □

Lemma 3. 

Let $\widehat{\mathbf{z}}={\widehat{\Lambda }}^{\left(q\right)}\widehat{\mathbf{u}}$, then the function F given in Corollary 1 can be written as

$$\begin{array}{cc}\hfill F\left({\widehat{\Lambda }}^{\left(q\right)}\widehat{\mathbf{u}}\right)=& -{\displaystyle \frac{\mu }{2}}\left({\displaystyle \frac{1}{\left|\right|{\widehat{x}}_1{\left|\right|}^2}}{\mathbf{u}}_{q-1}·{\mathbf{u}}_{q-1}+{\displaystyle \frac{2}{\left|\right|{\widehat{x}}_1\left|\right|\left|\right|{\widehat{x}}_2\left|\right|}}{\mathbf{u}}_{q-1}·{\mathbf{u}}_q+{\displaystyle \frac{1}{\left|\right|{\widehat{x}}_2{\left|\right|}^2}}{\mathbf{u}}_q·{\mathbf{u}}_q\right)\hfill \\ \hfill & +ln{q}^{-1}\sum _{l}exp\left\{-{\displaystyle \frac{\mu }{2}}\left(G_l+2{\delta }_{q,l}T\right)\right\},\hfill \end{array}$$

(22)

where we have

$$\begin{array}{cc}\hfill & {\widehat{x}}_j={(1,\dots ,1,x_j)}^{\mathrm{T}},j\in \{1,2\},\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill & x_j=1-q{m}_j,j\in \{1,2\},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & G_l=\sum _{m=1}^{q-2}\sum _{m^{\prime }=1}^{q-2}{\alpha }_{m,l}{\alpha }_{m^{\prime },l}{\mathbf{u}}_m·{\mathbf{u}}_{m^{\prime }}+{\displaystyle \frac{2}{\sqrt{q-1+x_1^2}}}\sum _{m=1}^{q-2}{\alpha }_{m,l}{\mathbf{u}}_m·{\mathbf{u}}_{q-1}+{\displaystyle \frac{2}{\sqrt{q-1+x_2^2}}}\sum _{m=1}^{q-2}{\alpha }_{m,l}{\mathbf{u}}_m·{\mathbf{u}}_q,\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & T={\displaystyle \frac{x_2^2-1}{2(q-1+x_2^2)}}{\mathbf{u}}_q·{\mathbf{u}}_q+{\displaystyle \frac{x_1^2-1}{2(q-1+x_1^2)}}{\mathbf{u}}_{q-1}·{\mathbf{u}}_{q-1}+{\displaystyle \frac{x_1{x}_2-1}{\sqrt{q-1+x_1^2}\sqrt{q-1+x_2^2}}}{\mathbf{u}}_q·{\mathbf{u}}_{q-1}.\hfill \end{array}$$

(26)

Proof. 

See Appendix D. □

2.5. Pulling Leading-Order Terms Out of the Integral

We now use Corollary 1 and Lemma 3 to further obtain an expression for the differential entropy that separates the leading contributions and the quantity S, which we define as follows:

$$S\equiv -{\mathbb{E}}_u\left[ln{q}^{-1}\sum _{l}exp\left\{-{\displaystyle \frac{\mu }{2}}\left(G_l+2{\delta }_{q,l}T\right)\right\}\right],$$

(27)

where ${\mathbb{E}}_u$ is the expectation taken with respect to the density ${\prod }_{i=1}^n{\prod }_{k=1}^q{g}_{0,d_k^{-1}}\left(u_{k,i}\right)$, and ${\left\{d_k\right\}}_{k=1}^q$ are given by:

$$d_k=\left\{\begin{array}{cc}1\hfill & k\le q-2\hfill \\ m_1\hfill & k=q-1\hfill \\ m_2\hfill & k=q\hfill \end{array}\right..$$

(28)

Theorem 2. 

Let $h_{\sigma }=ln\sigma \sqrt{2\pi e}$ be the differential entropy of a Gaussian distributed random variable with variance σ. The differential entropy $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ is given by

$$\begin{array}{c}\hfill h\left(\mathbf{X}\right|\widehat{\mathbf{W}})=n{h}_{\sigma }+{\displaystyle \frac{n}{2}}{\displaystyle \frac{q}{q-1}}\mu +S,\end{array}$$

(29)

where S is given by Equation (27).

Proof. 

See Appendix E. □

2.6. Third Change in Variables: Simplification

We would like to approximate S given in Equation (27). However, our current integration variables ${\left\{{\mathbf{u}}_k\right\}}_{k=1}^q$ do not give simple formulas for $G_l$, T, or S. By linearly mixing ${\mathbf{u}}_{q-1}$ and ${\mathbf{u}}_q$, we obtain a set of q-independent standard normal-distributed variables with the benefit that $G_l$ contains only one of the two newly introduced variables. We define the transformation matrix W as

$$W\equiv {\displaystyle \frac{1}{\sqrt{{\displaystyle \frac{1}{\mu }}+{\displaystyle \frac{q}{q-1}}}}}\left(\begin{array}{cc}{\displaystyle \frac{1/\sqrt{m_2}}{\sqrt{q-1+x_2^2}}}& {\displaystyle \frac{1/\sqrt{m_1}}{\sqrt{q-1+x_1^2}}}\\ {\displaystyle \frac{1/\sqrt{m_1}}{\sqrt{q-1+x_1^2}}}& -{\displaystyle \frac{1/\sqrt{m_2}}{\sqrt{q-1+x_2^2}}}\end{array}\right),$$

(30)

where note that we can use Equation (A21) to show that $\det \left(W\right)=-1$.

Lemma 4. 

Let $\mathbf{r}=\sqrt{m_2}{\mathbf{u}}_q$, $\mathbf{s}=\sqrt{m_1}{\mathbf{u}}_{q-1}$, and W be as given in Equation (30), then the variables $\tilde{\mathbf{r}}$ and $\tilde{\mathbf{s}}$ given by

$$\left(\begin{array}{c}\mathbf{r}\\ \mathbf{s}\end{array}\right)=W\left(\begin{array}{c}\tilde{\mathbf{r}}\\ \tilde{\mathbf{s}}\end{array}\right)$$

(31)

have independent standard normal distributions, i.e., we can write

$$\begin{array}{cc}\hfill & {\mathbb{E}}_u[·]={\mathbb{E}}_{u,\tilde{s},\tilde{r}}\left[\widetilde{·}\right],\hfill \end{array}$$

(32)

where $\left[\widetilde{·}\right]$ denotes the transformed expression, and ${\mathbb{E}}_{u,\tilde{s},\tilde{r}}\left[\widetilde{·}\right]$ is taken with regard to the density ${\prod }_{a=1}^n{g}_{0,1}\left({\tilde{r}}_a\right)g_{0,1}\left({\tilde{s}}_a\right){\prod }_{b=1}^{q-2}{g}_{0,1}\left(u_{b,a}\right)$.

Proof. 

See Appendix F. □

As a corollary, T and $G_l$ assume a simpler form in terms of our new variables.

Corollary 3. 

Let ${\mathbf{w}}_l$ be given by

$${\mathbf{w}}_l=\sum _{m=1}^{q-2}{\alpha }_{m,l}{\mathbf{u}}_m,$$

(33)

then after applying the change in variables in Lemma 4, T and $G_l$ can be written as

$$\begin{array}{cc}\hfill & G_l={\mathbf{w}}_l·{\mathbf{w}}_l+2\sqrt{{\displaystyle \frac{1}{\mu }}+{\displaystyle \frac{q}{q-1}}}{\mathbf{w}}_l·\tilde{\mathbf{r}},\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill & T={\displaystyle \frac{1}{2}}{\left(1+\mu {\displaystyle \frac{q}{q-1}}\right)}^{-1}\left[-{\displaystyle \frac{q}{q-1}}\left(2+\mu {\displaystyle \frac{q}{q-1}}\right)\tilde{\mathbf{r}}·\tilde{\mathbf{r}}+{\displaystyle \frac{q}{q-1}}\tilde{\mathbf{s}}·\tilde{\mathbf{s}}+{\displaystyle \frac{2}{\sqrt{\mu }}}\sqrt{{\displaystyle \frac{q}{q-1}}}\tilde{\mathbf{r}}·\tilde{\mathbf{s}}\right].\hfill \end{array}$$

(35)

Proof. 

It follows by direct substitution. □

In this new form (34) and (35), the powers of $\mu$ are explicitly visible. The variables ${\mathbf{w}}_l,\tilde{\mathbf{r}},\tilde{\mathbf{s}}$ do not generate any $\mu$-dependence when integrated. It is clear that $\mu G_l$ and $\mu T$ are of order $\mathcal{O}\left(\sqrt{\mu }\right)$; hence, we can formally proceed with a Taylor expansion of the $\exp$ function in (27).

3. Brute Force Expansion

We develop a series expansion in $\mu$ as follows. First, we expand the exp function in (27), with $G_l$ and T as defined in (34) and (35). This yields a complicated series, whose leading order behaviour is $1+\mathcal{O}\left(\sqrt{\mu }\right)$. Then, we substitute that series into the Taylor expansion of $ln(1+\epsilon )$ in order to evaluate S as a series expansion. Finally, we apply the expectation ${\mathbb{E}}_{u,\tilde{s},\tilde{r}}$, which is possible because we only obtain expectations of powers of $\mathbf{u},\tilde{\mathbf{r}},\tilde{\mathbf{s}}$, which are independent normal-distributed variables. Although all steps by themselves are straightforward enough, the result is a huge amount of bookkeeping. A number of things are worth noting:

• What helps in this exercise is that odd powers of ${\mathbf{w}}_l$, $\tilde{\mathbf{r}}$, and $\tilde{\mathbf{s}}$ lead to vanishing integrals.
• Furthermore, it is clear from the start that the odd powers of $\sqrt{\mu }$ will disappear in the end. This can be seen from the fact that a factor ${\displaystyle \frac{1}{\sqrt{\mu }}}$ in T and $G_l$ always occurs with an isolated $\tilde{\mathbf{r}}$; hence, any occurrence of an odd power of $\sqrt{\mu }$ comes with an odd power of $\tilde{\mathbf{r}}$.
• Due to the n-dimensional inner products that occur in $G_l$ and T, which consist of n independent terms, each power of $\mu$ is associated with a factor n. Consequently, the power series becomes a series not in $\mu$ but actually in $n\mu$. For convergence, the product $n\mu$ needs to be sufficiently small. Fortunately, we are allowed to work under the condition $n\le q$, as explained in Section 2.2.

We performed this brute-force exercise up to order $\mathcal{O}\left({\mu }^2\right)$.

Theorem 3. 

The differential entropy $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})$ up to order $\mathcal{O}\left({\mu }^2\right)$ is given by

$$\begin{array}{c}\hfill h\left(\mathbf{X}\right|\widehat{\mathbf{W}})=n{h}_{\sigma }+{\displaystyle \frac{n}{2}}\left(1-{\displaystyle \frac{1}{q}}\right)\mu -{\displaystyle \frac{n}{2}}\left(1-{\displaystyle \frac{1}{q}}\right){\displaystyle \frac{n{q}^{-1}+1}{2q}}{\mu }^2+\mathcal{O}\left({\mu }^3\right).\end{array}$$

(36)

Proof. 

See Appendix I. □

Theorem 3 is not directly comparable with results in the literature. However, we can make use of existing upper bounds in order to assess our expression. The Gaussian upper bound in Equation (2) is known to be loose, while the bound in (5) is closer to the value of the differential entropy. For our case, the bound in Equation (5) reads

$$h\left(\mathbf{X}\right|\widehat{\mathbf{W}})\le n{h}_{\sigma }+lnq.$$

(37)

It is easy to see that for $\mu \le q^{-1}$ and $n=q$, our second-order Taylor expansion in Equation (36) satisfies this bound up to the error produced by the truncation of the series, which is of order $o\left({\mu }^2\right)$.

4. Determinant-Based Approach

We introduce a less brute-force series expansion that does not expand the exp function but focuses on expanding the logarithm. This gives a power series in exponential expressions; each of these expressions can be integrated analytically, yielding a matrix determinant. For simplicity of notation we write the variables ${\mathbf{u}}_k,\tilde{\mathbf{r}},\tilde{\mathbf{s}}$ together as a q-component vector $\widehat{\mathbf{a}}$,

$$\widehat{\mathbf{a}}=({\mathbf{u}}_1,\dots ,{\mathbf{u}}_{q-2},\tilde{\mathbf{r}},\tilde{\mathbf{s}}).$$

(38)

The S is now written as follows:

$$S=-{\mathbb{E}}_a\left[ln{q}^{-1}\sum _{l=1}^{q}exp\left\{-{\displaystyle \frac{\mu }{2}}\left(G_l+2{\delta }_{q,l}T\right)\right\}\right],$$

(39)

where we have

$${\mathbb{E}}_a\left[\dots \right]={\left(\sqrt{2\pi }\right)}^{-nq}\int exp\left\{-{\displaystyle \frac{1}{2}}{\widehat{\mathbf{a}}}^{\mathrm{T}}\widehat{I}\widehat{\mathbf{a}}\right\}\left[\dots \right]\mathrm{d}{\mathbf{a}}_1\dots \mathrm{d}{\mathbf{a}}_q.$$

(40)

Next, we can write $G_l$ and T in vector-matrix-vector product form:

$$\mu G_l={\widehat{\mathbf{a}}}^{\mathrm{T}}{\widehat{Q}}_l\widehat{\mathbf{a}},2\mu T={\widehat{\mathbf{a}}}^{\mathrm{T}}\widehat{P}\widehat{\mathbf{a}},$$

(41)

where the matrix elements of ${\widehat{Q}}_l$ and $\widehat{P}$ are given by

$$\begin{array}{c}\hfill Q_{l,m{m}^{\prime }}=\mu {\omega }_{m,l}{\omega }_{m^{\prime },l}+\sqrt{\mu }\sqrt{1+\mu {\displaystyle \frac{q}{q-1}}}[{\omega }_{m,l}{\delta }_{m^{\prime },q-1}+{\omega }_{m^{\prime },l}{\delta }_{m,q-1}],\end{array}$$

(42)

where we define

$${\omega }_{m,l}\equiv \left\{\begin{array}{cc}{\alpha }_{m,l}\hfill & ,m\le q-2\hfill \\ 0\hfill & ,\mathrm{otherwise}\hfill \end{array}\right..$$

(43)

$$\begin{array}{c}\hfill P_{m,m^{\prime }}={\left(1+\mu {\displaystyle \frac{q}{q-1}}\right)}^{-1}\left\{\begin{array}{cc}\mu {\displaystyle \frac{q}{q-1}}\hfill & ,m=m^{\prime }=q\hfill \\ -\mu {\displaystyle \frac{q}{q-1}}\left(2+\mu {\displaystyle \frac{q}{q-1}}\right)\hfill & ,m=m^{\prime }=q-1\hfill \\ \sqrt{\mu }\sqrt{{\displaystyle \frac{q}{q-1}}}\hfill & ,m=q,m^{\prime }=q-1\hfill \\ \sqrt{\mu }\sqrt{{\displaystyle \frac{q}{q-1}}}\hfill & ,m=q-1,m^{\prime }=q\hfill \\ 0\hfill & ,\mathrm{otherwise}\hfill \end{array}\right..\end{array}$$

(44)

The $\widehat{P}$ is essentially a $2\times 2$ matrix. The ${\widehat{Q}}_{\ell }$ consists of the projector ${\widehat{\alpha }}_{\ell }{\widehat{\alpha }}_{\ell }^T$ plus extra entries in row and column $q-1$. Let Z be defined as

$$\begin{array}{cc}\hfill Z& =1-q^{-1}\sum _{l=1}^{q}exp\left\{-{\displaystyle \frac{\mu }{2}}\left(G_l+2{\delta }_{q,l}T\right)\right\}\hfill \\ \hfill & =1-q^{-1}\sum _{l=1}^{q-1}exp\left\{-{\displaystyle \frac{1}{2}}{\widehat{\mathbf{a}}}^{\mathrm{T}}{\widehat{Q}}_l\widehat{\mathbf{a}}\right\}-q^{-1}exp\left\{-{\displaystyle \frac{1}{2}}{\widehat{\mathbf{a}}}^{\mathrm{T}}\widehat{P}\widehat{\mathbf{a}}\right\},\hfill \end{array}$$

(45)

where equality follows from the definitions (41). We note that $Z=\mathcal{O}\left(\sqrt{\mu }\right)$. This allows us to perform an expansion in Z, knowing that $Z^k$ does not produce powers of $\mu$ lower than ${\mu }^{k/2}$. Furthermore, the power $\sqrt{\mu }$ occurs only in the off-diagonal components of $\widehat{P}$ and ${\widehat{Q}}_{\ell }$, whereas the diagonal components are $\mathcal{O}\left(\mu \right)$. As in the brute-force approach, the integration over $\widehat{\mathbf{a}}$ eliminates half-integer powers of $\mu$. Hence, ${\mathbb{E}}_a{Z}^k=\mathcal{O}\left({\mu }^{\lceil k/2\rceil }\right)$. In order to obtain the entropy estimation up to and including power ${\mu }^t$, we need all contributions up to and including ${\mathbb{E}}_a{Z}^{2t}$. Using $ln(1-Z)=-{\sum }_{k=1}^{\infty }{k}^{-1}{Z}^k$, one can easily show that S (39) can be written as

$$S=\sum _{k=1}^{\infty }{\mathcal{Z}}_k,{\mathcal{Z}}_k={\displaystyle \frac{1}{k}}{\mathbb{E}}_a{Z}^k.$$

(46)

This provides a recipe for going to arbitrary order in $\mu$. We show the calculation of the ${\mu }^1$ contribution in S. For this, we need only ${\mathcal{Z}}_1$ and ${\mathcal{Z}}_2$.

Lemma 5. 

For a positive definite matrix $\widehat{V}$, we have

$\int exp\left\{-{\displaystyle \frac{1}{2}}{\widehat{\mathbf{u}}}^{\mathrm{T}}\widehat{V}\widehat{\mathbf{u}}\right\}\mathrm{d}{\mathbf{u}}_1\dots \mathrm{d}{\mathbf{u}}_q={\displaystyle \frac{{\left(2\pi \right)}^{nq/2}}{det{\left(\widehat{V}\right)}^{n/2}}}$.

Proof. 

See Appendix G. □

Lemma 6. 

It holds that

$$\begin{array}{cc}\hfill & {\mathcal{Z}}_1=1-\sum _{\ell =1}^{q-1}{\displaystyle \frac{q^{-1}}{det{(\widehat{I}+{\widehat{Q}}_{\ell })}^{n/2}}}-{\displaystyle \frac{q^{-1}}{det{(\widehat{I}+\widehat{P})}^{n/2}}},\hfill \\ \hfill & {\mathcal{Z}}_2={\displaystyle \frac{1}{2}}-\sum _{\ell =1}^{q-1}{\displaystyle \frac{q^{-1}}{det{(\widehat{I}+{\widehat{Q}}_{\ell })}^{n/2}}}-{\displaystyle \frac{q^{-1}}{det{(\widehat{I}+\widehat{P})}^{n/2}}}+\sum _{\ell =1}^{q-1}{\displaystyle \frac{q^{-2}}{det{(\widehat{I}+{\widehat{Q}}_{\ell }+\widehat{P})}^{n/2}}}\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill & +\sum _{\ell =1}^{q-1}\sum _{{\ell }^{\prime }=1}^{q-1}{\displaystyle \frac{(1/2)q^{-2}}{det{(\widehat{I}+{\widehat{Q}}_{\ell }+{\widehat{Q}}_{{\ell }^{\prime }})}^{n/2}}}+{\displaystyle \frac{(1/2)q^{-2}}{det{(\widehat{I}+2\widehat{P})}^{n/2}}}.\hfill \end{array}$$

(48)

Proof. 

See Appendix H. □

Proposition 1. 

For $t\in \mathbb{N}$, ${\ell }^{\prime }\ne \ell$

$$\begin{array}{ccc}\hfill det(\widehat{I}+t\widehat{P})& =& 1-\mu t(t+1){\displaystyle \frac{q}{q-1}},\hfill \end{array}$$

(49)

$$\begin{array}{ccc}\hfill det(\widehat{I}+t{\widehat{Q}}_{\ell })& =& 1-\mu t(t-1){\displaystyle \frac{q-2}{q-1}}-t^2{\mu }^2{\displaystyle \frac{q(q-2)}{{(q-1)}^2}},\hfill \end{array}$$

(50)

$$\begin{array}{ccc}\hfill det(\widehat{I}+\widehat{P}+{\widehat{Q}}_{\ell })& =& 1-\mu {\displaystyle \frac{2q}{q-1}}-{\mu }^2{\displaystyle \frac{4q(q-2)}{{(q-1)}^2}},\hfill \end{array}$$

(51)

$$\begin{array}{ccc}\hfill det(\widehat{I}+{\widehat{Q}}_{\ell }+{\widehat{Q}}_{{\ell }^{\prime }})& =& 1+\mu {\displaystyle \frac{2}{q-1}}-{\mu }^2{\displaystyle \frac{(3q-1)(q-3)}{{(q-1)}^2}}-{\mu }^3{\displaystyle \frac{2q(q-3)}{{(q-1)}^2}}.\hfill \end{array}$$

(52)

Corollary 4. 

It holds that

$${\mathcal{Z}}_1=-{\displaystyle \frac{n}{q-1}}\mu +\mathcal{O}\left({\mu }^2\right),{\mathcal{Z}}_2={\displaystyle \frac{n/2}{q(q-1)}}\mu +\mathcal{O}\left({\mu }^2\right).$$

(53)

Using (29) it follows that $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})=n{h}_{\sigma }+{\displaystyle \frac{n}{2}}{\displaystyle \frac{q}{q-1}}\mu +S$, with $S\approx {\mathcal{Z}}_1+{\mathcal{Z}}_2$, yielding $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})=n{h}_{\sigma }+{\displaystyle \frac{n}{2}}(1-{\displaystyle \frac{1}{q}})\mu +\mathcal{O}\left({\mu }^2\right)$. This is consistent with the result of the brute-force approach, as in Theorem 3.

In higher orders of Z, determinants occur of the form $det(\widehat{I}+r\widehat{P}+t{\widehat{Q}}_{\ell }+t^{\prime }{\widehat{Q}}_{{\ell }^{\prime }}+t^{″}{\widehat{Q}}_{{\ell }^{″}}+\dots )$ for integer $r,t,t^{\prime },t^{″},\dots$. These are computable but involve a lot of bookkeeping. Because of the form ${[det(\dots )]}^{-n/2}$ and the fact that each determinant starts with 1, followed by integer powers of $\mu$, the leading order coefficient of ${\mu }^t$ is proportional to $\left({\displaystyle \genfrac{}{}{0pt}{}{n/2}{t}}\right)\propto n^t$. Hence, as in the brute force approach, the expansion in powers of $\mu$ effectively becomes an expansion in powers of $n\mu$.

5. Discussion

Gaussian mixtures occur in the literature either as an approximation to a probability density function or as the actual density of a two-stage stochastic process. Shannon entropy plays an important role in the signal processing, optimization, and analysis of systems. It is important to have (semi-)analytical methods for evaluating the Shannon entropy (differential entropy) of Gaussian mixtures, without lengthy numerical computations in high-dimensional spaces. We have developed an expansion method that has a number of advantages: (i) it yields an entropy estimate rather than merely bounds; (ii) the order of magnitude of the error is known; (iii) it avoids the process of [20] that splits wide Gaussians into narrow pieces, which leads to a large number of expansion terms and introduces inaccuracies. The inaccuracies in [20] grow with the number of wide Gaussians in the mixture. The approach in [20] is unable to exploit the existence of a small parameter $\mu$ when the mixture structure allows it. In our case, we make sure to leverage the trade-off between the support shared between the components and the variance of the components, which makes our method more efficient when the mixture is not composed of separated clusters. In the case of negligible shared support between the components, an approximation is not needed as the bound in Equation (37) becomes exact. From the second-order result (36), it seems that convergence is faster than what we expected. In leading order terms, every power of $\mu$ is accompanied by a power of n, which may endanger the radius of convergence. However, in the ${\mu }^2$ part in (36), we see that the additional power of n comes as $n/q$, which is not dangerous since we are allowed to work with $n\le q$. Setting $n=q$ and $q\gg 1$, (36) reduces to $h\left(\mathbf{X}\right|\widehat{\mathbf{W}})\approx {\displaystyle \frac{n}{2}}[ln2\pi e{\sigma }^2+\mu -{\displaystyle \frac{1}{q}}{\mu }^2+\mathcal{O}\left({\mu }^3\right)]$, which has fast convergence. In future work, we will try to apply our method for fixed displacements $\widehat{\mathbf{w}}$ and study the convergence at higher orders of $\mu$.