MEMe: An Accurate Maximum Entropy Method for Efficient Approximations in Large-Scale Machine Learning

Abstract

Efficient approximation lies at the heart of large-scale machine learning problems. In this paper, we propose a novel, robust maximum entropy algorithm, which is capable of dealing with hundreds of moments and allows for computationally efficient approximations. We showcase the usefulness of the proposed method, its equivalence to constrained Bayesian variational inference and demonstrate its superiority over existing approaches in two applications, namely, fast log determinant estimation and information-theoretic Bayesian optimisation.

1. Introduction

Algorithmic scalability is an important component of modern machine learning. Making high quality inference on large, feature rich datasets under a constrained computational budget is arguably the primary goal of the learning community. This, however, comes with significant challenges. On the one hand, the exact computation of linear algebraic quantities may be prohibitively expensive, such as that of the log determinant. On the other hand, an analytic expression for the quantity of interest may not exist at all, such as the case for the entropy of a Gaussian mixture model, and approximate methods are often both inefficient and inaccurate. These highlight the need for efficient approximations especially in solving large-scale machine learning problems.

In this paper, to address this challenge, we propose a novel, robust maximum entropy algorithm, stable for a large number of moments, surpassing the limit of previous maximum entropy algorithms [1,2,3]. We show that the ability to handle more moment information, which can be calculated cheaply either analytically or with the use of stochastic trace estimation, leads to significantly enhanced performance. We showcase the effectiveness of the proposed algorithm by applying it to log determinant estimation [4,5,6] and entropy term approximation in the information-theoretic Bayesian optimisation [7,8,9]. Specifically, we reformulate the log determinant estimation into an eigenvalue spectral estimation problem so that we can estimate the log determinant of a symmetric positive definite matrix via computing the maximum entropy spectral density of its eigenvalues. Similarly, we learn the maximum entropy spectral density for the Gaussian mixture and then approximate the entropy of the Gaussian mixture via the entropy of the maximum entropy spectral density, which provides an analytic upper bound. Furthermore, in developing our algorithm, we establish equivalence between maximum entropy methods and constrained Bayesian variational inference [10].

The main contributions of the paper are as follows:

• We propose a maximum entropy algorithm, which is stable and consistent for hundreds of moments, surpassing other off-the-shelf algorithms with a limit of a small number of moments. Based on this robust algorithm, we develop a new Maximum Entropy Method (MEMe) which improves upon the scalability of existing machine learning algorithms by efficiently approximating computational bottlenecks using maximum entropy and fast moment estimation techniques;
• We establish the link between maximum entropy methods and variational inference under moment constraints, hence connecting the former to well-known Bayesian approximation techniques;
• We apply MEMe to the problem of estimating the log determinant, crucial to inference in determinental point processes [11], and to that of estimating the entropy of a Gaussian mixture, important to state-of-the-art information-theoretic Bayesian optimisation algorithms.

2. Theoretical Framework

The method of maximum entropy, hereafter referred to as MaxEnt [12], is a procedure for generating the most conservative estimate of a probability distribution with the given information and the most non-committal one with regard to missing information [13]. Intuitively, in a bounded domain, the most conservative distribution, i.e., the distribution of maximum entropy, is the one that assigns equal probability to all the accessible states. Hence, the method of maximum entropy can be thought of as choosing the flattest, or most equiprobable distribution, satisfying the given moment constraints.

To determine the maximally entropic density $q(x)$, we maximise the entropic functional

$$S=-\int q(x)logq(x)dx-\sum _{i=0}^m{\alpha }_i\left[\int q(x)x^{i}dx-{\mu }_i\right],$$

(1)

with respect to $q(x)$, where the second term with ${\mu }_i={\mathbb{E}}_p[x^i]$ for some density $p(x)$ are the power moment constraints on the density $q(x)$, $\left\{{\alpha }_i\right\}$ are the corresponding Lagrange multipliers, and m is the number of moments. The first term in Equation (1) is referred to as the Boltzmann–Shannon–Gibbs (BSG) entropy, which has been applied in multiple fields, ranging from condensed matter physics [14] to finance [15]. For the case of $i\le 2$, the Lagrange multipliers can be calculated analytically; for $i\ge 3$, they must be determined numerically.

In this section, we first establish links between the method of MaxEnt and Bayesian variational inference. We then describe fast moment estimation techniques. Finally, we present the proposed MaxEnt algorithm.

2.1. Maximum Entropy as Constrained Bayesian Variational Inference

The work of Bretthorst [16] makes the claim that the method of maximum entropy (MaxEnt) is fundamentally at odds with Bayesian inference. At the same time, variational inference [17] is a widely used approximation technique that falls under the category of Bayesian learning. In this section, we show that the method of maximum relative entropy [18] is equivalent to constrained variational inference, thus establishing the link between MaxEnt and Bayesian approximation.

2.1.1. Variational Inference

Variational methods [10,17] in machine learning pose the problem of intractable density estimation from the application of Bayes’ rule as a functional optimisation problem:

$$p(z|x)=\frac{p(x|z)p(z)}{p(x)}\approx q(z),$$

(2)

where $p(z)$ and $p(z|x)$ represent the prior and posterior distributions of the random variable z, respectively. Variational inference therefore seeks to find $q(z)$ as an approximation of the posterior $p(z|x)$, which has the benefit of being a strict bound to the true posterior.

Typically, while the functional form of $p(x|z)$ is known, calculating $p(x)=\int p(x|z)p(z)dz$ is intractable. Using Jensen’s inequality, we can show that:

$$logp(x)\ge {\mathbb{E}}_q[logp(x,z)]-{\mathbb{E}}_q[logq(z)].$$

(3)

Furthermore, the reverse Kullback-Leibler (KL) divergence between the posterior and the variational distribution, ${\mathcal{D}}_{\mathrm{KL}}(q|p)$, can be written as:

$$logp(x)={\mathbb{E}}_q[logp(x,z)]-{\mathbb{E}}_q[logq(z)]+{\mathcal{D}}_{\mathrm{KL}}(q|p).$$

(4)

Hence, maximising the evidence lower bound is equivalent to minimising the reverse KL divergence between p and q.

2.1.2. MaxEnt Is Equivalent to Constrained Variational Inference

Minimising the reverse KL divergence between our posterior $q(x)$ and our prior $q_0(x)$ as is done in variational inference, with respect to $q(x)$:

$${\mathcal{D}}_{\mathrm{KL}}(q|q_0)=-H(q)-{\int }_{x\in \mathcal{D}}q(x)log{q}_0(x)dx,$$

(5)

where $H(q)$ denotes the differential entropy of the density $q(x)$, such that $\int q(x)dx=1$ and $\int q(x)x^{i}dx={\mu }_i$. By the theory of Lagrangian duality, the convexity of the KL divergence, and the affine nature of the moment constraints, we maximise the dual form [19] of Equation (5):

$$-H(q)-\int q(x)log{q}_0(x)dx-\sum _{i=0}^m{\alpha }_i\left(\int q(x)x^{i}dx-{\mu }_i\right)$$

(6)

with respect to $q(x)$ or, alternatively, we minimise

$$H(q)+\int q(x)log{q}_0(x)dx+\sum _{i=0}^m{\alpha }_i\left(\int q(x)x^{i}dx-{\mu }_i\right).$$

(7)

In the field of information physics, the minimisation of Equation (7) is known as the method of relative entropy [18]. It can be derived as the unique functional satisfying the axioms of locality, coordinate invariance, sub-system invariance and objectivity.

The restriction to a functional is derived from considering the set of all distributions $\mathcal{Q}=\left\{q_j(x)\right\}$ compatible with the constraints and devising a transitive ranking scheme (Transitive ranking means if $A>B$ and $B>C$, then $A>C$). Furthermore, it can be shown that Newton’s laws, non-relativistic quantum mechanics, and Bayes’ rule can all be derived under this formalism [18]. In the case of a flat prior, we reduce to the method of maximum entropy with moment constraints [13] as $q_0(x)$ is a constant. Hence, MaxEnt and constrained variational inference are equivalent.

2.2. Fast Moment Estimation Techniques

As shown in Section 2.1, constrained variational inference requires the knowledge of the moments of the density being approximated to make inference. In this section, we discuss fast moment estimation techniques for both the general case and the case for which analytic expressions exist.

2.2.1. Moments of a Gaussian Mixture

In some problems, there exist analytic expressions which can be used to compute the moments, One popular example is the Gaussian distribution (and the mixture of Gaussians). Specifically, for the one-dimensional Gaussian, we can find an analytic expression for the moments:

$${\int }_{-\infty }^{\infty }{x}^m{\mathrm{e}}^{-\frac{{(x-\mu )}^2}{2{\sigma }^2}}dx=\sum _{i=0}^m{(\sqrt{2}\sigma )}^{i+1}\left(\genfrac{}{}{0pt}{}{m}{i}\right){\mu }^{m-i}{\zeta }_i,$$

(8)

where $\mu$ and $\sigma$ are the mean and standard deviation of the Gaussian, respectively, and:

$${\zeta }_i={\int }_{-\infty }^{\infty }{y}^i{\mathrm{e}}^{-y^2}dy=\left\{\begin{array}{cc}\frac{(i-1)!!\sqrt{\pi }}{2^{i/2}},\hfill & i\mathrm{even},\hfill \\ [1/2(i-1)!,\hfill & i\mathrm{odd}.\hfill \end{array}\right.$$

(9)

where $(i-1)!!$ denotes double factorial. Hence, for a mixture of k Gaussians with mean ${\mu }_k$ and standard deviation ${\sigma }_k$, the n-th moment is analytic:

$$\sum _k{w}_k\sum _{i=0}^m{(\sqrt{2}{\sigma }_k)}^{i+1}\left(\genfrac{}{}{0pt}{}{m}{i}\right){\mu }_k^{m-i}{\zeta }_i.$$

(10)

where $w_k$ is the weight of k-th Gaussian in the mixture.

2.2.2. Moments of the Eigenvalue Spectrum of a Matrix

Stochastic trace estimation is an effective way of cheaply estimating the moments of the eigenvalue spectrum of a given matrix $K\in {\mathbb{R}}^{n\times n}$ [20]. The essential idea is that we can estimate the non-central moments of the spectrum by using matrix-vector multiplications.

Recall that for any multivariate random variable $\mathbf{z}$ with mean $\mathbf{m}$ and variance $\mathsf{\Sigma }$, we have:

$$\mathbb{E}({\mathbf{z}}^T\mathbf{z})={\mathbf{m}}^T\mathbf{m}+\mathsf{\Sigma }\stackrel{\mathbf{m}=\mathbf{0},\mathsf{\Sigma }=I}{=}I,$$

(11)

where for the second equality we have assumed (without loss of generality) that $\mathbf{z}$ possesses zero mean and unit variance. By the linearity of trace and expectation, for any number of moments $m\ge 0$ we have:

$$\sum _{s=1}^n{\lambda }_s^m=\mathrm{Tr}(I{K}^m)=\mathbb{E}(\mathbf{z}{K}^m{\mathbf{z}}^T),$$

(12)

where $\left\{{\lambda }_s\right\}$ are the eigenvalues of K. In practice, we approximate the expectation in Equation (12) with a set of probing vectors and a simple Monte Carlo average, i.e., for d random unit vectors ${\left\{{\mathbf{z}}_j\right\}}_{j=1}^d$:

$$\mathbb{E}(\mathbf{z}{K}^m{\mathbf{z}}^T)\approx \frac{1}{d}\left(\sum _{j=1}^d{\mathbf{z}}_j{K}^m{\mathbf{z}}_j^T\right),$$

(13)

where we take the product of the matrix K with the vector ${\mathbf{z}}_j^T$ a total of m times and update ${\mathbf{z}}_j^T$ each time, so as to avoid the costly matrix multiplication with $\mathcal{O}(n^3)$ complexity. This allows us to calculate the non-central moment expectations in $\mathcal{O}(dm{n}^2)$ complexity for dense matrices, or $\mathcal{O}(dm\times n_{nz})$ complexity for sparse matrices, where $d\times m\ll n$ and $n_{nz}$ is the number of non-zero entries in the matrix. The random vector ${\mathbf{z}}_j$ can be drawn from any distribution, such as a Gaussian. The resulting stochastic trace estimation algorithm is outlined in Algorithm 1.

Algorithm 1 Stochastic Trace Estimation [20]

1: Input: Matrix $K\in {\mathbb{R}}^{n\times n}$, Number of Moments m, Number of Probe Vectors d 2: Output: Moment Expectations $\widehat{{\mu }_i}$, $\forall i\le m$ 3: Initialise $\widehat{{\mu }_i}=0$ $\forall i$ 4: for$j=1,\cdots ,d$do 5:   Initialise ${\mathbf{z}}_j=\mathrm{rand}(1,n)$ 6:   for $i=1,\cdots ,m$ do 7:     ${\mathbf{z}}_j^T=K{\mathbf{z}}_j^T$ 8:     $\widehat{{\mu }_i}=\widehat{{\mu }_i}+\frac{1}{d}{\mathbf{z}}_j{\mathbf{z}}_j^T$ 9:   end for 10: end for

2.3. The Proposed MaxEnt Algorithm

In this section, we develop an algorithm for determining the maximum relative entropy density given moment constraints. Under the assumption of a flat prior in a bounded domain, this reduces to a generic MaxEnt density, which we use in various examples.

As we discussed previously, in order to obtain the MaxEnt distribution, we maximise the generic objective of Equation (1). In practice, we instead minimise the dual form of this objective [19], which can be written as follows:

$$\mathcal{S}(q,q_0)={\int }_{x\in \mathcal{D}}{q}_0(x)exp(-[1+\sum _{i=0}^m{\alpha }_i{x}^i])dx+\sum _{i=0}^m{\alpha }_i{\mu }_i.$$

(14)

Notice that this dual objective admits analytic expressions for both the gradient and the Hessian:

$$\frac{\partial \mathcal{S}(q,q_0)}{\partial {\alpha }_j}={\mu }_j-{\int }_{x\in \mathcal{D}}{q}_0(x)x^{j}exp(-[1+\sum _{i=0}^m{\alpha }_i{x}^i])dx,$$

(15)

$$\frac{{\partial }^2\mathcal{S}(q,q_0)}{\partial {\alpha }_j\partial {\alpha }_k}={\int }_{x\in \mathcal{D}}{q}_0(x)x^{j+k}exp(-[1+\sum _{i=0}^m{\alpha }_i{x}^i])dx.$$

(16)

For a flat prior in a bounded domain, which we use as a prior for the spectral densities of large matrices and for Gaussian mixture models, $q_0(x)$ is a constant that can be dropped out. With the notation

$$q_{\alpha }(x)=exp(-[1+\sum _{i=0}^m{\alpha }_i{x}^i]),$$

(17)

we obtain the MaxEnt algorithm in Algorithm 2.

The input $\left\{{\mu }_i\right\}$ of Algorithm 2 are estimated using fast moment estimation techniques, as explained in Section 2.2.2. In our implementation, we use Python’s SciPy Newton-conjugate gradient (CG) algorithm to solve the minimisation in step 4 (line 6), having firstly computed the gradient within a tolerance level of $ϵ$ as well as the Hessian. To make the Hessian better conditioned so as to achieve convergence, we symmetrise it and add jitter of intensity $\eta ={10}^{-8}$ along the diagonal. We estimate the given integrals with quadrature using a grid size of ${10}^{-4}$. Empirically, we observe that the algorithm is not overly sensitive to these choices. In particular, we find that, for well conditioned problems, the need for jitter is obviated and a smaller grid size works fine; in the case of worse conditioning, jitter helps improve convergence and the grid size becomes more important, where a smaller grid size leads to a computationally more intensive implementation.

Algorithm 2 The Proposed MaxEnt Algorithm

1: Input: Moments $\left\{{\mu }_i\right\}$, Tolerance Level $ϵ$, Jitter variance in Hessian $\eta ={10}^{-8}$ 2: Output: Coefficients $\left\{{\alpha }_i\right\}$ 3: Initialise ${\alpha }_i=0$ 4: Compute gradient: $g_j={\mu }_j-{\int }_0^1{q}_{\alpha }(x)x^{j}dx$ 5: Compute Hessian: $H_{jk}={\int }_0^1{q}_{\alpha }(x)x^{j+k}dx,H=\frac{1}{2}(H+H^T)+\eta I$ 6: Minimize ${\int }_0^1{q}_{\alpha }(x)dx+{\sum }_i{\alpha }_i{\mu }_i$ using Conjugate Gradients until $\forall j$: $g_j<ϵ$

Given that any polynomial sum can be written as ${\sum }_i{\alpha }_i{x}^i={\sum }_i{\beta }_i{f}_i(x)$, where $f_i(x)$ denotes another polynomial basis, whether we choose to use power moments in our constraints or another polynomial basis, such as the Chebyshev or Legendre basis, does not change the entropy or solution of our objective. However, the performance of optimisers working on these different formulations may vary [21]. For simplicity, we have kept all the formulas in terms of power moments. However, we find vastly improved performance and conditioning when we switch to orthogonal polynomial bases (so that the errors between moment estimations are uncorrelated), as shown in Section 3.2.2. We implement both Chebyshev and Legendre moments in our Lagrangian and find similar performance for both.

3. Applications

We apply the proposed algorithm to two problems of interest, namely, log determinant estimation and Bayesian optimisation. In both cases we demonstrate substantial speed up and improvement in performance.

3.1. Log Determinant Estimation

Calculation of the log determinant is a common hindrance in machine learning, appearing in Gaussian graphical models [22], Gaussian processes [23], variational inference [24], metric and kernel learning [25], Markov random fields [26] and determinantal point processes [11]. For a large positive definite matrix $K\in {\mathbb{R}}^{n\times n}$, the canonical solution involves the Cholesky decomposition of $K=L{L}^T$. The log determinant is then trivial to calculate as $log\det (K)=2{\sum }_{i=1}^{n}log{L}_{ii}$, where $L_{ii}$ is the $ii$-th entry of L. This computation invokes a computational complexity of $\mathcal{O}(n^3)$ and storage complexity of $\mathcal{O}(n^2)$, which becomes prohibitive for large n, i.e., $n>{10}^4$.

3.1.1. Log Determinant Estimation as a Spectral Estimation Problem Using MaxEnt

Any symmetric positive definite (PD) matrix K is diagonalisable by a unitary matrix U, i.e., $K=U^{T}DU$, where D is the diagonal matrix of eigenvalues of K. Hence we can write the log determinant as:

$$log\det (K)=log\prod _i{\lambda }_i=\sum _{i=1}^{n}log{\lambda }_i=n{\mathbb{E}}_p(log\lambda ),$$

(18)

where we have used the cyclicity of the determinant and ${\mathbb{E}}_p(log\lambda )$ denotes the expectation under the spectral measure. The latter can be written as:

$${\mathbb{E}}_p(log\lambda )={\int }_{{\lambda }_{\min }}^{{\lambda }_{\max }}p(\lambda )log\lambda d\lambda ={\int }_{{\lambda }_{\min }}^{{\lambda }_{\max }}\sum _{i=1}^n\frac{1}{n}\delta (\lambda -{\lambda }_i)log\lambda d\lambda ,$$

(19)

where ${\lambda }_{\max }$ and ${\lambda }_{\min }$ correspond to the largest and smallest eigenvalues, respectively.

Given that the matrix is PD, we know that ${\lambda }_{\min }>0$ and we can divide the matrix by an upper bound of the eigenvalue, ${\lambda }_{\max }\le {\lambda }_u$, via the Gershgorin circle theorem [27] such that:

$$log\det (K)=n{\mathbb{E}}_p(log{\lambda }^{\prime })+nlog{\lambda }_u,$$

(20)

where ${\lambda }^{\prime }=\lambda /{\lambda }_u$ and ${\lambda }_u={\max }_i({\sum }_{j=1}^n|K_{ij}|)$, i.e., the maximum sum of the rows of the absolute of the matrix K. Hence we can comfortably work with the transformed measure:

$${\int }_{{\lambda }_{\min }/{\lambda }_u}^{{\lambda }_{\max }/{\lambda }_u}p({\lambda }^{\prime })log{\lambda }^{\prime }d{\lambda }^{\prime }={\int }_0^{1}p({\lambda }^{\prime })log{\lambda }^{\prime }d{\lambda }^{\prime },$$

(21)

where the spectral density $p({\lambda }^{\prime })$ is 0 outside of its bounds of $[0,1]$. We therefore arrive at the following maximum entropy method (MEMe) for log determinant estimation detailed in Algorithm 3.

Algorithm 3 MEMe for Log Determinant Estimation

1: Input: PD Symmetric Matrix $K\in {\mathbb{R}}^{n\times n}$, Number of Moments m, Number of Probe Vectors d, Tolerance Level $ϵ$ 2: Output: Log Determinant Approximation $log\det (K)$ 3: ${\lambda }_u={\max }_i({\sum }_{j=1}^n|K_{ij}|)$ 4: $B=K/{\lambda }_u$ 5: Compute moments, $\left\{{\mu }_i\right\}$, via Stochastic Trace Estimation (Algorithm 1) with inputs $(B,m,d)$ 6: Compute coefficients, $\left\{{\alpha }_i\right\}$, via MaxEnt (Algorithm 2) with inputs $(\left\{{\mu }_i\right\},ϵ)$ 7: Compute $q(\lambda )=exp[-(1+{\sum }_i{\alpha }_i{\lambda }^i)]$ 8: Estimate $log\det (K)\approx n\int log(\lambda )q(\lambda )d\lambda +nlog({\lambda }_u)$

3.1.2. Experiments

Log Determinant Estimation for Synthetic Kernel Matrix

To specifically compare against commonly used Chebyshev [4] and Lanczos approximations [28] to the log determinant, and see how their accuracy degrades with worsening condition number, we generate a typical squared exponential kernel matrix [23], $K\in {\mathbb{R}}^{n\times n}$, using the Python GPy package with 6 dimensional Gaussian inputs with a variety of realistic uniform length-scales. We then add noise of variance $\eta ={10}^{-8}$ along the diagonal of K.

We use $m=30$ moments and $d=50$ Hutchinson probe vectors [20] in MEMe for the log determinant estimation, and display the absolute relative estimation error for different approaches in

Table 1. Relative estimation error for MEMe, Chebyshev, and Lanczos approaches, with various length-scale l and condition number $\kappa$ on the squared exponential kernel matrix $K\in {\mathbb{R}}^{1000\times 1000}$.

$\mathbf{\kappa }$	l	MEMe	Chebyshev	Lanczos
$3\times {10}^1$	0.05	0.0014	0.0037	0.0024
$1.1\times {10}^3$	0.15	0.0522	0.0104	0.0024
$1.0\times {10}^5$	0.25	0.0387	0.0795	0.0072
$2.4\times {10}^6$	0.35	0.0263	0.2302	0.0196
$8.3\times {10}^7$	0.45	0.0284	0.3439	0.0502
$4.2\times {10}^8$	0.55	$\mathbf{0.0256}$	0.4089	0.0646
$4.3\times {10}^9$	0.65	$\mathbf{0.00048}$	0.5049	0.0838
$1.4\times {10}^{10}$	0.75	0.0086	0.5049	0.1050
$4.2\times {10}^{10}$	0.85	0.0177	0.5358	0.1199

. We see that, for low condition numbers, the benefit of framing the log determinant as an optimisation problem is marginal, whereas for large condition numbers, the benefit becomes substantial, with orders of magnitude better results than competing methods.

Log Determinant Estimation on Real Data

The work in Granziol and Roberts [29] has shown that the addition of an extra moment constraint cannot increase the entropy of the MaxEnt solution. For the problem of log determinant, this signifies that the entropy of the spectral approximation should decrease with the addition of every moment constraint. We implement the MaxEnt algorithm proposed in Bandyopadhyay et al. [2], which we refer to as OMxnt, in the same manner as applied for log determinant estimation in Fitzsimons et al. [30], and compare it against the proposed MEMe approach. Specifically, we show results on the Ecology dataset [31], with $n=999,999$, for which the true log determinant can be calculated. For OMxnt, we see that after the initial decrease, the error (Figure 1b) begins to increase for $m>3$ moments and the entropy (Figure 1a) increases at $m=6$ and $m=12$ moments. For the proposed MEMe algorithm, the performance is vastly improved in terms of estimation error (Figure 1d); furthermore, the error continually decreases with increasing number of moments, and the entropy (Figure 1c) decreases smoothly. This demonstrates the superiority both in terms of consistency and performance of our novel algorithm over established existing alternatives.

3.2. Bayesian Optimisation

Bayesian Optimisation (BO) is a powerful tool to tackle global optimisation challenges. It is particularly useful when the objective function is unknown, non-convex and very expensive to evaluate [32], and has been successfully applied in a wide range of applications including automatic machine learning [33,34,35,36], robotics [37,38] and experimental design [39]. When applying BO, we need to choose a statistical prior to model the objective function and define an acquisition function which trades off exploration and exploitation to recommend the next query location [40]. The generic BO algorithm is presented in Algorithm A1 in Appendix B. In the literature, one of the most popular prior models is the Gaussian processes (GPs), and the most recent class of acquisition functions that demonstrate state-of-the-art empirical performance is the information-theoretic ones [7,8,9,41].

3.2.1. MaxEnt for Information-Theoretic Bayesian Optimisation

Information-theoretic BO methods select the next query point that maximises information gain about the unknown global optimiser/optimum. Despite their impressive performance, the computation of their acquisition functions involves an intractable term of Gaussian mixture entropy, as shown in Equation (22), if we perform a fully Bayesian treatment for the GP hyperparameters. Accurate approximation of this Gaussian mixture entropy term is crucial for the performance of information-theoretic BO, but can be difficult and/or expensive. In this paper, we propose an efficient approximation of the Gaussian mixture entropy by using MEMe, which allows for efficient computation of the information-theoretic acquisition functions.

As an concrete example, we consider the application of MEMe for Fast Information-Theoretic Bayesian Optimisation (FITBO) [9] as described in Algorithm 4, which is a highly practical information-based BO method proposed recently. The FITBO acquisition function has the following form:

$$\alpha (\mathbf{x}|{\mathcal{D}}_t)=H\left[\frac{1}{M}\sum _j^{M}p(y|{\mathcal{I}}^{(j)})\right]-\frac{1}{M}\sum _j^{M}H\left[p(y|{\mathcal{I}}^{(j)})\right],$$

(22)

where $p(y|{\mathcal{I}}^{(j)})=p(y|{\mathcal{D}}_t,\mathbf{x},{\theta }^{(j)})$ is the predictive posterior distribution for y conditioned on the observed data ${\mathcal{D}}_t$, the test location $\mathbf{x}$, and a hyperparameter sample ${\theta }^{(j)}$. The first entropy term is the entropy of a Gaussian mixture, where M is the number of Gaussian components.

The entropy of a Gaussian mixture does not have a closed-form solution and needs to be approximated. In FITBO, the authors approximate the quantity using numerical quadrature and moment matching (This corresponds to the maximum entropy solution for two moment constraints as well as the normalization constraint.). In this paper, we develop an effective analytic upper bound of the Gaussian mixture entropy using MEMe, which is derived from the non-negative relative entropy between the true density $p(x)$ and the MaxEnt distribution $q(x)$ [29]:

$${\mathcal{D}}_{\mathrm{KL}}(p||q)=-H(p)+H(q)\ge 0,$$

(23)

hence

$$H(p)\le H(q).$$

(24)

Notice that $q(x)$ shares the same moment constraints as $p(x)$; furthermore, the entropy of the MaxEnt distribution $q(x)$ can be derived analytically:

$$H(q)=1+\sum _{i=0}^m{\alpha }_i{\mu }_i,$$

(25)

where $\left\{{\mu }_i\right\}$ are the moments of a Gaussian mixture, which can be computed analytically using Equation (10), and $\left\{{\alpha }_i\right\}$ are the Lagrange multipliers that can be obtained by applying Algorithm 2. The overall algorithm for approximating the Gaussian mixture entropy is then summarised in Algorithm 5. In Appendix B.1, we also prove that the moments of the Gaussian mixture uniquely determine its density, and the bound becomes tighter with every extra moment constraint: in the $m\to \infty$ limit, the entropy of the MaxEnt distribution converges to the true Gaussian mixture entropy. Hence, the use of a moment-based MaxEnt approximation can be justified [3].

Algorithm 4 MEMe for FITBO

1: Input: Observed data $D_t$ 2: Output: Acquisition function ${\alpha }_n(\mathbf{x}|D_t)$ 3: Sample M hyperparameters $\left\{{\mathit{\theta }}^{(j)}\right\}$ 4: for$j=1,\cdots ,M$do 5:  Approximately compute $p(y|{\mathcal{D}}_t,\mathbf{x},{\mathit{\theta }}^{(j)})=\mathcal{N}(y;m_j,K_j)$ and its entropy $H\left[p(y|{\mathcal{D}}_t,\mathbf{x},{\mathit{\theta }}^{(j)})\right]$ 6: end for 7: Approximate $H\left[\frac{1}{M}{\sum }_j^M\mathcal{N}(y;m_j,K_j)\right]$ with MEMe following Algorithm 5 8: Compute $\alpha (\mathbf{x}|D_t)$ as in Equation (22)

Algorithm 5 MEMe for Approximating Gaussian Mixture Entropy

1: Input: A univariate Gaussian mixture $\mathrm{GM}=\frac{1}{M}{\sum }_j^M\mathcal{N}(y;m_j,{\sigma }_j^2)$ with mean $m_j$ and variance ${\sigma }_j^2$ 2: Output:$H_{\mathrm{GM}}\approx H\left[\frac{1}{M}{\sum }_j^M\mathcal{N}(y;m_j,{\sigma }_j^2)\right]$ 3: Compute the moments of GM, $\left\{{\mu }_i\right\}$, analytically using Equation (10) 4: Compute the Lagrange multipliers, $\left\{{\alpha }_i\right\}$, using Algorithm 2 5: $H_{\mathrm{GM}}\approx -(1+{\sum }_i{\alpha }_i\mathbb{E}[y^i])=-(1+{\sum }_i{\alpha }_i{\mu }_i)$

3.2.2. Experiments

Entropy of the Gaussian Mixture in Bayesian Optimisation

We first test a diverse set of methods to approximate the entropy of two sets of Gaussian mixtures, which are generated using FITBO with 200 hyperparameter samples and 400 hyperparameter samples on a 2D problem. Specifically, we compare the following approaches for entropy approximation: MaxEnt methods using 10 and 30 power moments (MEMe-10 and MEMe-30), MaxEnt methods using 10 and 30 Legendre moments (MEMeL-10 and MEMeL-30), variational upper bound (VUB) [42], method proposed in [43] with 2 Taylor expansion terms (Huber-2), Monte Carlo with 100 samples (MC-100), and simple moment matching (MM).

We evaluate the performance in terms of the approximation error, i.e., the relative error between the approximated entropy value and the true entropy value, the latter of which is estimated via expensive numerical integration. The results of the mean approximation errors by all methods over 80 different Gaussian mixtures are shown in

Table 2. Mean fractional error in approximating the entropy of the mixture of M Gaussians using various methods.

Methods	$\mathit{M}=200$	$\mathit{M}=400$
MEMe-10	$1.24\times {10}^{-2}$	$1.38\times {10}^{-2}$
MEMe-30	$1.13\times {10}^{-2}$	$1.06\times {10}^{-2}$
MEMeL-10	$1.01\times {10}^{-2}$	$0.85\times {10}^{-2}$
MEMeL-30	$\mathbf{0.50}\times {\mathbf{10}}^{-\mathbf{2}}$	$\mathbf{0.36}\times {\mathbf{10}}^{-\mathbf{2}}$
VUB	$22.0\times {10}^{-2}$	$29.1\times {10}^{-2}$
Huber-2	$24.7\times {10}^{-2}$	$35.5\times {10}^{-2}$
MC-100	$1.60\times {10}^{-2}$	$2.72\times {10}^{-2}$
MM	$2.78\times {10}^{-2}$	$3.22\times {10}^{-2}$

(The version with the standard deviation of errors is presented as

Table A1. Fractional error in approximating the entropy of Gaussian mixtures using various methods.

Methods	M = 200	M = 400
MEMe-10	$1.24\times {10}^{-2}$	$1.38\times {10}^{-2}$
	($\pm 4.12\times {10}^{-2}$)	($\pm 5.46\times {10}^{-2}$)
MEMe-30	$\mathbf{1.13}\times {\mathbf{10}}^{-\mathbf{2}}$	$\mathbf{1.06}\times {\mathbf{10}}^{-\mathbf{2}}$
	($\pm 3.68\times {10}^{-2}$)	($\pm 4.47\times {10}^{-2}$)
MEMeL-10	$\mathbf{1.01}\times {\mathbf{10}}^{-\mathbf{2}}$	$\mathbf{0.85}\times {\mathbf{10}}^{-\mathbf{2}}$
	($\pm 3.68\times {10}^{-2}$)	($\pm 3.81\times {10}^{-2}$)
MEMeL-30	$\mathbf{0.50}\times {\mathbf{10}}^{-\mathbf{2}}$	$\mathbf{0.36}\times {\mathbf{10}}^{-\mathbf{2}}$
	($\pm 2.05\times {10}^{-2}$)	($\pm 1.62\times {10}^{-2}$)
Variational	$22.0\times {10}^{-2}$	$29.1\times {10}^{-2}$
Upper Bound	($\pm 28.0\times {10}^{-2}$)	($\pm 78.6\times {10}^{-2}$)
Huber-2	$24.7\times {10}^{-2}$	$35.5\times {10}^{-2}$
	($\pm 46.1\times {10}^{-2}$)	($\pm 130.4\times {10}^{-2}$)
MC-100	$1.60\times {10}^{-2}$	$2.72\times {10}^{-2}$
	($\pm 3.80\times {10}^{-2}$)	($\pm 11.9\times {10}^{-2}$)
Moment	$2.78\times {10}^{-2}$	$3.22\times {10}^{-2}$
Matching	($\pm 4.85\times {10}^{-2}$)	($\pm 7.94\times {10}^{-2}$)

in Appendix B.2). We can see clearly that all MEMe approaches outperform other methods in terms of the approximation error. Among the MEMe approaches, the use of Legendre moments, which apply orthogonal Legendre polynomial bases, outperforms the use of simple power moments.

The mean runtime taken by all approximation methods over 80 different Gaussian mixtures are shown in

Table 3. Mean runtime of approximating the entropy of the mixture of M Gaussians using various methods.

Methods	$\mathbf{M}=200$	$\mathbf{M}=400$
MEMe-10	$1.38\times {10}^{-2}$	$1.48\times {10}^{-2}$
MEMe-30	$2.59\times {10}^{-2}$	$3.21\times {10}^{-2}$
MEMeL-10	$1.70\times {10}^{-2}$	$1.75\times {10}^{-2}$
MEMeL-30	$4.18\times {10}^{-2}$	$4.66\times {10}^{-2}$
VUB	$12.9\times {10}^{-2}$	$50.7\times {10}^{-2}$
Huber-2	$20.9\times {10}^{-2}$	$82.2\times {10}^{-2}$
MC-100	$10.6\times {10}^{-2}$	$40.1\times {10}^{-2}$
MM	$\mathbf{2.71}\times {\mathbf{10}}^{-\mathbf{5}}$	$\mathbf{2.87}\times {\mathbf{10}}^{-\mathbf{5}}$

(The version with the standard deviation is presented as

Table A2. Runtime of approximating the entropy of Gaussian mixtures using various methods.

Methods	M = 200	M = 400
MEMe-10	$\mathbf{1.38}\times {\mathbf{10}}^{-\mathbf{2}}$	$\mathbf{1.48}\times {\mathbf{10}}^{-\mathbf{2}}$
	($\pm 2.59\times {10}^{-3}$)	($\pm 2.22\times {10}^{-3}$)
MEMe-30	$2.59\times {10}^{-2}$	$3.21\times {10}^{-2}$
	($\pm 4.68\times {10}^{-3}$)	($\pm 5.17\times {10}^{-3}$)
MEMeL-10	$\mathbf{1.70}\times {\mathbf{10}}^{-\mathbf{2}}$	$\mathbf{1.75}\times {\mathbf{10}}^{-\mathbf{2}}$
	($\pm 3.00\times {10}^{-3}$)	($\pm 3.17\times {10}^{-3}$)
MEMeL-30	$4.18\times {10}^{-2}$	$4.66\times {10}^{-2}$
	($\pm 5.63\times {10}^{-3}$)	($\pm 5.00\times {10}^{-3}$)
Variational	$12.9\times {10}^{-2}$	$50.7\times {10}^{-2}$
Upper Bound	($\pm 8.22\times {10}^{-3}$)	($\pm 10.4\times {10}^{-3}$)
Huber-2	$20.9\times {10}^{-2}$	$82.2\times {10}^{-2}$
	($\pm 7.54\times {10}^{-3}$)	($\pm 14.8\times {10}^{-3}$)
MC-100	$10.6\times {10}^{-2}$	$40.1\times {10}^{-2}$
	($\pm 6.19\times {10}^{-3}$)	($\pm 17.8\times {10}^{-3}$)
Moment	$\mathbf{2.71}\times {\mathbf{10}}^{-\mathbf{5}}$	$\mathbf{2.87}\times {\mathbf{10}}^{-\mathbf{5}}$
Matching	($\pm 1.16\times {10}^{-4}$)	($\pm 1.19\times {10}^{-4}$)

in Appendix B.2). To ensure a fair comparison, all the methods are implemented in MATLAB and all the timing tests are performed on a 2.3 GHz Intel Core i5 processor. As we expect, the moment matching technique, which enables us to obtain an analytic approximation for the Gaussian mixture entropy, is the fastest method. MEMe approaches are significantly faster than the rest of approximation methods. This demonstrates that MEMe approaches are highly efficient in terms of both approximation accuracy and computational speed. Among all the MEMe approaches, we choose to apply MaxEnt with 10 Legendre moments in the BO for the next set of experiments, as it is able to achieve lower approximation error than MaxEnt with higher power moments while preserving the computational benefit of FITBO.

Information-Theoretic Bayesian Optimisation

We now test the effectiveness of MEMe for information-theoretic BO. We first illustrate the entropy approximation performance of MEMe using a 1D example. In Figure 2, the top plot shows the objective function we want to optimise (red dash line) and the posterior distribution of our surrogate model (blue solid line and shaded area). The bottom plot shows the acquisition functions computed based on Equation (22) using the same surrogate model but three different methods for Gaussian mixture entropy approximation, i.e., expensive numerical quadrature or Quad (red solid line), MM (black dash line), and MEMe using 10 Legendre moments (green dash line). In BO, the next query location is obtained by maximising the acquisition function, therefore the location instead of the magnitude of the modes of the acquisition function matters most. We can see from the bottom plot that MEMeL-10 results in an approximation that well matches the true acquisition function obtained by Quad. As a result, MEMeL-10 manages to recommend the same next query location as Quad. In comparison, the loose upper bound of the MM method, though successfully capturing the locations of the peak values, fails to correctly predict the global maximiser of the true acquisition function. MM therefore recommends a query location that is different from the optimal choice. As previously mentioned, the acquisition function in information-theoretic BO represents the information gain about the global optimum by querying at a new location. The sub-optimal choice of the next query location thus imposes a penalty on the optimisation performance as seen in Figure 3.

In the next set of experiments, we evaluate the optimisation performance of three versions of FITBO that use different approximation methods. Specifically, FITBO-Quad denotes the version that uses expensive numerical quadrature to approximate the entropy of the Gaussian mixture, FITBO-MM denotes the one using simple moment matching, and FITBO-MEMeL denotes the one using MEMe with 10 Legendre moments. We test these BO algorithms on two challenging optimisation problems, i.e., the Michalewicz-5D function [44] and the Hartmann-6D function [45], and measure the optimisation performance in terms of the immediate regret (IR): $\mathrm{IR}=|f^{\ast }-\widehat{f}|$, which measures the absolute difference between the true global minimum value $f^{\ast }$ and the best guess of the global minimum value $\widehat{f}$ by the BO algorithm. The average (median) result over 10 random initialisations for each experiment is shown in Figure 3. It is evident that the MEMe approach (FITBO-MEMeL-10), which better approximates the Gaussian mixture entropy, leads to a superior performance of the BO algorithm compared to the BO algorithm using simple moment matching technique (FITBO-MM).

4. Conclusions

In this paper, we established the equivalence between the method of maximum entropy and Bayesian variational inference under moment constraints, and proposed a novel maximum entropy algorithm (MEMe) that is stable and consistent for a large number of moments. We apply MEMe in two applications, i.e., log determinant estimation and Bayesian optimisation, to demonstrate its effectiveness and superiority over state-of-the-art approaches. The proposed algorithm can further benefit a wide range of large-scale machine learning applications where efficient approximation is of crucial importance.