Variational Bayesian Approximation (VBA): Implementation and Comparison of Different Optimization Algorithms

Abstract

In any Bayesian computations, the first step is to derive the joint distribution of all the unknown variables given the observed data. Then, we have to do the computations. There are four general methods for performing computations: Joint MAP optimization; Posterior expectation computations that require integration methods; Sampling-based methods, such as MCMC, slice sampling, nested sampling, etc., for generating samples and numerically computing expectations; and finally, Variational Bayesian Approximation (VBA). In this last method, which is the focus of this paper, the objective is to search for an approximation for the joint posterior with a simpler one that allows for analytical computations. The main tool in VBA is to use the Kullback–Leibler Divergence (KLD) as a criterion to obtain that approximation. Even if, theoretically, this can be conducted formally, for practical reasons, we consider the case where the joint distribution is in the exponential family, and so is its approximation. In this case, the KLD becomes a function of the usual parameters or the natural parameters of the exponential family, where the problem becomes parametric optimization. Thus, we compare four optimization algorithms: general alternate functional optimization; parametric gradient-based with the normal and natural parameters; and the natural gradient algorithm. We then study their relative performances on three examples to demonstrate the implementation of each algorithm and their efficiency performance.

1. Introduction

The Bayesian inference starts by defining the joint posterior distribution of all the unknowns in the models given the data. Then, to obtain a point estimation, there are two main solutions: Joint Maximum A Posterior (JMAP), and computation of the posterior expectations. JMAP requires optimization to determine the unknown parameters that maximize the joint posterior distribution. Computing posterior expectations requires integration and involves calculating the expected values of unknown parameters. These computations are possible analytically in the simple case of linear and Normal priors or for exponential families with conjugate priors. In all other cases, the computation of multivariate integration is too costly. Then, there are two main tools. First, sampling methods such as Markov Chain Monte Carlo (MCMC), slice sampling, nested sampling, etc., where we generate samples and then compute means, variances, covariances, etc. Second, analytic approximation methods, which first approximate the original posterior with a simpler one to facilitate analytical computations. The Variational Bayesian Approximation (VBA) is the main method of this kind, using the KLD divergence as the criterion to obtain such an approximation. However, even if theoretically, the functional optimization of KLD is possible, it still needs integral computations, which are possible if we have some conjugacy properties. In particular, when the original posterior expression is in the exponential families, the approximation expression is also in the exponential families thanks to the KLD properties. In this paper, we consider this case and accomplish the optimization task in various ways. The first approach involves directly applying alternate optimization to the KLD criterion. Other methods consist of expressing the KLD as a function of parameters and using gradient-based algorithms. As the exponential families can be represented using normal or natural parameters, we obtain different algorithms that we compare.

For a general Bayesian inference, the main computation tool is the exploration of the joint posterior law via the sampling methods, such as the MCMC, or other more recent techniques such as slice sampling [1] or nested sampling [2]. However, these methods become very costly for high dimensional problems in Computer Vision, Machine Learning, and Artificial Intelligence. Blei et al. (2017) [3] mentioned in their review article the alternative of MCMC with the support of variational inference. They found Bayesian inference fast and easy to scale up to large data sets. Variational inference has less complexity than MCMC in many examples, such as large-scale document analysis, computational neuroscience, and Computer Vision.

The VBA is constructed for a variety of models to approximate a posterior distribution with many variables whose aim is to minimize the computational costs while keeping controllable accuracy. The main reduction in computational cost is achieved by searching for a good and tractable approximation for the joint posterior. VBA has acceptable efficiency computational costs compared with sampling methods.

Parisi and Shankar [4] have been a pioneer in VBA since 1988, and after that MacKay and Neal [5,6] were forerunners in Bayesian Neural Network. Šmídl and Quinn [7] provided a theoretical background of VBA and its iterative algorithms applied in signal processing. Sarkka and Nummenmaa [8] used VBA on an autoregressive model with unknown variance, which is an application of VBA in signal processing. Zheng et al. [9] introduced a new version of VBA for image restoration concerning an ill-posed linear inverse problem applying all variation priors. Their method structure was on the memory transposition subspace algorithm for a probability density function space. Many other authors used this approach for different applications [7,10,11,12,13]. Renard et al. [14] worked on a Bayesian inference application for regional frequency analysis in a nonstationary context to handle intricate models getting physical reality and statistical necessities. Li and Shi [15] explained the matter of the model accuracy in the frameworks of wind power. Bayesian approaches have many advantages in the uncertainty and variability estimations, which are crucial in model prediction. They mentioned Bayesian methods being appropriate for various aspects of wind energy conversion systems, such as enhancing the precision and reliability of wind resource estimation and short-term predictions. Bayesian models have applications on enormous data of microarray technology, which are challenging in making inferences from such massive data sets. Yang et al. [16] introduced Bayesian methods as popular techniques for their benefits in microarray analysis.

In many applications, with indirect observations, such as using a thermometer sensor to measure the temperature (hidden variable), Bayesian inference starts with obtaining the expression of the joint distribution of all the unknown variables (hidden variables and parameters of the model) given the observed data. Then, we have to use it to do inference. In general, this expression is not separable in all its variables. So, the computations become hard and costly. For example, obtaining the marginals for each variable and computing the expectations are difficult and expensive. This problem becomes even more crucial in high-dimensional Computer Vision and Machine Learning and becomes an issue in inverse problems. We may then need to propose a surrogate expression with which we can do approximate computations.

The VBA is a technique that approximates the joint distribution p with an easier method; for example, a separable one q by minimizing $KL(q|p)$, which makes the marginal computations much easier. For example, in the case of two variables, $p(x,y)$ is approximated by $q(x,y)=q_1\left(x\right)q_2\left(y\right)$ via minimizing $KL(q_1{q}_2|p)$. When q is separable in all the variables of p, the approximation is also called Mean Field Approximation (MFA).

To obtain the approximate marginals $q_1$ and $q_2$, we must minimize $KL(q_1{q}_2|p)$. A first standard and general algorithm is the alternate analytic optimization of $KL(q_1{q}_2|p)$ with respect to $q_1$, and then, to $q_2$. Finding the expression of the functional derivatives of $KL(q_1{q}_2|p)$ for $q_1$ and $q_2$ and equating them to zero alternatively, we obtain an iterative optimization algorithm. A second general approach is its optimization in the Riemannian Manifold, where we consider the case p is in the exponential family and so are $q_1$ and $q_2$. For this case, $KL(q_1{q}_2|p)$ becomes a function of the parameters $\mathit{\theta }$ of the exponential family. Then, we can use any other optimization algorithm to obtain those parameters.

In this paper, we review the VBA technique and then compare four different VBA optimization algorithms in three examples: standard alternate analytic optimization, a gradient-based algorithm for normal and natural parameter space [17,18] and a natural gradient algorithm [19,20,21] The paper aims to consider the first algorithm as our principal method and compare it with the three other algorithms. Building upon our initial proceedings paper [22], presented at the 41st International Workshop, on Bayesian inference and maximum entropy methods in 2022, this journal article further investigates more theoretical details, algorithms, and comparison study.

In this article, we consider the case of exponential families, so that p and q are both in exponential families. Then, we write the expression of the $KL(q|p)$ and explore four different estimation algorithms for unknown parameters in a model that incorporates prior information. The first iterative algorithm handles directly $KL(·)$, alternatively optimizing it with respect to each marginal $q_i$. This is the standard optimization used often in VBA. The function to be optimized is KLD. First, the gradient of KLD for all unknown parameters needs to be found. Then, we can begin with initial values for the parameters, either estimated from data or chosen deliberately. Then, we repeat the iterative algorithm until it converges to some points. If we denote the unknown normal parameter space with $\mathit{\theta }$, then the iterative optimization algorithm can be written as: ${\tilde{\mathit{\theta }}}^{(k+1)}={\tilde{\mathit{\theta }}}^{\left(k\right)}-\gamma \nabla KL\left({\tilde{\mathit{\theta }}}^{\left(k\right)}\right)$ for gradient-based algorithm and the same with the natural parameters ${\tilde{\mathsf{\Lambda }}}^{(k+1)}={\tilde{\mathsf{\Lambda }}}^{\left(k\right)}-\gamma \nabla KL\left({\tilde{\mathsf{\Lambda }}}^{\left(k\right)}\right)$ with different values of $\gamma$. The natural gradient definition, mentioned by Amari [23], is:

$$\tilde{\nabla }h=F^{-1}\nabla h,$$

(1)

where F and h are the Fisher information matrix and objective function, respectively. In our case, h is the KLD. To make Fisher’s formula understandable, we change $p(x,y)$ to $p(x,y|\mathit{\theta })$ to explicitly show the parameters $p(x,y)$, and differentiate for $\mathit{\theta }$. The Fisher information matrix of $p(x,y|\mathit{\theta })$ is given by:

$$\begin{array}{cc}\hfill F=& {\langle \nabla \ln p(x,y|\mathit{\theta })\nabla \ln p{(x,y|\mathit{\theta })}^{⊺}\rangle }_{p(x,y)}\hfill \\ \hfill =& -{\langle H_{\ln p(x,y|\mathit{\theta })}\rangle }_{p(x,y)},\hfill \end{array}$$

(2)

where $H_{\ln p(x,y|\mathit{\theta })}$ and ${\langle ·\rangle }_{p(x,y)}$ are the Hessian matrix of $\ln p(x,y|\mathit{\theta })$ concerning $\mathit{\theta }$ and the expectation respect to $p(x,y)$ distribution, respectively. An approximation of the Fisher information, introduced by Schraudolph [24], is called Empirical Fisher:

$$\begin{array}{cc}\hfill \overline{F}=& {\langle \nabla \ln p(x,y|\mathit{\theta })\nabla \ln p{(x,y|\mathit{\theta })}^{⊺}\rangle }_{q(x,y)}\hfill \\ \hfill =& {\displaystyle \frac{1}{\parallel {\mathbb{S}}_{XY}\parallel }}{\displaystyle \sum _{(x,y)\in {\mathbb{S}}_{XY}}}\nabla \ln p(y|x,\mathit{\theta })\nabla \ln p{(y|x,\mathit{\theta })}^{⊺}.\hfill \end{array}$$

The fundamental natural gradient iteration algorithm is:

$${\tilde{\mathit{\theta }}}^{(k+1)}={\tilde{\mathit{\theta }}}^{\left(k\right)}-{\rho }_k\tilde{\nabla }h,$$

(3)

where ${\rho }_k$ is a learning-rate schedule. Martens [25] suggested an optimum update based on a particular second-order local approximation of h, ${\rho }_k=1$.

We consider three examples: Normal-Inverse-Gamma, multivariate Normal, and linear inverse problems to assess the performance and convergence speed of the algorithms through multiple simulations. We propose the following organization of this paper: In Section 2, we present a brief explanation of the basic VBA analytical alternate optimization algorithm. In Section 3, we illustrate our first example related to Normal-Inverse-Gamma distribution analytically and, in practice, explain the outcomes of four algorithms to estimate the unknown parameters. In Section 4, we study a more complex example of a multivariate Normal distribution whose means and variance-covariance matrix are unknown and have a Normal-Inverse-Wishart distribution. This section aims to demonstrate marginal distributions of $\tilde{\mathit{\mu }}$ and $\tilde{\mathsf{\Sigma }}$ using a set of multivariate Normal observations using these mean and variance. In Section 5, the example is closest to realistic situations and is a linear inverse problem. We simulate the model with different dimensions to see the changes in the performance of the algorithms. In Section 6, we present our work summary in the article and compare the four recursive algorithms in three different examples.

2. Variational Bayesian Approach (VBA)

In this paper, our focus is on MFA as a subset of VBA. In MFA, the posterior distribution $p(x,y)$ is approximated by separable $q(x,y)$:

$$q(x,y)=q_1\left(x\right)q_2\left(y\right).$$

$KL(q|p)$ [26] is an information measure of discrepancy between two probability functions and one of the most likely used information for divergence and separation measurement and disparity of two density functions [27]. One advantage of $KL(q|p)$ is its effectiveness in solving distributionally robust optimization problems [28]. Despite its computational and theoretical benefits, $KL(q|p)$ faces certain challenges. These include asymmetries, which complicate optimal model selection [27], and the analytical complexity of $KL(q|p)$ when comparing normal mixture models, along with a lack of practical computational algorithms [29]. Let $p\left(x\right)$ and $q\left(x\right)$ be two density functions of a continuous random variable x for support set ${\mathbb{S}}_X$. $KL(q|p)$ function is introduced as:

$$KL(q|p)={\int }_{x\in {\mathbb{S}}_X}q\left(x\right)\ln {\displaystyle \frac{q\left(x\right)}{p\left(x\right)}}\underset{·}{\mathrm{x}}.$$

(4)

The basic structure of VBA is to minimize $KL(q|p)$ and find an estimation for p with the density factorial over all variables, consisting of state, hidden, and all unknown parameters. For instance, x and y represent the state and hidden variables, respectively. The unknown parameters refer to the parameters of the prior distributions of x and y.

For simplicity, we assume a bivariate case of distribution $p(x,y)$, and want to assess it via the alternative optimization, then we have:

$$KL(q|p)=-H\left(q_1\right)-H\left(q_2\right)-{\langle \ln p(x,y)\rangle }_{q_1{q}_2},$$

(5)

where:

$$H\left(q_1\right)=-{\int }_{x\in {\mathbb{S}}_X}{q}_1\left(x\right)\ln q_1\left(x\right)\underset{·}{\mathrm{x}}\mathrm{and}H\left(q_2\right)=-{\int }_{y\in {\mathbb{S}}_Y}{q}_2\left(y\right)\ln q_2\left(y\right)\underset{·}{\mathrm{y}},$$

are, respectively, the Shannon entropies of x and of y, and:

$${\langle \ln p(x,y)\rangle }_{q_1{q}_2}=\int {\int }_{(x,y)\in {\mathbb{S}}_{XY}}{q}_1\left(x\right)q_2\left(y\right)\ln p(x,y)\underset{·}{\mathrm{x}}\underset{·}{\mathrm{y}}.$$

$H\left(q_1\right)$ and $H\left(q_2\right)$ are fixed term, so the minimization is only on ${\langle \ln p(x,y)\rangle }_{q_1{q}_2}$. Now, differentiating the Equation (5) with respect to $q_1$, and then with respect to $q_2$ and equating them to zero, we obtain:

$$q_1\left(x\right)\propto exp\left\{{\langle \ln p(x,y)\rangle }_{q_2\left(y\right)}\right\}\mathrm{and}{q}_2\left(y\right)\propto exp\left\{{\langle \ln p(x,y)\rangle }_{q_1\left(x\right)}\right\}.$$

(6)

These results can be easily extended to more dimensions [10]. They do not have any closed form, because they depend on the expression of $p(x,y)$ and those of $q_1$ and $q_2$. An interesting case is the case of exponential families and conjugate priors, where writing:

$$p(x,y)=p(x|y)p\left(y\right), \mathrm{and}p(y|x)={\displaystyle \frac{p(x,y)}{p\left(x\right)}}={\displaystyle \frac{p(x|y\left)p\right(y)}{p\left(x\right)}},$$

(7)

we can consider $p\left(y\right)$ as prior, $p(x|y)$ as the likelihood, and $p(y|x)$ as the posterior distributions. We know that, if $p(x,y)$ is in the exponential families, $q_1\left(x\right)$ and $q_2\left(y\right)$ are also in the exponential families, and $q_1\left(x\right)$ is conjugate to $p(y|x)$ and $q_2\left(y\right)$ is conjugate to $p(x|y)$. To illustrate all these properties, we give details of these expressions for a first simple example of Normal-Inverse-Gamma $p(x,y)=\mathcal{N}(x|\mu ,y\left)\mathcal{I}\mathcal{G}\right(y|\alpha ,\beta )$ with $q_1\left(x\right)=\mathcal{N}(x|{\mu }^{\prime },v)$ and $q_2=\mathcal{I}\mathcal{G}(y|\alpha ,\beta )$. For this simple case, first we give the expression of $KL(q|p)$ with $q_1\left(x\right)=\mathcal{N}(x|{\tilde{\mu }}^{\prime },\tilde{v})$ and $q_2\left(y\right)=\mathcal{I}\mathcal{G}(y|\tilde{\alpha },\tilde{\beta })$ as a function of the parameters $\mathit{\theta }=({\tilde{\mu }}^{\prime },\tilde{v},\tilde{\alpha },\tilde{\beta })$ and then the expressions of the four above-mentioned algorithms and we study their convergence.

For a numerical comparison, we start by generating $n=100$ samples from $p(x,y)=\mathcal{N}(x|0,y\left)\mathcal{I}\mathcal{G}\right(y|3,1)$, so we know the true parameters $(\mu =0,\alpha =3,\beta =1)$. Then, for different initializations of $\mathit{\theta }=({\tilde{\mu }}^{\prime },\tilde{v},\tilde{\alpha },\tilde{\beta })$, we run the four algorithms and compare the results. The final resulting margin, which is in this case the Student-t $S(x|\mu ,\alpha ,\beta )=\int \mathcal{N}(x|\mu ,y)\mathcal{I}\mathcal{G}(y|\alpha ,\beta )\underset{·}{\mathrm{y}}$. We can then compare the true $S(x|\mu ,\alpha ,\beta )$ with the obtained $S(x|\widehat{\mu },\widehat{\alpha },\widehat{\beta })$ as well as with $q(x,y|\widehat{\mu },\widehat{v},\widehat{\alpha },\widehat{\beta })=\mathcal{N}(x|\widehat{\mu },\widehat{v})\mathcal{I}\mathcal{G}(y|\widehat{\alpha },\widehat{\beta })$. As we see in the proceeding of the paper, the initialization is very important. When we have samples of $(x,y)$, we can try to initialize the parameters by their empirical values obtained by simple classical methods such as the method of moments. Another issue is the stopping criteria, which can be based either on the KLD criterion during successive iterations or any distances between the parameters at successive iterations.

3. Normal-Inverse-Gamma Distribution Example

The purpose of this section is to explain in detail the process of performing calculations in the alternative analytic algorithm. For this, we consider a simple case Normal-Inverse-Gamma Distribution for which, we have all the necessary expressions. This distribution has many applications in fields, like finance, econometrics, engineering, and Machine Learning. The objective here is to compare the four different algorithms mentioned earlier. The practical problem considered here is the following:

Suppose the data as $\mathit{z}=z_1,...z_N$ and model it by $Z=X+E$, where $E\sim \mathcal{N}(0,v)$ and so $Z|X\sim \mathcal{N}(X,v)$. Thus, we have $X\sim \mathcal{N}(\mu ,Y)$ and $Y\sim \mathcal{I}\mathcal{G}(\alpha ,\beta )$, then:

$$p(x,y|\mathit{z})={\displaystyle \frac{p(\mathit{z}|x,y\left)p\right(x,y)}{p\left(\mathit{z}\right)}}={\displaystyle \frac{p(\mathit{z}|x\left)p\right(x|y)p\left(y\right)}{p\left(\mathit{z}\right)}}.$$

Putting all the elements, we see that $p(x,y|\mathit{z})$ is a $\mathcal{N}\mathcal{I}\mathcal{G}$ model and not separable in x and y. We like to approximate it with a separable one $q(x,y)=q_1\left(x\right)q_2\left(y\right)$. However, in this simple case, assume we have a sensor, which delivers a physical quantity X, N times, $\mathbf{x}=\{x_1,x_2,\cdots ,x_N\}$. We want to model these data. In the first step, we model it as $N(x|{\mu }^{\prime },v)$ with fixed ${\mu }^{\prime }$ and v. Then, it is easy to estimate the parameters $({\mu }^{\prime },v)$ either by Maximum Likelihood or Bayesian strategy. If we assume that the model is Normal with unknown variance and call this variance y and assign an $\mathcal{I}\mathcal{G}$ prior to it, then we have a model $\mathcal{N}\mathcal{I}\mathcal{G}$ for $p(x,y)$. The $\mathcal{N}\mathcal{I}\mathcal{G}$ priors have been applied to the wavelet context with correlated structures because they are conjugated with Normal priors [30]. We choose Normal-Inverse-Gamma distribution because of this conjugated property and ease of handling.

We showed that the margins are $St$ and $\mathcal{I}\mathcal{G}$. Working directly with $St$ is difficult. So, we want to approximate it with a Normal $q_1\left(x\right)$. This is equivalent to approximating $p(x,y)$ with $q_1\left(x\right)q_2\left(y\right)$. Now, we want to find the parameters ${\mu }^{\prime }$, v, $\alpha$, and $\beta$, which minimize $KL(q_1{q}_2|p)$. This process is called MFVBA. Then, we compare four algorithms to obtain the parameters, which minimize $KL(q_1{q}_2|p)$. $KL(q_1{q}_2|p)$ is convex with respect to $q_1$ if $q_2$ is fixed and is convex with respect to $q_2$ if $q_1$ is fixed. So, we hope that the iterative algorithm converges. However, $KL(q_1{q}_2|p)$ may not be convex in the space of parameters. So, we have to study the shape of this criterion concerning the parameters ${\tilde{\mu }}^{\prime }$, $\tilde{v}$, $\tilde{\alpha }$, and $\tilde{\beta }$.

We want to find $p\left(x\right)$. For this process, we assume a simple Normal model, but with unknown variance y. So that, the forward model can be written as $p(x,y)=\mathcal{N}(x|\mu ,y\left)\mathcal{I}\mathcal{G}\right(y|\alpha ,\beta )$. In this simple example, we know that $p\left(x\right)$ is a Student-t distribution obtained by:

$$S(x|\mu ,\alpha ,\beta )=\int \mathcal{N}(x|\mu ,y)\mathcal{I}\mathcal{G}(y|\alpha ,\beta )\underset{·}{\mathrm{y}}.$$

(8)

We approximate three parameters $\mathit{\theta }=(\mu ,\alpha ,\beta )$ from the data $\mathbf{x}$ and find an approximated distribution $q\left(x\right)$ for $p\left(x\right)$.

The main idea is to find such $q_1\left(x\right)q_2\left(y\right)$ as an approximation of $p(x,y)$. Here, we show the standard alternative analytic optimization, step by step. For this, we start by choosing the conjugate families $q_1\left(x\right)=\mathcal{N}(x|{\tilde{\mu }}^{\prime },\tilde{v})$ and $q_2\left(y\right)=\mathcal{I}\mathcal{G}(y|\tilde{\alpha },\tilde{\beta })$. Note that ${\langle x\rangle }_p=\tilde{\mu }$ and ${\langle x\rangle }_{q_1}={\tilde{\mu }}^{\prime }$.

In the first step, we have to calculate $\ln p(x,y)$ mentioned earlier:

$$\ln p(x,y)=c-{\displaystyle \frac{1}{2}}\ln y-{\displaystyle \frac{1}{2y}}{(x-\tilde{\mu })}^2-(\tilde{\alpha }+{\displaystyle \frac{1}{2}})\ln y-{\displaystyle \frac{\tilde{\beta }}{y}},$$

(9)

where c is a constant value term independent of x and y. First of all, to use the iterative algorithm given in (6), starting by $q_1=N(x|{\tilde{\mu }}^{\prime },\tilde{v})$ we have to find $q_2\left(y\right)$, so we start by finding $q_2\left(y\right)$. The integration of $\ln p(x,y)$ is with respect to $q_1\left(x\right)$:

$${\langle \ln p(x,y)\rangle }_{q_1}=c-{\displaystyle \frac{1}{2y}}{\langle {(x-\tilde{\mu })}^2\rangle }_{q_1}-(\tilde{\alpha }+1)\ln y-{\displaystyle \frac{\tilde{\beta }}{y}}.$$

(10)

Since ${\langle {(x-\tilde{\mu })}^2\rangle }_{q_1}=\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2$:

$$q_2\left(y\right)\propto exp\left[-(\tilde{\alpha }+1)\ln y-({\displaystyle \frac{\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}{2}}+\tilde{\beta }){\displaystyle \frac{1}{y}}\right].$$

(11)

Thus, the function $q_2\left(y\right)$ is equivalent to an inverse gamma distribution $\mathcal{I}\mathcal{G}(\tilde{\alpha },{\displaystyle \frac{\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}{2}}+\tilde{\beta })$. To do so, we only use $q_1$ distribution. We have to take integral of $\ln p(x,y)$ over $q_2$ to find $q_1$:

$${\langle \ln p(x,y)\rangle }_{q_2}=c-(\tilde{\alpha }+1){\langle \ln y\rangle }_{q_2}-(\tilde{\beta }+{\displaystyle \frac{1}{2}}{(x-\tilde{\mu })}^2){\langle {\displaystyle \frac{1}{y}}\rangle }_{q_2}.$$

(12)

Note that the first term does not depend on x and ${\langle {\displaystyle \frac{1}{y}}\rangle }_{q_2}={\displaystyle \frac{2\tilde{\alpha }}{2\tilde{\beta }+\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}}$, so:

$$q_1\left(x\right)\propto exp\left[-{\displaystyle \frac{2\tilde{\alpha }}{2\tilde{\beta }+\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}}(\tilde{\beta }+{\displaystyle \frac{1}{2}}{(x-\tilde{\mu })}^2)\right]\propto exp\left[-{\displaystyle \frac{{(x-\tilde{\mu })}^2}{2\frac{2\tilde{\beta }+\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}{2\tilde{\alpha }}}}\right].$$

(13)

We see that $q_1$ is again a Normal distribution but with updated parameters $\mathcal{N}(\tilde{\mu },{\displaystyle \frac{2\tilde{\beta }+\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}{2\tilde{\alpha }}})$, so $\tilde{v}={\displaystyle \frac{2\tilde{\beta }+\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2}{2\tilde{\alpha }}}$. Note that, we obtained the conjugal property: If $p(x|y)=\mathcal{N}(x|\mu ,y)$ and $p\left(y\right)=\mathcal{I}\mathcal{G}(y|\alpha ,\beta )$, then $p(y|x)=\mathcal{I}\mathcal{G}(y|{\alpha }^{\prime },{\beta }^{\prime })$ where ${\mu }^{\prime }$, ${\alpha }^{\prime }$ and ${\beta }^{\prime }$ are ${\mu }^{\prime }=\mu$, ${\alpha }^{\prime }=\alpha$, ${\beta }^{\prime }=\beta +{\displaystyle \frac{2\beta +v+{(\mu -{\mu }^{\prime })}^2}{2\alpha }}$. In this case, we also know that $p(x|\alpha ,\beta )=St(x|{\mu }^{\prime },\alpha ,\beta )$.

In these calculations, we first calculated the distribution of $q_2$ and then $q_1$. If we do the opposite and first obtain $q_1$ and then $q_2$, the parameter v is eliminated in the iterative calculations and there is no recursive relationship between $\beta$ and v, and it is only necessary to calculate the value of parameter $\beta$.

In standard alternate optimization, there is no need for an iterative process for $\tilde{\mu }$ and $\tilde{\alpha }$, which are approximated by $\tilde{\mu }={\mu }_0$ and $\tilde{\alpha }={\alpha }_0$, respectively. The situation for $\tilde{\beta }$ and $\tilde{v}$ is different because there are circular dependencies among them. So, the approximation needs an iterative process, starting from ${\tilde{\mu }}^{\left(1\right)}={\mu }_0$, ${\tilde{v}}^{\left(1\right)}=v_0$, ${\alpha }^{\left(1\right)}={\alpha }_0$, and ${\beta }^{\left(1\right)}={\beta }_0$.

1. Standard alternate optimization algorithm:

$$\begin{array}{ccc}\hfill {\tilde{\alpha }}^{(k+1)}& =& {\tilde{\alpha }}^{\left(k\right)},\hfill \\ \hfill {\tilde{\beta }}^{(k+1)}& =& {\tilde{\beta }}^{\left(k\right)}+{\displaystyle \frac{{\tilde{v}}^{\left(k\right)}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime \left(k\right)})}^2}{2}},\hfill \\ \hfill {\tilde{\mu }}^{\prime (k+1)}& =& {\tilde{\mu }}^{\prime \left(k\right)},\hfill \\ \hfill {\tilde{v}}^{(k+1)}& =& {\displaystyle \frac{2{\tilde{\beta }}^{(k+1)}+{\tilde{v}}^{\left(k\right)}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime (k+1)})}^2}{2{\tilde{\alpha }}^{(k+1)}}}.\hfill \end{array}$$

This algorithm converges to $\tilde{v}=(2\tilde{\beta }+\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2)/\left(2\tilde{\alpha }\right)$, which gives $\tilde{v}=(2\tilde{\beta }+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2)/(2\tilde{\alpha }-1)$ and $\tilde{\beta }=0$, so $\tilde{v}=0$ that means very strange convergence. That is why this alternate algorithm can not work in this case.

For other algorithms based on normal parameters, such as gradient-based and natural gradient algorithms, it is necessary to find the expression of $KL(q_1{q}_2|p)$ as a function of the normal parameters $\tilde{\mathit{\theta }}=(\tilde{\alpha },\tilde{\beta },{\tilde{\mu }}^{\prime },\tilde{v})$:

$$\begin{array}{cc}\hfill KL\left(\tilde{\mathit{\theta }}\right)\propto & -{\displaystyle \frac{1}{2}}\ln \tilde{v}+{\displaystyle \frac{1}{2}}(\ln \tilde{\beta }-{\psi }_0\left(\tilde{\alpha }\right))+{\displaystyle \frac{\tilde{\alpha }(\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2)}{2\tilde{\beta }}}.\hfill \end{array}$$

(14)

Then, we also need the gradient expression of $\nabla KL\left(\tilde{\mathit{\theta }}\right)$ for $\tilde{\mathit{\theta }}$:

$$\begin{array}{cc}\hfill \nabla KL\left(\tilde{\mathit{\theta }}\right)=& ({\displaystyle \frac{\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2-\tilde{\beta }{\psi }_1\left(\tilde{\alpha }\right)}{2\tilde{\beta }}},{\displaystyle \frac{-\tilde{\alpha }(\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2)+\tilde{\beta }}{2{\tilde{\beta }}^2}},\hfill \\ \hfill & {\displaystyle \frac{\tilde{\alpha }({\tilde{\mu }}^{\prime }-\tilde{\mu })}{\tilde{\beta }}},\left.{\displaystyle \frac{\tilde{\alpha }}{2\tilde{\beta }}}-{\displaystyle \frac{1}{2\tilde{v}}}\right).\hfill \end{array}$$

(15)

The details are available in Appendix A.

2. The gradient-based algorithm with normal parameters:

$$\begin{array}{ccc}\hfill {\tilde{\alpha }}^{(k+1)}& =& {\tilde{\alpha }}^{\left(k\right)}-\gamma \left({\displaystyle \frac{{\tilde{v}}^{\left(k\right)}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime \left(k\right)})}^2-{\tilde{\beta }}^{\left(k\right)}{\psi }_1\left({\tilde{\alpha }}^{\left(k\right)}\right)}{2{\tilde{\beta }}^{\left(k\right)}}}\right),\hfill \\ \hfill {\tilde{\beta }}^{(k+1)}& =& {\tilde{\beta }}^{\left(k\right)}-\gamma \left({\displaystyle \frac{-{\tilde{\alpha }}^{(k+1)}({\tilde{v}}^{\left(k\right)}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime \left(k\right)})}^2)+{\tilde{\beta }}^{\left(k\right)}}{2{\left[{\tilde{\beta }}^{\left(k\right)}\right]}^2}}\right),\hfill \\ \hfill {\tilde{\mu }}^{\prime (k+1)}& =& {\tilde{\mu }}^{\prime \left(k\right)}-\gamma \left({\displaystyle \frac{{\tilde{\alpha }}^{(k+1)}({\tilde{\mu }}^{\prime \left(k\right)}-\tilde{\mu })}{{\tilde{\beta }}^{(k+1)}}}\right),\hfill \\ \hfill {\tilde{v}}^{(k+1)}& =& {\tilde{v}}^{(k+1)}-\gamma \left({\displaystyle \frac{{\tilde{\alpha }}^{(k+1)}}{2{\tilde{\beta }}^{(k+1)}}}-{\displaystyle \frac{1}{2{\tilde{v}}^{\left(k\right)}}}\right),\hfill \end{array}$$

where $\gamma$ is a fixed value for the gradient algorithm. We propose two values for $\gamma$ be equal to 1 and ${\displaystyle \frac{1}{\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right)\parallel }}$. We need to calculate the Fisher information for the third algorithm based on the normal parameters. Then, the natural gradient based on $KL\left(\tilde{\mathit{\theta }}\right)$ is calculated as follows:

$$\begin{array}{cc}\hfill \tilde{\nabla }KL\left(\tilde{\mathit{\theta }}\right)=& \left(-{\displaystyle \frac{1}{2}},-{\displaystyle \frac{1}{2}}(\tilde{v}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime })}^2),-{\displaystyle \frac{\tilde{\alpha }\tilde{v}(\tilde{\mu }-{\tilde{\mu }}^{\prime })}{\tilde{\beta }}},-{\displaystyle \frac{\tilde{v}(\tilde{\beta }-\tilde{\alpha }\tilde{v})}{\tilde{\beta }}}\right).\hfill \end{array}$$

The corresponding algorithm is in the following:

3. The natural gradient algorithm:

$$\begin{array}{ccc}\hfill {\tilde{\alpha }}^{(k+1)}& =& {\tilde{\alpha }}^{\left(k\right)},\hfill \\ \hfill {\tilde{\beta }}^{(k+1)}& =& {\tilde{\beta }}^{\left(k\right)}+{\displaystyle \frac{1}{2}}({\tilde{v}}^{\left(k\right)}+{(\tilde{\mu }-{\tilde{\mu }}^{\prime \left(k\right)})}^2),\hfill \\ \hfill {\tilde{\mu }}^{\prime (k+1)}& =& {\tilde{\mu }}^{\prime \left(k\right)}+{\displaystyle \frac{{\tilde{\alpha }}^{(k+1)}{\tilde{v}}^{\left(k\right)}(\tilde{\mu }-{\tilde{\mu }}^{\prime \left(k\right)})}{{\tilde{\beta }}^{(k+1)}}},\hfill \\ \hfill {\tilde{v}}^{(k+1)}& =& {\tilde{v}}^{\left(k\right)}+{\displaystyle \frac{{\tilde{v}}^{\left(k\right)}({\tilde{\beta }}^{(k+1)}-{\tilde{\alpha }}^{(k+1)}{\tilde{v}}^{\left(k\right)})}{{\tilde{\beta }}^{(k+1)}}}.\hfill \end{array}$$

We consider another sub-algorithm for gradient-based optimization with natural parameters, explained in detail in Appendix A. The algorithm for the last two components produces different results.

We generate $n=100$ samples from the model $p(x,y)=\mathcal{N}(x|1,y\left)\mathcal{I}\mathcal{G}\right(y|3,1)$ for the numerical computations. Thus, we know the exact values of the unknown parameters, just keeping in mind, not used in algorithms. The estimated parameters are in

Table 1. Simulation results for five different models using the Normal-Inverse-Gamma distribution are presented here. We use two initialization point groups. The first column shows the MLE for each parameter. The second column contains some intended initializations that we refer to as evidence-free initialization. They are not based on any information and are optional. It is evident that when we apply data-derived points, the results are highly accurate. However, when we rely on evidence-free points, the results deviate significantly from the true models. Also, we calculate the $KL(·)$ values for the initializations and the final estimations for each method. Each algorithm optimizes the function $KL(·)$ in different ways. The alternative provides acceptable parameter estimations for the correct parameters, although the criterion is to minimize $KL(·)$. The corresponding plots can be found in Figure 2 to provide a comprehensive overview of the approximations and facilitate comparisons.

Models	Algorithm↓		MLE Initializations					Evidence-Free Initializations
			Parameters				$\mathit{KL}(·)$	Parameters				$\mathit{KL}(·)$
			${\mathit{\mu }}^{\prime }$	$\mathit{v}$	$\mathit{\alpha }$	$\mathit{\beta }$		${\mathit{\mu }}^{\prime }$	$\mathit{v}$	$\mathit{\alpha }$	$\mathit{\beta }$
$\mathcal{N}(x|0,y\left)\mathcal{I}\mathcal{G}\right(y|3,1)$	Initial Points →		$0.04$	$0.59$	$3.79$	$1.47$	$0.62$	0	$0.5$	10	15	$0.74$
	Alternative		$0.04$	$0.54$	$3.79$	$1.77$	$0.58$	0	$1.55$	10	$15.25$	$0.53$
	Gradient	$\gamma =1$	$0.04$	$1.05$	$3.27$	$3.39$	$0.58$	$0.04$	$1.45$	$10.09$	$14.95$	$0.53$
		$\gamma =\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$0.04$	$1.77$	$4.29$	$7.64$	$0.56$	$0.04$	$1.50$	$10.05$	$15.07$	$0.53$
		Natural Parameters	0	$0.55$	$3.71$	$2.73$	$0.59$	0	$1.52$	$10.17$	$15.47$	$0.52$
	Natural Gradient		$0.04$	$0.74$	$4.29$	$3.54$	$0.56$	$0.04$	$2.52$	$10.50$	$27.67$	$0.52$
$\mathcal{N}(x|1,y\left)\mathcal{I}\mathcal{G}\right(y|4,6)$	Initial Points →		$1.09$	$1.20$	$5.35$	$9.62$	$0.58$	1	$3.5$	3	3	$1.22$
	Alternative		$1.09$	$2.02$	$5.35$	$10.22$	$0.55$	1	$3.24$	3	$8.37$	$0.60$
	Gradient	$\gamma =1$	$1.09$	$1.68$	$5.56$	$9.55$	$0.55$	$1.09$	$1.40$	$3.26$	$4.45$	$0.58$
		$\gamma =\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$1.09$	$2.33$	$6.15$	$14.96$	$0.54$	$1.09$	$2.55$	$2.53$	$6.58$	$0.68$
		Natural Parameters	$1.36$	$1.54$	$5.44$	$10.44$	$0.58$	$1.10$	$1.91$	$4.18$	$8.21$	$0.56$
	Natural Gradient		$1.09$	$2.94$	$5.85$	$18.66$	$0.55$	1	$3.5$	3	3	$1.22$
$\mathcal{N}(x|-1,y\left)\mathcal{I}\mathcal{G}\right(y|7,10)$	Initial Points →		$-0.92$	$1.11$	$10.82$	$19.99$	$0.52$	$-1$	$0.75$	5	8	$0.67$
	Alternative		$-0.92$	$1.95$	$10.82$	$20.54$	$0.52$	$-1$	$1.75$	5	$8.38$	$0.55$
	Gradient	$\gamma =1$	$-0.92$	$1.77$	$10.94$	$19.94$	$0.52$	$-0.92$	$1.48$	$5.24$	$7.91$	$0.55$
		$\gamma =\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$-0.92$	$3.24$	$11.84$	$36.34$	$0.52$	$-0.86$	$1.69$	$5.17$	$8.25$	$0.55$
		Natural Parameters	$-1.19$	$1.6$	$10.87$	$20.25$	$0.55$	$-0.97$	$1.68$	$5.20$	$9.01$	$0.55$
	Natural Gradient		$-0.92$	$2.72$	$11.32$	$32.21$	$0.52$	$-0.92$	$2.60$	$5.50$	$15.58$	$0.55$
$\mathcal{N}(x|2,y\left)\mathcal{I}\mathcal{G}\right(y|6,10)$	Initial Points →		$2.09$	$1.21$	$8.86$	$18.66$	$0.59$	2	2	8	5	$1.06$
	Alternative		$2.09$	$2.24$	$8.86$	$19.27$	$0.53$	2	$0.98$	8	$7.34$	$0.54$
	Gradient	$\gamma =1$	$2.09$	$1.91$	$9.01$	$18.61$	$0.53$	$2.09$	$2.08$	$8.94$	$18.65$	$0.53$
		$\gamma =\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$2.09$	$3.15$	$10.27$	$33.24$	$0.52$	2	2	$8.86$	$18.66$	$0.53$
		Natural Parameters	$2.09$	$1.21$	$8.86$	$18.66$	$0.59$	2	2	5	5	$0.71$
	Natural Gradient		$2.09$	$2.88$	$9.36$	$28.39$	$0.53$	2	2	$8.5$	5	$1.06$
$\mathcal{N}(x|-2,y\left)\mathcal{I}\mathcal{G}\right(y|10,11)$	Initial Points →		$-1.93$	$0.96$	$17.52$	$26.54$	$0.56$	$-2$	$0.5$	8	7	$0.60$
	Alternative		$-1.93$	$1.57$	$17.52$	$27.02$	$0.51$	$-2$	$1.61$	8	$12.13$	$0.53$
	Gradient	$\gamma =1$	$-1.93$	$1.47$	$17.56$	$26.51$	$0.51$	$-1.93$	$1.48$	$17.57$	$26.51$	$0.51$
		$\gamma =\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$-1.93$	$1.33$	$17.42$	$29.43$	$0.53$	$-1.93$	$1.50$	$17.55$	$26.57$	$0.51$
		Natural Parameters	$-1.93$	$0.96$	$17.52$	$26.54$	$0.56$	$-2.01$	$0.56$	$8.20$	$7.96$	$0.60$
	Natural Gradient		$-1.93$	$1.80$	$18.02$	$33.32$	$0.51$	$-1.93$	$1.58$	$18.02$	$29.30$	$0.51$

using the alternative, gradient-based with $\gamma =1,{\displaystyle \frac{1}{\parallel \nabla KL\left(\tilde{\mathit{\theta }}\right)\parallel }}$, natural gradient algorithms and gradient considering natural parameters along with their contour and surface plots in Figure 1 and Figure 2, respectively.

All four algorithms attempt to minimize the same criterion. So, the objectives are consistent, but the number of steps in the recursive process may vary. The requirements must meet the minimum $KL(·)$. In this simple example of the Normal-Inverse-Gamma distribution with the model $p(x,y)=\mathcal{N}(x|1,y\left)\mathcal{I}\mathcal{G}\right(y|3,1)$, the convergence step numbers of the alternative, gradient-based with $\gamma =1, \parallel KL(·)\parallel$, gradient with natural parameters, and natural gradient algorithms are 1, 10, 8, 2, and 9 using Maximum Likelihood Estimation (MLE) initializations. One crucial point is that the process should not be repeated excessively because we need to find the local minimum of $KL(·)$. In this model, the VBA and gradient with respect to the natural parameters are the fastest, and the VBA provides the best approximation for the model depicted in Figure 2.

We simulate more models in Table 1 with the same joint distribution of the Normal-Inverse-Gamma along with different parameters. We draw all final results in Figure 2 and see their visual appearances. We start the recursive processes with two primary value groups for the unknown variables containing the MLE and the desired with no evidence. The selection for $\mu$ and v is quite simple because we have its data and can plot the histogram and approximately guess the correct values. The situation for $\alpha$ and $\beta$ is different and a bit problematic to surmise perfectly. If we assume any outliers for $\alpha$ and $\beta$, the algorithm finds a minimization for $KL(·)$ in the first stage of the iteration or another local minimum far from the truth. The more the iteration loop is executed, the KLD value increases or decreases in any case. Thus, the initializations are crucial to obtain the closest results for the unknown parameters. For instance, in $\mathcal{N}(x|1,y\left)\mathcal{I}\mathcal{G}\right(y|4,6)$ with elementary points $\mu =1$, $v=3.5$, $\alpha =3$, and $\beta =3$ in the natural gradient analytic algorithm, the repetitive result is the same as the initial points. So, these initializations are not suitable for the available data.

An algorithm with the lowest KLD value provides a more accurate estimate of the model parameters. As we discussed earlier, for the model $\mathcal{N}(x|0,y\left)\mathcal{I}\mathcal{G}\right(y|3,1)$, the first, fourth, and fifth algorithms estimate the parameters well, as shown in Figure 2, and Table 1. Remember, we use two values for $\gamma$ in the gradient-based algorithm with normal parameters, so the total estimated values are five for each unknown parameter. The first three algorithms approximate the parameters of the model $\mathcal{N}(x|1,y\left)\mathcal{I}\mathcal{G}\right(y|4,6)$ well, but the issue lies in fitting the center, resulting in a shifted approximation. All methods perform well in the models $\mathcal{N}(x|-1,y\left)\mathcal{I}\mathcal{G}\right(y|7,10)$, $\mathcal{N}(x|2,y\left)\mathcal{I}\mathcal{G}\right(y|6,10)$, and $\mathcal{N}(x|-2,y\left)\mathcal{I}\mathcal{G}\right(y|10,11)$. The optimal options for $\mathcal{N}(x|-1,y\left)\mathcal{I}\mathcal{G}\right(y|7,10)$ include VBA, gradient with $\gamma =1$, and gradient with respect to natural parameters. Considering the model $\mathcal{N}(x|2,y\left)\mathcal{I}\mathcal{G}\right(y|6,10)$, the best approach is to use gradient algorithms. The best option for the final model is the alternative approximation.

To gain an overview of the $KL(·)$ process, we plot its trend with respect to the number of iterations in Figure 3. The left column shows the results for the MLE initializations, where the minimization of $KL(·)$ occurs more quickly compared to the right column, which is associated with the evidence-free initializations. We do not plot the trend of the natural gradient’s $KL(·)$ in the second and last two models. The reason is that the $KL(·)$ becomes NaN in the initial iterations, so it is necessary to start the recursive process with some points estimated from the available data. The algorithms approximate the joint density function with a separable one but with different accuracy. In the following section, we tackle a more complex model.

4. Multivariate Normal-Inverse-Wishart Example

In the previous section, we explain how to perform VBA optimization methods to approximate a complicated joint distribution function by a tractable factorial of margins over a simple case study. In this section, a multivariate Normal case $p\left(\mathit{x}\right)=\mathcal{N}(\mathit{x}|\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }})$ is considered, which is approximated by $q\left(\mathbf{x}\right)={\prod }_i\mathcal{N}(x_i|{\tilde{\mu }}_i,{\tilde{\nu }}_i)$ for different shapes for the covariance matrix $\tilde{\mathsf{\Sigma }}$.

We assume that the basic structure of an available data set is multivariate Normal with the unknown mean vector $\tilde{\mathit{\mu }}$ and variance-covariance matrix $\tilde{\mathsf{\Sigma }}$. Their joint prior distribution is a Normal-Inverse-Wishart distribution of $\mathcal{N}\mathcal{I}\mathcal{W}(\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }}|{\tilde{\mathit{\mu }}}_0,\tilde{\kappa },\tilde{\mathsf{\Psi }},\tilde{\nu })$ defined by $\mathcal{N}(\tilde{\mathit{\mu }}|{\tilde{\mathit{\mu }}}_0,{\displaystyle \frac{1}{\tilde{\kappa }}}\tilde{\mathsf{\Sigma }})\mathcal{I}\mathcal{W}(\tilde{\mathsf{\Sigma }}|\tilde{\mathsf{\Psi }},\tilde{\nu })$, which is the generalized form of classical $\mathcal{N}\mathcal{I}\mathcal{G}$. One of its applications is in image segmentation tasks. The posteriors are multivariate Normal for the mean vector and Inverse Wishart for the variance-covariance matrix. Inverse Wishart distribution has many properties on the density function of unknown variances, and so there is a variety of references. For example, Bouriga and Féron [31] worked on covariance matrix estimation using Inverse Wishart distribution. In this regard, they inquired about posterior properties and Bayesian risk. Also, they applied Daniels and Kass prior [32] to the hyper-parameters of $\mathsf{\Sigma }$.

Before approximating the expression of $p(\mathbf{x},\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }})$, let us define some notations that we are using here:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ ⋮\\ x_p\end{array}\right],\tilde{\mathit{\mu }}=\left[\begin{array}{c}{\tilde{\mu }}_1\\ ⋮\\ {\tilde{\mu }}_p\end{array}\right], \mathrm{and}\tilde{\mathsf{\Sigma }}=\left[\begin{array}{ccc}{\tilde{\mathsf{\Sigma }}}_{11}& \cdots & {\tilde{\mathsf{\Sigma }}}_{1p}\\ {\tilde{\mathsf{\Sigma }}}_{21}& \cdots & {\tilde{\mathsf{\Sigma }}}_{2p}\\ ⋮& \ddots & ⋮\\ {\tilde{\mathsf{\Sigma }}}_{p1}& \cdots & {\tilde{\mathsf{\Sigma }}}_{pp}\end{array}\right].$$

Since the Normal-Inverse-Wishart distribution is a conjugate prior distribution for multivariate Normal, the posterior distribution of $\tilde{\mathit{\mu }}$ and $\tilde{\mathsf{\Sigma }}$ again belongs to the same family, and their corresponding margins are:

$$\mathcal{M}\mathcal{N}\left({\displaystyle \frac{\tilde{\kappa }{\tilde{\mathit{\mu }}}_0+n\overline{\mathbf{x}}}{\tilde{\kappa }+n}},{\displaystyle \frac{1}{\tilde{\kappa }+n}}\tilde{\mathsf{\Lambda }}\right),\mathcal{I}\mathcal{W}\left(\tilde{\mathsf{\Lambda }}+\tilde{\mathsf{\Psi }}+\sum _{i=1}^n({\mathbf{x}}_i-\overline{\mathbf{x}}){({\mathbf{x}}_i-\overline{\mathbf{x}})}^{⊺},\tilde{\nu }+n\right),$$

(16)

where n and $\mathbf{x}$ are the sample size and observations of $\mathbf{x}$, respectively. The proof is shown in Appendix B.

For other algorithms, we define $\tilde{\mathsf{\Theta }}=(\tilde{\kappa },{\tilde{\mathit{\mu }}}_0,\tilde{\mathsf{\Lambda }},\tilde{\nu },\tilde{\mathsf{\Psi }})$ for calculation of $KL\left(\tilde{\mathsf{\Theta }}\right)$. After some tedious computation available in Appendix C, the $KL\left(\tilde{\mathsf{\Theta }}\right)$ is equivalence by:

$$\begin{array}{c}\hfill KL\left(\tilde{\mathsf{\Theta }}\right)\propto {\displaystyle \frac{p-1}{2}}\ln \tilde{\kappa }+{\displaystyle \frac{1}{2}}\ln \parallel \tilde{\mathsf{\Psi }}{\tilde{\mathsf{\Lambda }}}^{-1}\parallel +{\displaystyle \frac{\tilde{\nu }}{2}}\mathrm{Tr}\left[{\tilde{\mathsf{\Psi }}}^{-1}\tilde{\mathsf{\Lambda }}+\mathbf{I}\right]-{\displaystyle \frac{p\tilde{\nu }}{2}}-{\displaystyle \frac{1}{2}}{\displaystyle \sum _{i=1}^p}{\psi }_0\left({\displaystyle \frac{\tilde{\nu }-p+i}{2}}\right),\end{array}$$

(17)

and the corresponding gradient-based and natural gradient algorithms are determined similar to Section 3 based on $\nabla KL\left(\tilde{\mathsf{\Theta }}\right)$, also available in Appendix C.

To present the performance of the four algorithms, we work on a data set coming from $\mathbf{x}\sim \mathcal{N}\mathcal{I}\mathcal{W}(\mathbf{x}|\mathit{\mu },\mathsf{\Sigma })$ whose parameters have the below density structure:

$$\mathit{\mu }\sim \mathcal{M}\mathcal{N}\left(\mathit{\mu }|\left[\begin{array}{c}2\\ 1\end{array}\right],{\displaystyle \frac{1}{2}}\left[\begin{array}{cc}3& -1\\ -1& 1\end{array}\right]\right),\mathsf{\Sigma }\sim \mathcal{I}\mathcal{W}\left(\mathsf{\Sigma }|\left[\begin{array}{cc}3& -1\\ -1& 1\end{array}\right],6\right).$$

(18)

We use only the data of $\mathbf{x}$ in the estimation processes. The results of algorithms are in

Table 2. There are some models with their estimation via four optimization algorithms with two different initializations. The lower the $KL(·)$, the more accurate the fitted posterior distribution.

Models	Algorithm↓		MLE Initializations						Evidence-Free Initializations
			Parameters					$\mathit{KL}(·)$	Parameters					$\mathit{KL}(·)$
			${\mathit{\mu }}_{\mathbf{0}}$	$\mathit{\kappa }$	$\mathsf{\Lambda }$	${\displaystyle \frac{\mathsf{\Psi }}{\mathit{\nu }+\mathit{n}-\mathit{p}-\mathbf{1}}}$	$\mathit{\nu }$		${\mathit{\mu }}_{\mathbf{0}}$	$\mathit{\kappa }$	$\mathsf{\Lambda }$	${\displaystyle \frac{\mathsf{\Psi }}{\mathit{\nu }+\mathbf{n}-\mathbf{p}-\mathbf{1}}}$	$\tilde{\mathit{\nu }}$
	Initial Points →		$\left[\begin{array}{c}1.94\\ 1.01\end{array}\right]$	5	$\left[\begin{array}{cc}158.08& -50.41\\ -50.41& 49.38\end{array}\right]$	$\left[\begin{array}{cc}0.01& 0\\ 0& 0\end{array}\right]$	5	$520.59$	$\left[\begin{array}{c}4\\ 2\end{array}\right]$	5	$\left[\begin{array}{cc}10& -5\\ -5& 4\end{array}\right]$	$\left[\begin{array}{cc}0.10& 0.05\\ 0.05& 0.04\end{array}\right]$	5	$21.91$
	Alternative		$\left[\begin{array}{c}1.94\\ 1.01\end{array}\right]$	5	$\left[\begin{array}{cc}1.51& -0.48\\ -0.48& 0.47\end{array}\right]$	$\left[\begin{array}{cc}3.03& -0.96\\ -0.96& 0.96\end{array}\right]$	5	$5.59$	$\left[\begin{array}{c}2.04\\ 1.06\end{array}\right]$	5	$\left[\begin{array}{cc}0.10& -0.05\\ -0.05& 0.04\end{array}\right]$	$\left[\begin{array}{cc}1.66& -0.47\\ -0.47& 0.53\end{array}\right]$	5	7
		$\gamma =1$	$\left[\begin{array}{c}1.94\\ 1.01\end{array}\right]$	5	$\left[\begin{array}{cc}158.08& -50.41\\ -50.41& 49.38\end{array}\right]$	$\left[\begin{array}{cc}0.01& -0.01\\ -0.01& 0.01\end{array}\right]$	5	$520.59$	$\left[\begin{array}{c}4\\ 2\end{array}\right]$	5	$\left[\begin{array}{cc}10& -5\\ -5& 4\end{array}\right]$	$\left[\begin{array}{cc}0.10& 0.05\\ 0.05& 0.04\end{array}\right]$	5	$21.91$
$\mathcal{N}\mathcal{I}\mathcal{W}\left(\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }}|\left[\begin{array}{c}2\\ 1\end{array}\right],2,\left[\begin{array}{cc}3& -1\\ -1& 1\end{array}\right],6\right)$	Gradient	$\gamma =\parallel \nabla KL\left(\tilde{\mathsf{\Theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}1.94\\ 1.01\end{array}\right]$	$4.85$	$\left[\begin{array}{cc}158.04& -50.42\\ -50.42& 49.32\end{array}\right]$	$\left[\begin{array}{cc}0.04& -0.01\\ -0.01& 0.03\end{array}\right]$	$3.99$	$88.82$	$\left[\begin{array}{c}4\\ 2\end{array}\right]$	$1.31$	$\left[\begin{array}{cc}7.82& -1.72\\ -1.72& 2.79\end{array}\right]$	$\left[\begin{array}{cc}0.14& 0.01\\ 0.01& 0.09\end{array}\right]$	$3.99$	$2.64$
		Natural Parameters	$\left[\begin{array}{c}1.94\\ 1.01\end{array}\right]$	5	$\left[\begin{array}{cc}158.08& -50.41\\ -50.41& 49.38\end{array}\right]$	$\left[\begin{array}{cc}0.01& 0\\ 0& 0\end{array}\right]$	5	$520.59$	$\left[\begin{array}{c}4\\ 2\end{array}\right]$	5	$\left[\begin{array}{cc}10& -5\\ -5& 4\end{array}\right]$	$\left[\begin{array}{cc}0.10& 0.05\\ 0.05& 0.04\end{array}\right]$	5	$21.91$
	Natural Gradient		$\left[\begin{array}{c}1.94\\ 1.01\end{array}\right]$	5	$\left[\begin{array}{cc}158.08& -50.41\\ -50.41& 49.38\end{array}\right]$	$\left[\begin{array}{cc}0.01& 0\\ 0& 0\end{array}\right]$	5	$520.59$	$\left[\begin{array}{c}4\\ 2\end{array}\right]$	5	$\left[\begin{array}{cc}16.67& -9.33\\ -9.33& 6.67\end{array}\right]$	$\left[\begin{array}{cc}0.11& 0.03\\ 0.03& 0.06\end{array}\right]$	5	$11.67$
	Initial Points →		$\left[\begin{array}{c}3.35\\ -1.39\\ 2.25\end{array}\right]$	5	$\left[\begin{array}{ccc}857.08& -388.16& 553.55\\ -388.16& 1041.13& -310.01\\ 553.55& -310.01& 1127.20\end{array}\right]$	$\left[\begin{array}{ccc}0.08& -0.04& 0.05\\ -0.04& 0.10& -0.03\\ 0.05& -0.03& 0.11\end{array}\right]$	5	$781.55$	$\left[\begin{array}{c}4\\ -2\\ 3\end{array}\right]$	5	$\left[\begin{array}{ccc}9& -6& 6\\ -6& 10& -6\\ 6& -6& 6\end{array}\right]$	$\left[\begin{array}{ccc}0.09& 0.06& 0.06\\ 0.06& 0.10& 0.06\\ 0.06& 0.06& 0.06\end{array}\right]$	5	$23.53$
	Alternative		$\left[\begin{array}{c}3.35\\ -1.39\\ 2.25\end{array}\right]$	5	$\left[\begin{array}{ccc}8.16& -3.70& 5.27\\ -3.70& 9.92& -2.95\\ 5.27& -2.95& 10.74\end{array}\right]$	$\left[\begin{array}{ccc}16.57& -7.50& 10.70\\ -7.50& 20.13& -5.99\\ 10.70& -5.99& 21.79\end{array}\right]$	5	$9.05$	$\left[\begin{array}{c}3.19\\ -1.32\\ 2.15\end{array}\right]$	5	$\left[\begin{array}{ccc}0.09& -0.06& 0.06\\ -0.06& 0.10& -0.06\\ 0.06& -0.06& 0.06\end{array}\right]$	$\left[\begin{array}{ccc}8.18& -3.62& 5.29\\ -3.62& 9.92& -2.89\\ 5.29& -2.89& 10.64\end{array}\right]$	5	$15.89$
		$\gamma =1$	$\left[\begin{array}{c}3.35\\ -1.39\\ 2.25\end{array}\right]$	5	$\left[\begin{array}{ccc}857.08& -388.16& 553.55\\ -388.16& 1041.13& -310.01\\ 553.55& -310.01& 1127.20\end{array}\right]$	$\left[\begin{array}{ccc}0.08& -0.04& 0.05\\ -0.04& 0.10& -0.03\\ 0.05& -0.03& 0.11\end{array}\right]$	5	$781.55$	$\left[\begin{array}{c}4\\ -2\\ 3\end{array}\right]$	5	$\left[\begin{array}{ccc}9& -6& 6\\ -6& 10& -6\\ 6& -6& 6\end{array}\right]$	$\left[\begin{array}{ccc}0.09& 0.06& 0.06\\ 0.06& 0.10& 0.06\\ 0.06& 0.06& 0.06\end{array}\right]$	5	$23.53$
$\mathcal{N}\mathcal{I}\mathcal{W}\left(\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }}|\left[\begin{array}{c}3\\ -1\\ 2\end{array}\right],3,\left[\begin{array}{ccc}3& -1& 2\\ -1& 4& -1\\ 2& -1& 5\end{array}\right],4\right)$	Gradient	$\gamma =\parallel \nabla KL\left(\tilde{\mathsf{\Theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}3.35\\ -1.39\\ 2.25\end{array}\right]$	5	$\left[\begin{array}{ccc}857.08& -388.16& 553.55\\ -388.16& 1041.13& -310.01\\ 553.55& -310.01& 1127.20\end{array}\right]$	$\left[\begin{array}{ccc}0.08& -0.04& 0.05\\ -0.04& 0.10& -0.03\\ 0.05& -0.03& 0.11\end{array}\right]$	$4.11$	$614.41$	$\left[\begin{array}{c}4\\ -2\\ 3\end{array}\right]$	$4.98$	$\left[\begin{array}{ccc}8.95& -6& 6.05\\ -6& 9.96& -5.94\\ 6.05& -5.94& 5.88\end{array}\right]$	$\left[\begin{array}{ccc}0.09& 0.06& 0.06\\ 0.06& 0.10& 0.06\\ 0.06& 0.06& 0.07\end{array}\right]$	$4.76$	$16.44$
		Natural Parameters	$\left[\begin{array}{c}3.35\\ -1.39\\ 2.25\end{array}\right]$	5	$\left[\begin{array}{ccc}857.08& -388.16& 553.55\\ -388.16& 1041.13& -310.01\\ 553.55& -310.01& 1127.2\end{array}\right]$	$\left[\begin{array}{ccc}0.08& -0.04& 0.05\\ -0.04& 0.1& -0.03\\ 0.05& -0.03& 0.11\end{array}\right]$	5	$781.55$	$\left[\begin{array}{c}4\\ -2\\ 3\end{array}\right]$	5	$\left[\begin{array}{ccc}9& -6& 6\\ -6& 10& -6\\ 6& -6& 6\end{array}\right]$	$\left[\begin{array}{ccc}0.09& 0.06& 0.06\\ 0.06& 0.10& 0.06\\ 0.06& 0.06& 0.06\end{array}\right]$	5	$23.53$
	Natural Gradient		$\left[\begin{array}{c}3.35\\ -1.39\\ 2.25\end{array}\right]$	5	$\left[\begin{array}{ccc}857.08& -388.16& 553.55\\ -388.16& 1041.13& -310.01\\ 553.55& -310.01& 1127.2\end{array}\right]$	$\left[\begin{array}{ccc}0.08& -0.04& 0.05\\ -0.04& 0.1& -0.03\\ 0.05& -0.03& 0.11\end{array}\right]$	5	$781.55$	$\left[\begin{array}{c}4\\ -2\\ 3\end{array}\right]$	5	$\left[\begin{array}{ccc}4& -2& 3\\ -10.8& 16& -10.8\\ 9.6& -10.8& 9.6\end{array}\right]$	$\left[\begin{array}{ccc}0.09& 0.06& 0.06\\ 0.06& 0.11& 0.05\\ 0.06& 0.05& 0.07\end{array}\right]$	$21.75$	$21.75$
	Initial Points →		$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}473.25& 152.04& 66.00& 31.17& -58.19\\ 152.04& 334.02& 34.61& -72.16& 221.59\\ 66.00& 34.61& 947.32& -90.75& 16.37\\ 31.17& -72.16& -90.75& 553.61& -68.83\\ -58.19& 221.59& 16.37& -68.83& 545.75\end{array}\right]$	$\left[\begin{array}{ccccc}0.05& 0.015& 0.01& 0& -0.01\\ 0.015& 0.032& 0& -0.01& 0.02\\ 0.01& 0& 0.09& -0.01& 0\\ 0& -0.01& -0.01& 0.05& -0.01\\ -0.01& 0.02& 0& -0.01& 0.05\end{array}\right]$	5	$1304.77$	$\left[\begin{array}{c}0.5\\ 1\\ -0.5\\ 1\\ 0.5\end{array}\right]$	5	$\left[\begin{array}{ccccc}10& 10& 10& 10& -5\\ 10& 10& 10& -10& 10\\ 10& 10& 8& -10& 10\\ 10& -10& -10& 10& -6\\ -5& 10& 10& -6& 10\end{array}\right]$	$\left[\begin{array}{ccccc}0.10& 0.10& 0.10& 0.10& 0.05\\ 0.10& 0.10& 0.10& -0.10& 0.10\\ 0.10& 0.10& 0.08& 0.10& 0.10\\ 0.10& -0.10& 0.10& 0.10& -0.06\\ 0.05& 0.10& 0.10& -0.06& 0.10\end{array}\right]$	5	$21.85$
	Alternative		$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}4.51& 1.45& 0.63& 0.30& -0.55\\ 1.45& 3.18& 0.33& -0.69& 2.11\\ 0.63& 0.33& 9.02& -0.86& 0.16\\ 0.30& -0.69& -0.86& 5.27& -0.66\\ -0.55& 2.11& 0.16& -0.66& 5.20\end{array}\right]$	$\left[\begin{array}{ccccc}9.33& 2.30& 1.30& 0.61& -1.15\\ 2.30& 6.59& 0.68& -1.42& 4.37\\ 1.30& 0.68& 18.68& -1.79& 0.32\\ 0.61& -1.42& -1.79& 10.92& -1.36\\ -1.15& 4.37& 0.32& -1.36& 10.76\end{array}\right]$	5	$17.28$	$\left[\begin{array}{c}0\\ 0\\ 0\\ 0\\ 0\end{array}\right]$	5	$\left[\begin{array}{ccccc}10& 10& 10& 10& -5\\ 10& 10& 10& -10& 10\\ 10& 10& 8& -10& 10\\ 10& -10& -10& 10& -6\\ -5& 10& 10& -6& 10\end{array}\right]$	$\left[\begin{array}{ccccc}0.10& 0.10& 0.10& 0.10& 0.05\\ 0.10& 0.10& 0.10& -0.10& 0.10\\ 0.10& 0.10& 0.18& 0.10& 0.10\\ 0.10& -0.10& 0.10& 0.10& 0.06\\ 0.05& 0.10& 0.10& -0.06& 0.10\end{array}\right]$	5	$21.85$
		$\gamma =1$	$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}473.25& 152.04& 66.00& 31.17& -58.19\\ 152.04& 334.02& 34.61& -72.16& 221.59\\ 66.00& 34.61& 947.32& -90.75& 16.37\\ 31.17& -72.16& -90.75& 553.61& -68.83\\ -58.19& 221.59& 16.37& -68.83& 545.75\end{array}\right]$	$\left[\begin{array}{ccccc}0.05& 0.01& 0.01& 0.00& -0.01\\ 0.01& 0.03& 0.00& -0.01& 0.02\\ 0.01& 0.00& 0.09& -0.01& 0.00\\ 0.00& 0.00& -0.01& 0.05& -0.01\\ -0.01& 0.02& 0.00& 0.00& 0.05\end{array}\right]$	5	$1304.77$	$\left[\begin{array}{c}0.5\\ 1\\ -0.5\\ 1\\ 0.5\end{array}\right]$	$4.6$	$\left[\begin{array}{ccccc}10.00& 9.64& 9.91& 10.00& -4.55\\ 9.64& 9.70& 10.49& -9.88& 10.09\\ 9.91& 10.49& 7.92& -10.11& 9.57\\ 10.00& -9.88& -10.11& 10.04& -5.98\\ -4.55& 10.09& 9.57& -5.98& 9.91\end{array}\right]$	$\left[\begin{array}{ccccc}0.10& 0.11& 0.10& 0.10& 0.04\\ 0.11& 0.10& 0.09& -0.10& 0.10\\ 0.10& 0.09& 0.08& 0.10& 0.11\\ 0.10& -0.10& 0.10& 0.10& -0.06\\ 0.04& 0.10& 0.11& -0.06& 0.11\end{array}\right]$	$3.66$	$11.15$
$\mathcal{N}\mathcal{I}\mathcal{W}\left(\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }}|\left[\begin{array}{c}0\\ 0\\ 0\\ 0\\ 0\end{array}\right],4,\left[\begin{array}{ccccc}6& 2& 1& 1& 0\\ 2& 4& 0& -1& 2\\ 1& 0& 9& 1& 0\\ 1& -1& 1& 7& 0\\ 0& 2& 0& 0& 5\end{array}\right],7\right)$	Gradient	$\gamma =\parallel \nabla KL\left(\tilde{\mathsf{\Theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}473.25& 152.04& 66.00& 31.17& -58.19\\ 152.04& 334.02& 34.61& -72.16& 221.59\\ 66.00& 34.61& 947.32& -90.75& 16.37\\ 31.17& -72.16& -90.75& 553.61& -68.83\\ -58.19& 221.59& 16.37& -68.83& 545.75\end{array}\right]$	$\left[\begin{array}{ccccc}0.05& 0.01& 0.01& 0.00& 0.00\\ 0.01& 0.04& 0.00& 0.00& 0.02\\ 0.01& 0.00& 0.10& 0.00& 0.00\\ 0.00& 0.00& -0.01& 0.06& -0.01\\ 0.00& 0.02& 0.00& -0.01& 0.06\end{array}\right]$	$4.28$	$1018.67$	$\left[\begin{array}{c}0.5\\ 1\\ -0.5\\ 1\\ 0.5\end{array}\right]$	$4.88$	$\left[\begin{array}{ccccc}10.00& 9.90& 9.97& 10.00& -4.87\\ 9.90& 9.91& 10.14& -9.96& 10.03\\ 9.97& 10.14& 7.98& -10.03& 9.87\\ 10.00& -9.96& -10.03& 10.01& -5.99\\ -4.87& 10.03& 9.87& -5.99& 9.97\end{array}\right]$	$\left[\begin{array}{ccccc}0.10& 0.10& 0.10& 0.10& 0.05\\ 0.10& 0.10& -0.10& 0.10& 0.10\\ 0.10& 0.10& 0.08& 0.10& 0.10\\ 0.10& -0.10& 0.10& 0.10& -0.06\\ 0.05& 0.10& 0.10& -0.06& 0.10\end{array}\right]$	$4.71$	$19.13$
		Natural Parameters	$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}473.25& 152.04& 66.00& 31.17& -58.19\\ 152.04& 334.02& 34.61& -72.16& 221.59\\ 66.00& 34.61& 947.32& -90.75& 16.37\\ 31.17& -72.16& -90.75& 553.61& -68.83\\ -58.19& 221.59& 16.37& -68.83& 545.75\end{array}\right]$	$\left[\begin{array}{ccccc}0.05& 0.015& 0.01& 0& -0.01\\ 0.015& 0.032& 0& -0.01& 0.02\\ 0.01& 0& 0.09& -0.01& 0\\ 0& -0.01& -0.01& 0.05& -0.01\\ -0.01& 0.02& 0& -0.01& 0.05\end{array}\right]$	5	$1304.77$	$\left[\begin{array}{c}0.5\\ 1\\ -0.5\\ 1\\ 0.5\end{array}\right]$	5	$\left[\begin{array}{ccccc}10& 10& 10& 10& -5\\ 10& 10& 10& -10& 10\\ 10& 10& 8& -10& 10\\ 10& -10& -10& 10& -6\\ -5& 10& 10& -6& 10\end{array}\right]$	$\left[\begin{array}{ccccc}0.10& 0.10& 0.10& 0.10& 0.05\\ 0.10& 0.10& 0.10& -0.10& 0.10\\ 0.10& 0.10& 0.08& 0.10& 0.10\\ 0.10& -0.10& 0.10& 0.10& -0.06\\ 0.05& 0.10& 0.10& -0.06& 0.10\end{array}\right]$	5	$21.85$
	Natural Gradient		$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}473.25& 152.04& 66.00& 31.17& -58.19\\ 152.04& 334.02& 34.61& -72.16& 221.59\\ 66.00& 34.61& 947.32& -90.75& 16.37\\ 31.17& -72.16& -90.75& 553.61& -68.83\\ -58.19& 221.59& 16.37& -68.83& 545.75\end{array}\right]$	$\left[\begin{array}{ccccc}0.05& 0.015& 0.01& 0& -0.01\\ 0.015& 0.032& 0& -0.01& 0.02\\ 0.01& 0& 0.09& -0.01& 0\\ 0& -0.01& -0.01& 0.05& -0.01\\ -0.01& 0.02& 0& -0.01& 0.05\end{array}\right]$	5	$1304.77$	$\left[\begin{array}{c}-0.18\\ -0.29\\ 0.42\\ -0.23\\ -0.05\end{array}\right]$	5	$\left[\begin{array}{ccccc}473.25& 152.04& 66.00& 31.17& -58.19\\ 152.04& 334.02& 34.61& -72.16& 221.59\\ 66.00& 34.61& 947.32& -90.75& 16.37\\ 31.17& -72.16& -90.75& 553.61& -68.83\\ -58.19& 221.59& 16.37& -68.83& 545.75\end{array}\right]$	$\left[\begin{array}{ccccc}0.05& 0.015& 0.01& 0& -0.01\\ 0.015& 0.032& 0& -0.01& 0.02\\ 0.01& 0& 0.09& -0.01& 0\\ 0& -0.01& -0.01& 0.05& -0.01\\ -0.01& 0.02& 0& -0.01& 0.05\end{array}\right]$	5	$1304.77$

and drowned in Figure 4 along with its true contour plot of the model. In this figure, we have two different rows of results. In the first row, we use the MLE estimation of the mean and covariance parameters as our pre-information to start the algorithms and find the best local minimum for the $KL(·)$. In this regard, we put some gusted value for $\kappa$ and $\nu$. At the bottom of the figure, we use some uninformative initializations. This choice is crucial because the evidence-free primitive starter may end in some local optimization, which is far from the part we need. Thus, using MLE prevents this problem to a great extent. Although the result for the alternative optimization is good using evidence-free elementary points in Figure 4, it does not always work this way and depends on luck. Other gradient-based and natural gradient algorithms do not work well in this example. One reason can be the more dimensions, the worse the results.

In Table 2, we have more models with a variety of dimensions 2, 3, and 5 applying two initialization groups, MLE and uninformative starters. In all cases, the standard alternative algorithm has the closest estimation to the real model in the case of $KL(·)$ minimization. The larger the $KL(·)$, the greater the distance between the true and estimated posterior distributions. Therefore, all gradient-based and natural gradient algorithms are not worthy practices to extract the posterior distribution with more dimensions in the data, while our alternative acts better in this situation. In

Table 3. We provide an example of running the algorithms on a large-scale dataset. The standard alternative algorithm is outstanding here.

Models	Algorithm↓		MLE Initializations
			Parameters					$\mathit{KL}(·)$
			${\mathit{\mu }}_{\mathbf{0}}$	$\mathit{\kappa }$	$\mathsf{\Lambda }$	${\displaystyle \frac{\mathsf{\Psi }}{\mathit{\nu }+\mathit{n}-\mathit{p}-\mathbf{1}}}$	$\mathit{\nu }$
	Initial Points →		$\left[\begin{array}{c}1\\ 2.02\\ 1.02\\ 3.05\\ -1.02\\ 4.05\\ -4.98\\ 10.03\\ 1.08\\ 0.03\end{array}\right]$	5	$\left[\begin{array}{cccccccccc}3.03& 0.18& -2.69& 1.07& 0.43& 0.38& 1.49& 0.49& 0.5& -1.33\\ 0.18& 4.73& 2.56& 1.57& 3.75& 4.62& 3.12& 3.48& 3.49& 1.02\\ -2.69& 2.56& 11.84& -0.18& 4.91& 3.69& 2.44& 1.37& 1.64& 0.25\\ 1.07& 1.57& -0.18& 6.67& 2.21& 4.75& 2.65& 1.87& 4.21& 0.16\\ 0.43& 3.75& 4.91& 2.21& 7.62& 4.31& 4.63& 2.93& 2.3& 0.33\\ 0.38& 4.62& 3.69& 4.75& 4.31& 8.59& 4.52& 4.69& 7.47& 1.78\\ 1.49& 3.12& 2.44& 2.65& 4.63& 4.52& 5.31& 3.66& 4.01& 0.21\\ 0.49& 3.48& 1.37& 1.87& 2.93& 4.69& 3.66& 7.73& 4.21& 5.67\\ 0.5& 3.49& 1.64& 4.21& 2.3& 7.47& 4.01& 4.21& 9.51& 1.81\\ -1.33& 1.02& 0.25& 0.16& 0.33& 1.78& 0.21& 5.67& 1.81& 8.67\end{array}\right]$	$\left[\begin{array}{cccccccccc}0.2& 0.01& -0.18& 0.07& 0.03& 0.02& 0.1& 0.03& 0.03& -0.09\\ 0.01& 0.31& 0.17& 0.1& 0.25& 0.3& 0.21& 0.23& 0.23& 0.07\\ -0.18& 0.17& 0.78& -0.01& 0.32& 0.24& 0.16& 0.09& 0.11& 0.02\\ 0.07& 0.1& -0.01& 0.44& 0.15& 0.31& 0.17& 0.12& 0.28& 0.01\\ 0.03& 0.25& 0.32& 0.15& 0.5& 0.28& 0.3& 0.19& 0.15& 0.02\\ 0.02& 0.3& 0.24& 0.31& 0.28& 0.56& 0.3& 0.31& 0.49& 0.12\\ 0.1& 0.21& 0.16& 0.17& 0.3& 0.3& 0.35& 0.24& 0.26& 0.01\\ 0.03& 0.23& 0.09& 0.12& 0.19& 0.31& 0.24& 0.51& 0.28& 0.37\\ 0.03& 0.23& 0.11& 0.28& 0.15& 0.49& 0.26& 0.28& 0.63& 0.12\\ -0.09& 0.07& 0.02& 0.01& 0.02& 0.12& 0.01& 0.37& 0.12& 0.57\end{array}\right]$	10	$5231.57$
	Alternative		$\left[\begin{array}{c}1.09\\ 2.44\\ 1.44\\ 4.19\\ -1.59\\ 5.2\\ -4.61\\ 10.81\\ 3.14\\ 0.79\end{array}\right]$	5	$\left[\begin{array}{cccccccccc}19.75& 1.15& -17.53& 6.96& 2.79& 2.45& 9.72& 3.21& 3.26& -8.68\\ 1.15& 30.8& 16.64& 10.21& 24.44& 30.1& 20.3& 22.69& 22.69& 6.63\\ -17.53& 16.64& 77.06& -1.17& 31.99& 24.04& 15.86& 8.9& 10.66& 1.61\\ 6.96& 10.21& -1.17& 43.41& 14.38& 30.95& 17.28& 12.16& 27.39& 1.05\\ 2.79& 24.44& 31.99& 14.38& 49.64& 28.05& 30.12& 19.05& 15& 2.16\\ 2.45& 30.1& 24.04& 30.95& 28.05& 55.9& 29.45& 30.51& 48.65& 11.62\\ 9.72& 20.3& 15.86& 17.28& 30.12& 29.45& 34.55& 23.8& 26.11& 1.35\\ 3.21& 22.69& 8.9& 12.16& 19.05& 30.51& 23.8& 50.35& 27.43& 36.89\\ 3.26& 22.69& 10.66& 27.39& 15& 48.65& 26.11& 27.43& 61.89& 11.82\\ -8.68& 6.63& 1.61& 1.05& 2.16& 11.62& 1.35& 36.89& 11.82& 56.43\end{array}\right]$	$\left[\begin{array}{cccccccccc}40.9& 2.37& -36.3& 14.41& 5.77& 5.07& 20.13& 6.66& 6.75& -17.97\\ 2.37& 63.77& 34.45& 21.13& 50.62& 62.33& 42.03& 46.98& 46.99& 13.72\\ -36.3& 34.45& 159.58& -2.43& 66.25& 49.77& 32.84& 18.44& 22.08& 3.34\\ 14.41& 21.13& -2.43& 89.89& 29.77& 64.08& 35.79& 25.18& 56.72& 2.17\\ 5.77& 50.62& 66.25& 29.77& 102.79& 58.09& 62.37& 39.44& 31.05& 4.46\\ 5.07& 62.33& 49.77& 64.08& 58.09& 115.75& 60.97& 63.17& 100.74& 24.06\\ 20.13& 42.03& 32.84& 35.79& 62.37& 60.97& 71.53& 49.29& 54.07& 2.8\\ 6.66& 46.98& 18.44& 25.18& 39.44& 63.17& 49.29& 104.26& 56.79& 76.38\\ 6.75& 46.99& 22.08& 56.72& 31.05& 100.74& 54.07& 56.79& 128.16& 24.47\\ -17.97& 13.72& 3.34& 2.17& 4.46& 24.06& 2.8& 76.38& 24.47& 116.85\end{array}\right]$	10	$31.70$
		$\gamma =1$	$\left[\begin{array}{c}1.09\\ 2.44\\ 1.44\\ 4.19\\ -1.59\\ 5.2\\ -4.61\\ 10.81\\ 3.14\\ 0.79\end{array}\right]$	5	$\left[\begin{array}{cccccccccc}2074.17& 120.32& -1840.86& 730.59& 292.65& 257.14& 1020.81& 337.48& 342.19& -911.26\\ 120.32& 3233.78& 1746.98& 1071.53& 2566.57& 3160.45& 2131.31& 2382.43& 2382.86& 695.73\\ -1840.86& 1746.98& 8091.77& -123.13& 3359.43& 2523.71& 1665.11& 934.91& 1119.54& 169.52\\ 730.59& 1071.53& -123.13& 4558.21& 1509.46& 3249.33& 1814.84& 1276.68& 2875.88& 109.91\\ 292.65& 2566.57& 3359.43& 1509.46& 5212.43& 2945.44& 3162.79& 1999.9& 1574.61& 226.38\\ 257.14& 3160.45& 2523.71& 3249.33& 2945.44& 5869.57& 3091.85& 3203.23& 5108.31& 1219.85\\ 1020.81& 2131.31& 1665.11& 1814.84& 3162.79& 3091.85& 3627.32& 2499.28& 2741.79& 142.06\\ 337.48& 2382.43& 934.91& 1276.68& 1999.9& 3203.23& 2499.28& 5286.96& 2879.7& 3873.26\\ 342.19& 2382.86& 1119.54& 2875.88& 1574.61& 5108.31& 2741.79& 2879.7& 6498.71& 1240.72\\ -911.26& 695.73& 169.52& 109.91& 226.38& 1219.85& 142.06& 3873.26& 1240.72& 5925.01\end{array}\right]$	$\left[\begin{array}{cccccccccc}0.2& 0.01& -0.18& 0.07& 0.03& 0.02& 0.1& 0.03& 0.03& -0.09\\ 0.01& 0.31& 0.17& 0.1& 0.25& 0.3& 0.21& 0.23& 0.23& 0.07\\ -0.18& 0.17& 0.78& -0.01& 0.32& 0.24& 0.16& 0.09& 0.11& 0.02\\ 0.07& 0.1& -0.01& 0.44& 0.15& 0.31& 0.17& 0.12& 0.28& 0.01\\ 0.03& 0.25& 0.32& 0.15& 0.5& 0.28& 0.3& 0.19& 0.15& 0.02\\ 0.02& 0.3& 0.24& 0.31& 0.28& 0.56& 0.3& 0.31& 0.49& 0.12\\ 0.1& 0.21& 0.16& 0.17& 0.3& 0.3& 0.35& 0.24& 0.26& 0.01\\ 0.03& 0.23& 0.09& 0.12& 0.19& 0.31& 0.24& 0.51& 0.28& 0.37\\ 0.03& 0.23& 0.11& 0.28& 0.15& 0.49& 0.26& 0.28& 0.63& 0.12\\ -0.09& 0.07& 0.02& 0.01& 0.02& 0.12& 0.01& 0.37& 0.12& 0.57\end{array}\right]$	10	$5231.57$
$\mathcal{N}\mathcal{I}\mathcal{W}\left(\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }}|\left[\begin{array}{c}1\\ 2\\ 1\\ 3\\ -1\\ 4\\ -5\\ 10\\ 1\\ 0\end{array}\right],10,\left[\begin{array}{cccccccccc}67& -11& -50& 16& -4& -2& 21& 21& 10& -16\\ -11& 92& 37& 20& 49& 78& 37& 51& 53& 32\\ -50& 37& 177& 14& 53& 76& 41& 29& 61& 9\\ 16& 20& 14& 121& 10& 88& 28& 59& 89& 15\\ -4& 49& 53& 10& 114& 57& 53& 44& 30& 26\\ -2& 78& 76& 88& 57& 160& 69& 92& 145& 43\\ 21& 37& 41& 28& 53& 69& 71& 60& 73& 11\\ 21& 51& 29& 59& 44& 92& 60& 170& 86& 120\\ 10& 53& 61& 89& 30& 145& 73& 86& 196& 33\\ -16& 32& 9& 15& 26& 43& 11& 120& 33& 165\end{array}\right],15\right)$	Gradient	$\gamma =\parallel \nabla KL\left(\tilde{\mathsf{\Theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}1.09\\ 2.44\\ 1.44\\ 4.19\\ -1.59\\ 5.2\\ -4.61\\ 10.81\\ 3.14\\ 0.79\end{array}\right]$	5	$\left[\begin{array}{cccccccccc}2074.17& 120.32& -1840.86& 730.59& 292.65& 257.14& 1020.81& 337.48& 342.19& -911.27\\ 120.32& 3233.78& 1746.98& 1071.53& 2566.57& 3160.45& 2131.31& 2382.43& 2382.86& 695.73\\ -1840.86& 1746.98& 8091.77& -123.13& 3359.43& 2523.71& 1665.11& 934.91& 1119.54& 169.52\\ 730.59& 1071.53& -123.13& 4558.21& 1509.46& 3249.33& 1814.84& 1276.68& 2875.88& 109.91\\ 292.65& 2566.57& 3359.43& 1509.46& 5212.43& 2945.44& 3162.79& 1999.9& 1574.61& 226.38\\ 257.14& 3160.45& 2523.71& 3249.33& 2945.44& 5869.57& 3091.85& 3203.23& 5108.31& 1219.85\\ 1020.81& 2131.31& 1665.11& 1814.84& 3162.79& 3091.85& 3627.32& 2499.28& 2741.79& 142.06\\ 337.48& 2382.43& 934.91& 1276.68& 1999.9& 3203.23& 2499.28& 5286.96& 2879.7& 3873.26\\ 342.19& 2382.86& 1119.54& 2875.88& 1574.61& 5108.31& 2741.79& 2879.7& 6498.71& 1240.72\\ -911.27& 695.73& 169.52& 109.91& 226.38& 1219.85& 142.06& 3873.26& 1240.72& 5925.01\end{array}\right]$	$\left[\begin{array}{cccccccccc}0.2& 0.01& -0.18& 0.07& 0.03& 0.02& 0.1& 0.03& 0.03& -0.09\\ 0.01& 0.32& 0.17& 0.1& 0.25& 0.31& 0.21& 0.23& 0.23& 0.07\\ -0.18& 0.17& 0.79& -0.01& 0.33& 0.24& 0.16& 0.09& 0.11& 0.02\\ 0.07& 0.1& -0.01& 0.44& 0.15& 0.31& 0.18& 0.12& 0.28& 0.01\\ 0.03& 0.25& 0.33& 0.15& 0.51& 0.29& 0.31& 0.19& 0.15& 0.02\\ 0.02& 0.31& 0.24& 0.31& 0.29& 0.57& 0.3& 0.31& 0.5& 0.12\\ 0.1& 0.21& 0.16& 0.18& 0.31& 0.3& 0.35& 0.24& 0.27& 0.01\\ 0.03& 0.23& 0.09& 0.12& 0.19& 0.31& 0.24& 0.51& 0.28& 0.38\\ 0.03& 0.23& 0.11& 0.28& 0.15& 0.5& 0.27& 0.28& 0.63& 0.12\\ -0.09& 0.07& 0.02& 0.01& 0.02& 0.12& 0.01& 0.38& 0.12& 0.58\end{array}\right]$	$9.09$	$4681.41$
		Natural Parameters	$\left[\begin{array}{c}1\\ 2.02\\ 1.02\\ 3.05\\ -1.02\\ 4.05\\ -4.98\\ 10.03\\ 1.08\\ 0.03\end{array}\right]$	5	$\left[\begin{array}{cccccccccc}3.03& 0.18& -2.69& 1.07& 0.43& 0.38& 1.49& 0.49& 0.5& -1.33\\ 0.18& 4.73& 2.56& 1.57& 3.75& 4.62& 3.12& 3.48& 3.49& 1.02\\ -2.69& 2.56& 11.84& -0.18& 4.91& 3.69& 2.44& 1.37& 1.64& 0.25\\ 1.07& 1.57& -0.18& 6.67& 2.21& 4.75& 2.65& 1.87& 4.21& 0.16\\ 0.43& 3.75& 4.91& 2.21& 7.62& 4.31& 4.63& 2.93& 2.3& 0.33\\ 0.38& 4.62& 3.69& 4.75& 4.31& 8.59& 4.52& 4.69& 7.47& 1.78\\ 1.49& 3.12& 2.44& 2.65& 4.63& 4.52& 5.31& 3.66& 4.01& 0.21\\ 0.49& 3.48& 1.37& 1.87& 2.93& 4.69& 3.66& 7.73& 4.21& 5.67\\ 0.5& 3.49& 1.64& 4.21& 2.3& 7.47& 4.01& 4.21& 9.51& 1.81\\ -1.33& 1.02& 0.25& 0.16& 0.33& 1.78& 0.21& 5.67& 1.81& 8.67\end{array}\right]$	$\left[\begin{array}{cccccccccc}0.2& 0.01& -0.18& 0.07& 0.03& 0.02& 0.1& 0.03& 0.03& -0.09\\ 0.01& 0.31& 0.17& 0.1& 0.25& 0.3& 0.21& 0.23& 0.23& 0.07\\ -0.18& 0.17& 0.78& -0.01& 0.32& 0.24& 0.16& 0.09& 0.11& 0.02\\ 0.07& 0.1& -0.01& 0.44& 0.15& 0.31& 0.17& 0.12& 0.28& 0.01\\ 0.03& 0.25& 0.32& 0.15& 0.5& 0.28& 0.3& 0.19& 0.15& 0.02\\ 0.02& 0.3& 0.24& 0.31& 0.28& 0.56& 0.3& 0.31& 0.49& 0.12\\ 0.1& 0.21& 0.16& 0.17& 0.3& 0.3& 0.35& 0.24& 0.26& 0.01\\ 0.03& 0.23& 0.09& 0.12& 0.19& 0.31& 0.24& 0.51& 0.28& 0.37\\ 0.03& 0.23& 0.11& 0.28& 0.15& 0.49& 0.26& 0.28& 0.63& 0.12\\ -0.09& 0.07& 0.02& 0.01& 0.02& 0.12& 0.01& 0.37& 0.12& 0.57\end{array}\right]$	10	$5231.57$
	Natural Gradient		$\left[\begin{array}{c}1\\ 2.02\\ 1.02\\ 3.05\\ -1.02\\ 4.05\\ -4.98\\ 10.03\\ 1.08\\ 0.03\end{array}\right]$	5	$\left[\begin{array}{cccccccccc}3.03& 0.18& -2.69& 1.07& 0.43& 0.38& 1.49& 0.49& 0.5& -1.33\\ 0.18& 4.73& 2.56& 1.57& 3.75& 4.62& 3.12& 3.48& 3.49& 1.02\\ -2.69& 2.56& 11.84& -0.18& 4.91& 3.69& 2.44& 1.37& 1.64& 0.25\\ 1.07& 1.57& -0.18& 6.67& 2.21& 4.75& 2.65& 1.87& 4.21& 0.16\\ 0.43& 3.75& 4.91& 2.21& 7.62& 4.31& 4.63& 2.93& 2.3& 0.33\\ 0.38& 4.62& 3.69& 4.75& 4.31& 8.59& 4.52& 4.69& 7.47& 1.78\\ 1.49& 3.12& 2.44& 2.65& 4.63& 4.52& 5.31& 3.66& 4.01& 0.21\\ 0.49& 3.48& 1.37& 1.87& 2.93& 4.69& 3.66& 7.73& 4.21& 5.67\\ 0.5& 3.49& 1.64& 4.21& 2.3& 7.47& 4.01& 4.21& 9.51& 1.81\\ -1.33& 1.02& 0.25& 0.16& 0.33& 1.78& 0.21& 5.67& 1.81& 8.67\end{array}\right]$	$\left[\begin{array}{cccccccccc}0.2& 0.01& -0.18& 0.07& 0.03& 0.02& 0.1& 0.03& 0.03& -0.09\\ 0.01& 0.31& 0.17& 0.1& 0.25& 0.3& 0.21& 0.23& 0.23& 0.07\\ -0.18& 0.17& 0.78& -0.01& 0.32& 0.24& 0.16& 0.09& 0.11& 0.02\\ 0.07& 0.1& -0.01& 0.44& 0.15& 0.31& 0.17& 0.12& 0.28& 0.01\\ 0.03& 0.25& 0.32& 0.15& 0.5& 0.28& 0.3& 0.19& 0.15& 0.02\\ 0.02& 0.3& 0.24& 0.31& 0.28& 0.56& 0.3& 0.31& 0.49& 0.12\\ 0.1& 0.21& 0.16& 0.17& 0.3& 0.3& 0.35& 0.24& 0.26& 0.01\\ 0.03& 0.23& 0.09& 0.12& 0.19& 0.31& 0.24& 0.51& 0.28& 0.37\\ 0.03& 0.23& 0.11& 0.28& 0.15& 0.49& 0.26& 0.28& 0.63& 0.12\\ -0.09& 0.07& 0.02& 0.01& 0.02& 0.12& 0.01& 0.37& 0.12& 0.57\end{array}\right]$	10	$5231.57$

, we provide a Normal-Inverse-Wishart model with 10 dimensions. In comparison, the alternative method approximates the distribution with so many parameters more precisely than the other methods, which seems easy to use with any large-scale dimension dataset, and the result is suitable enough.

5. Simple Linear Inverse Problem

The third example is the case of linear inverse problems $\mathit{g}=\mathit{H}\mathit{f}+\mathit{ϵ}$ with priors $p\left(\mathit{ϵ}\right)=\mathcal{N}(\mathit{ϵ}|\mathbf{0},v_{ϵ}\mathit{I})$ and $p\left(\mathit{f}\right)=\mathcal{N}(\mathit{f}|\mathbf{0},\mathrm{diag}\left[\mathit{v}\right])$, where $\mathit{f}=[f_1,f_2,\cdots ,f_N]$ and $\mathit{v}=[v_1,v_2,\cdots ,v_N]$ for which we have $p(\mathit{f},\mathit{v}|\mathit{g})\propto p(\mathit{g}|\mathit{f},v_{ϵ})p(\mathit{f}|\mathit{v})p\left(\mathit{v}\right)$ with $p(\mathit{g}|\mathit{f},v_{ϵ})=\mathcal{N}(\mathit{g}|\mathit{H}\mathit{f},v_{ϵ}\mathit{I})$, $p(\mathit{f}|\mathit{v})=\mathcal{N}(\mathit{f}|\mathbf{0},\mathrm{diag}\left[\mathit{v}\right])$ and $p(\mathit{v}|\alpha ,\beta )={\prod }_j\mathcal{I}\mathcal{G}(v_j|\alpha ,\beta )$, [33].

In the inverse problem, we have the same multivariate case, except in Section 3 that this time, the sensor does not give directly $\mathit{f}$, but $\mathit{g}$, such that $\mathit{g}=\mathit{H}\mathit{f}+\mathit{ϵ}$ where $\mathit{ϵ}$ is again a Normal with known or unknown variance. In the first step, we assume that the variance $v_{ϵ}$ and $\mathit{H}$ are known. $\mathit{H}$ is called the transfer function of the sensor. So, this time we have $p(\mathit{g},\mathit{f},\mathit{v})=\mathcal{N}(\mathit{g}|\mathit{f}\left)\mathcal{N}\right(\mathit{f}|\mathit{v})\mathcal{I}\mathcal{G}\left(\mathit{v}\right)$ and we approximate it by $q_1(\tilde{\mathit{f}}|\mathit{g},\tilde{\mathit{v}})q_2\left(\tilde{\mathit{v}}\right)$. The main reason of this choice for $q_1(\tilde{\mathit{f}}|\mathit{g},\tilde{\mathit{v}})$ is that, when $\mathit{g}$ and $\tilde{\mathit{v}}$ are known, this becomes a Normal $q_1(\tilde{\mathit{f}}|\mathit{g},\tilde{\mathit{v}})=N(\tilde{\mathit{f}}|\tilde{\mathit{\mu }},\tilde{\mathsf{\Sigma }})$ where the expressions of $\tilde{\mathit{\mu }}$ and $\tilde{\mathsf{\Sigma }}$ are given in the paper. Now, we have a multivariate $\mathcal{N}\mathcal{I}\mathcal{G}$ problem. Thus, the joint distribution of $\mathit{g}$, $\mathit{f}$, and $\mathit{v}$ is estimated by VBA as the following relation:

$$p(\mathit{f},\mathit{v}|\mathit{g})\propto p(\mathit{f}|\mathit{g},\mathit{v})p\left(\mathit{v}\right).$$

(19)

Although we can mathematically compute the margins in this particular example, we desire to approximate them via the iterative alternative algorithm $q(\mathit{g},\tilde{\mathit{f}},\tilde{\mathit{v}})=q_1(\mathit{g}|\tilde{\mathit{f}})q_2\left(\tilde{\mathit{f}}\right)$$q_3\left(\tilde{\mathit{v}}\right)$ compared them with gradient and natural gradient-based algorithms. The objective function is the estimation of $q_2\left(\tilde{\mathit{f}}\right)$, but in the recursive process, $q_1(\mathit{g}|\tilde{\mathit{f}})$ and $q_3\left(\tilde{\mathit{v}}\right)$ are updated, too. For simplicity, we suppose that the transposition matrix $\mathit{H}$ is an identical matrix $\mathit{I}$. The final outputs are as follows, with the details and algorithm in Appendix D:

$$\begin{array}{c}\hfill \tilde{\mathit{f}}\sim \mathcal{M}\mathcal{N}\left({\displaystyle \frac{{\tilde{\mathit{\mu }}}_{\tilde{\mathit{f}}}}{1+\frac{2\widetilde{v_{ϵ}}\tilde{\mathit{\alpha }}}{n({\tilde{\mathit{v}}}_{\tilde{\mathit{f}}}+{\tilde{\mathit{\mu }}}_{\tilde{\mathit{f}}}^2+2\tilde{\mathit{\beta }})}}},\mathrm{diag}\left[{\displaystyle \frac{\widetilde{v_{ϵ}}({\tilde{\mathit{v}}}_{\tilde{\mathit{f}}}+{\tilde{\mathit{\mu }}}_{\tilde{\mathit{f}}}^2+2\tilde{\mathit{\beta }})}{n({\tilde{\mathit{v}}}_{\tilde{\mathit{f}}}+{\tilde{\mathit{\mu }}}_{\tilde{\mathit{f}}}^2)+2n\tilde{\mathit{\beta }}+2\widetilde{v_{ϵ}}\tilde{\mathit{\alpha }}}}\right]\right),\\ \hfill \mathit{g}\sim \mathcal{N}({\tilde{\mathit{\mu }}}_{\tilde{\mathit{f}}},\widetilde{v_{ϵ}}\mathit{I}),\mathrm{and}{\tilde{v}}_k\sim \mathcal{I}\mathcal{G}\left({\tilde{\alpha }}_k,{\displaystyle \frac{{\tilde{v}}_{{\tilde{f}}_k}+{\tilde{\mu }}_{{\tilde{f}}_k}^2}{2}}+{\tilde{\beta }}_k\right),k=1,\cdots ,p.\end{array}$$

(20)

The corresponding $KL\left(\tilde{\mathit{\theta }}\right)$ for $\tilde{\mathit{\theta }}=({\tilde{\mathit{\mu }}}_{\mathit{g}},\widetilde{v_{ϵ}},{\tilde{\mathit{v}}}_{\tilde{\mathit{f}}},\tilde{\mathit{\alpha }},\tilde{\mathit{\beta }})$ is below and the details are available in Appendix E along with its gradient and other algorithms:

$$\begin{array}{cc}\hfill KL\left(\tilde{\mathit{\theta }}\right)\propto & -{\displaystyle \frac{1}{2}}{\displaystyle \sum _{j=1}^p}\ln \left({\tilde{v}}_{{\tilde{f}}_j}\right)+{\displaystyle \frac{n}{2\widetilde{v_{ϵ}}}}{\displaystyle \sum _{j=1}^p}({\tilde{\mu }}_{g_j}^2+{\tilde{v}}_{{\tilde{f}}_j})+{\displaystyle \sum _{j=1}^p}{\displaystyle \frac{{\tilde{\alpha }}_j{\tilde{v}}_{{\tilde{f}}_j}}{{\tilde{\beta }}_j}}-{\displaystyle \frac{1}{2}}{\displaystyle \sum _{j=1}^p}{\psi }_0\left({\tilde{\alpha }}_j\right)+{\displaystyle \frac{1}{2}}{\displaystyle \sum _{j=1}^p}\ln {\tilde{\beta }}_j.\hfill \end{array}$$

We choose a model to see the performance of these margins and compare them with gradient-based considering normal and natural parameters and natural gradient algorithms. The selected model is $\mathit{g}=\mathit{H}\mathit{f}+ϵ$ with the following knowledge:

$$\begin{array}{cc}\hfill \mathit{H}& =\mathit{I},\hfill \\ \hfill \mathit{f}& \sim \mathcal{M}\mathcal{N}(\mathit{f}|\mathbf{0},\mathrm{diag}\left[v_1,v_2\right]),\hfill \\ \hfill v_1& \sim \mathcal{I}\mathcal{G}(v_1|3,2)\hfill \\ \hfill v_2& \sim \mathcal{I}\mathcal{G}(v_2|4,3)\hfill \\ \hfill ϵ& \sim \mathcal{M}\mathcal{N}(ϵ|\mathbf{0},\mathit{I}).\hfill \end{array}$$

(21)

In the assessment procedure, we do not apply the above information. The outputs of algorithms are shown in Figure 5, as well as the actual contour plot. The $KL(·)$ for the four algorithms, starting from $108.91$, are $4.93$, $108.91$, $19.57$, $108.91$, and $108.91$, respectively. In this example, the best diagnosis is from the alternative algorithm with the minimum of $KL(·)$.

The objective of the inverse problem is to approximate the distribution of $\mathit{f}$. We use the distribution of $\mathit{g}$ in Figure 5 to show the number of method accuracies. In Figure 5, we can observe improved approximations for the second and last two plots. In these algorithms, the best choice is the initialization, and the distribution has a higher $KL(·)$ by reputation. Here, are the conjectures of the standard alternative, gradient-based with $\gamma =1,\parallel KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$ and natural parameters, and natural gradient algorithms, respectively:

$$\left\{\begin{array}{c}{\mathit{\mu }}_{\tilde{\mathit{f}}}=\left[\begin{array}{c}0.036\\ 0.118\end{array}\right],\hfill \\ \\ {\tilde{\mathsf{\Sigma }}}_{\tilde{\mathit{f}}}=\left[\begin{array}{cc}0.023& 0\\ 0& 0.022\end{array}\right],\hfill \end{array}\right.\left\{\begin{array}{c}{\mathit{\mu }}_{\tilde{\mathit{f}}}=\left[\begin{array}{c}0.038\\ 0.129\end{array}\right],\hfill \\ \\ {\tilde{\mathsf{\Sigma }}}_{\tilde{\mathit{f}}}=\left[\begin{array}{cc}2.459& 0\\ 0& 2.178\end{array}\right],\hfill \end{array}\right.\left\{\begin{array}{c}{\mathit{\mu }}_{\tilde{\mathit{f}}}=\left[\begin{array}{c}0\\ 0\end{array}\right],\hfill \\ \\ {\tilde{\mathsf{\Sigma }}}_{\tilde{\mathit{f}}}=\left[\begin{array}{cc}0.891& 0\\ 0& 0.523\end{array}\right],\hfill \end{array}\right.$$

$$\left\{\begin{array}{c}{\mathit{\mu }}_{\tilde{\mathit{f}}}=\left[\begin{array}{c}0.04\\ 0.13\end{array}\right],\hfill \\ \\ {\tilde{\mathsf{\Sigma }}}_{\tilde{\mathit{f}}}=\left[\begin{array}{cc}2.459& 0\\ 0& 2.178\end{array}\right],\hfill \end{array}\right.\left\{\begin{array}{c}{\mathit{\mu }}_{\tilde{\mathit{f}}}=\left[\begin{array}{c}0.04\\ 0.13\end{array}\right],\hfill \\ \\ {\tilde{\mathsf{\Sigma }}}_{\tilde{\mathit{f}}}=\left[\begin{array}{cc}2.459& 0\\ 0& 2.178\end{array}\right].\hfill \end{array}\right.$$

We simulate additional $\mathit{g}=\mathit{H}\mathit{f}+\mathit{ϵ}$ models with varying dimensions in

Table 4. We have additional simulation results for the model $\mathit{g}=\mathit{H}\mathit{f}+\mathit{ϵ}$ with 4, 6, and 10 dimensions. The initializations are calculated based on some guesses using available data $\mathit{g}$. Although we do not need the values of ${\mathit{\mu }}_{\mathit{f}}$ and ${\mathit{v}}_{\mathit{f}}$ for the algorithm implementations, we have included two columns for them. The purpose of inverse problems is to extract $\mathit{f}$ from the dataset of $\mathit{g}$. ${\mathit{v}}_{\mathit{f}}$ is a diagonal matrix, which we represent by its main diagonal components as a vector here.

Models	Algorithm↓		Data-Based Initializations
			Parameters					$\mathit{KL}(·)$
			$\mathit{\alpha }$	$\mathit{\beta }$	${\mathit{v}}_{\mathit{ϵ}}$	${\mathit{\mu }}_{\mathit{f}}$	${\mathit{v}}_{\mathit{f}}$
	Initial Points →		$\left[\begin{array}{c}3.08\\ 3.13\\ 2.7\\ 3.13\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$3.06$	−	−	$212.55$
	Alternative		$\left[\begin{array}{c}3.08\\ 3.13\\ 2.7\\ 3.13\end{array}\right]$	$\left[\begin{array}{c}3.92\\ 2.99\\ 8.17\\ 3.5\end{array}\right]$	$3.06$	$\left[\begin{array}{c}-0.02\\ -0.07\\ 0.36\\ -0.07\end{array}\right]$	$\left[\begin{array}{c}0.03\\ 0.03\\ 0.03\\ 0.03\end{array}\right]$	$12.37$
		$\gamma =1$	$\left[\begin{array}{c}3.08\\ 3.13\\ 2.7\\ 3.13\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$3.06$	$\left[\begin{array}{c}-0.02\\ -0.07\\ 0.37\\ -0.07\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$212.55$
$\mathit{\alpha }=\left[\begin{array}{c}7\\ 4\\ 3\\ 5\end{array}\right],\mathit{\beta }=\left[\begin{array}{c}6\\ 3\\ 8\\ 4\end{array}\right],v_{ϵ}=,{\mathit{\mu }}_{\mathit{f}}=\mathbf{0},{\mathit{v}}_{\mathit{f}}=\left[\begin{array}{c}1.23\\ 0.82\\ 3.23\\ 1.36\end{array}\right]$	Gradient	$\gamma =\parallel KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}3.01\\ 3.07\\ 2.6\\ 3.06\end{array}\right]$	$\left[\begin{array}{c}2.89\\ 2.22\\ 5.68\\ 2.59\end{array}\right]$	$6.5$	$\mathbf{0}$	$\left[\begin{array}{c}0.89\\ 0.24\\ 3.71\\ 0.6\end{array}\right]$	$46.37$
		Natural Parameters	$\left[\begin{array}{c}3.08\\ 3.13\\ 2.7\\ 3.13\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$3.06$	$\left[\begin{array}{c}-0.02\\ -0.07\\ 0.37\\ -0.07\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$212.55$
	Natural Gradient		$\left[\begin{array}{c}3.08\\ 3.13\\ 2.7\\ 3.13\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$3.06$	$\left[\begin{array}{c}-0.02\\ -0.07\\ 0.37\\ -0.07\end{array}\right]$	$\left[\begin{array}{c}2.6\\ 1.98\\ 5.35\\ 2.32\end{array}\right]$	$212.55$
	Initial Points →		$\left[\begin{array}{c}3.58\\ 3.64\\ 3.91\\ 3.41\\ 3.65\\ 3.52\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$3.67$	−	−	$320.4$
	Alternative		$\left[\begin{array}{c}3.58\\ 3.64\\ 3.91\\ 3.41\\ 3.65\\ 3.52\end{array}\right]$	$\left[\begin{array}{c}2.3\\ 4.36\\ 7.23\\ 11.43\\ 4.32\\ 3.63\end{array}\right]$	$3.67$	$\left[\begin{array}{c}0.08\\ 0.03\\ -0.23\\ 0.26\\ 0.02\\ 0.13\end{array}\right]$	$\left[\begin{array}{c}0.03\\ 0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\end{array}\right]$	$16.38$
		$\gamma =1$	$\left[\begin{array}{c}3.58\\ 3.64\\ 3.91\\ 3.41\\ 3.65\\ 3.52\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$3.67$	$\left[\begin{array}{c}0.09\\ 0.03\\ -0.24\\ 0.26\\ 0.02\\ 0.14\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$320.4$
$\mathit{\alpha }=\left[\begin{array}{c}6\\ 4\\ 3\\ 2\\ 3\\ 5\end{array}\right],\mathit{\beta }=\left[\begin{array}{c}4\\ 5\\ 6\\ 8\\ 3\\ 7\end{array}\right],v_{ϵ}=,{\mathit{\mu }}_{\mathit{f}}=\mathbf{0},{\mathit{v}}_{\mathit{f}}=\left[\begin{array}{c}0.73\\ 1.91\\ 3.61\\ 7.41\\ 1.3\\ 1.41\end{array}\right]$	Gradient	$\gamma =\parallel KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}3.52\\ 3.56\\ 3.82\\ 3.32\\ 3.57\\ 3.46\end{array}\right]$	$\left[\begin{array}{c}1.73\\ 3.21\\ 5.16\\ 7.92\\ 3.18\\ 2.68\end{array}\right]$	$7.58$	$\left[\begin{array}{c}0.01\\ 0\\ -0.01\\ 0.02\\ 0\\ 0.01\end{array}\right]$	$\left[\begin{array}{c}0.27\\ 1.72\\ 3.63\\ 6.46\\ 1.69\\ 1.2\end{array}\right]$	$108.9$
		Natural Parameters	$\left[\begin{array}{c}3.58\\ 3.64\\ 3.91\\ 3.41\\ 3.65\\ 3.52\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$3.67$	$\left[\begin{array}{c}0.09\\ 0.03\\ -0.24\\ 0.26\\ 0.02\\ 0.14\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$320.4$
	Natural Gradient		$\left[\begin{array}{c}3.58\\ 3.64\\ 3.91\\ 3.41\\ 3.65\\ 3.52\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$3.67$	$\left[\begin{array}{c}0.09\\ 0.03\\ -0.24\\ 0.26\\ 0.02\\ 0.14\end{array}\right]$	$\left[\begin{array}{c}1.52\\ 2.89\\ 4.77\\ 7.56\\ 2.86\\ 2.39\end{array}\right]$	$320.4$
	Initial Points →		$\left[\begin{array}{c}3.89\\ 3.41\\ 3.66\\ 3.76\\ 3.55\\ 3.9\\ 3.88\\ 3.8\\ 3.51\\ 3.95\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$3.8$	−	−	$536.28$
	Alternative		$\left[\begin{array}{c}3.89\\ 3.41\\ 3.66\\ 3.76\\ 3.55\\ 3.9\\ 3.88\\ 3.8\\ 3.51\\ 3.95\end{array}\right]$	$\left[\begin{array}{c}3.08\\ 5.85\\ 3.44\\ 11.04\\ 19.46\\ 4.38\\ 2.06\\ 2.01\\ 2.13\\ 4.09\end{array}\right]$	$3.8$	$\left[\begin{array}{c}-0.08\\ 0.38\\ 0.13\\ 0.04\\ 0.25\\ -0.1\\ -0.07\\ 0\\ 0.26\\ -0.14\end{array}\right]$	$\left[\begin{array}{c}0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\\ 0.04\end{array}\right]$	$27.43$
		$\gamma =1$	$\left[\begin{array}{c}3.89\\ 3.41\\ 3.66\\ 3.76\\ 3.55\\ 3.9\\ 3.88\\ 3.8\\ 3.51\\ 3.95\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$3.8$	$\left[\begin{array}{c}-0.09\\ 0.39\\ 0.14\\ 0.04\\ 0.25\\ -0.1\\ -0.08\\ 0\\ 0.29\\ -0.15\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$536.28$
$\mathit{\alpha }=\left[\begin{array}{c}6\\ 2\\ 3\\ 2\\ 1\\ 5\\ 5\\ 4\\ 4\\ 3\end{array}\right],\mathit{\beta }=\left[\begin{array}{c}4\\ 5\\ 2\\ 6\\ 3\\ 7\\ 2\\ 1\\ 1\\ 3\end{array}\right],v_{ϵ}=,{\mathit{\mu }}_{\mathit{f}}=\mathbf{0},{\mathit{v}}_{\mathit{f}}=\left[\begin{array}{c}0.91\\ 3.51\\ 1.2\\ 6.76\\ 12.4\\ 2.2\\ 0.41\\ 0.33\\ 0.36\\ 1.27\end{array}\right]$	Gradient	$\gamma =\parallel KL\left(\tilde{\mathit{\theta }}\right){\parallel }^{-1}$	$\left[\begin{array}{c}3.81\\ 3.29\\ 3.57\\ 3.62\\ 3.41\\ 3.8\\ 3.84\\ 3.76\\ 3.47\\ 3.85\end{array}\right]$	$\left[\begin{array}{c}2.41\\ 4.25\\ 2.65\\ 7.94\\ 13.52\\ 3.38\\ 1.58\\ 1.53\\ 1.55\\ 3.16\end{array}\right]$	$10.04$	$\left[\begin{array}{c}-0\\ 0.02\\ 0.01\\ 0\\ 0.01\\ 0\\ 0\\ 0\\ 0.01\\ -0.01\end{array}\right]$	$\left[\begin{array}{c}0.51\\ 2.38\\ 0.77\\ 5.99\\ 11.59\\ 1.43\\ 0.05\\ 0.19\\ 0\\ 1.22\end{array}\right]$	$136.63$
		Natural Parameters	$\left[\begin{array}{c}3.89\\ 3.41\\ 3.66\\ 3.76\\ 3.55\\ 3.9\\ 3.88\\ 3.8\\ 3.51\\ 3.95\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$3.8$	$\left[\begin{array}{c}-0.09\\ 0.39\\ 0.14\\ 0.04\\ 0.25\\ -0.1\\ -0.08\\ 0\\ 0.29\\ -0.15\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$536.28$
	Natural Gradient		$\left[\begin{array}{c}3.89\\ 3.41\\ 3.66\\ 3.76\\ 3.55\\ 3.9\\ 3.88\\ 3.8\\ 3.51\\ 3.95\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$3.8$	$\left[\begin{array}{c}-0.09\\ 0.39\\ 0.14\\ 0.04\\ 0.25\\ -0.1\\ -0.08\\ 0\\ 0.29\\ -0.15\end{array}\right]$	$\left[\begin{array}{c}2.04\\ 3.79\\ 2.27\\ 7.35\\ 12.92\\ 2.9\\ 1.36\\ 1.33\\ 1.35\\ 2.7\end{array}\right]$	$536.28$

. Based on the results, the optimal posterior is obtained from the standard alternative algorithm when we increase the dimensions, as it exhibits lower values of $KL(·)$. In the inverse problem, the primary focus is on observing $\mathit{f}$, so another way to compare could be the accuracy of ${\mathit{v}}_{\mathit{f}}$. When it comes to the variance-covariance matrix of $\mathit{f}$, the most effective algorithm is parametric natural gradient optimization for approximating the posterior distribution of $\mathit{f}$ because it yields results that closely align with the data of $\mathit{f}$. We would like to remind you that we do not use $\mathit{f}$ data in the recursive processes; we only have $\mathit{g}$ data. It is applicable to real-world situations, but since this is a simulated example, we have the $\mathit{f}$ data, but we do not utilize it. In the example above, we observe that the alternative method yields a lower $KL(·)$ in the two-dimensional dataset, too. As the number of dimensions increases, so does the value of $KL(·)$. However, the accuracy of the variance-covariance of $\mathit{f}$ is not reduced.

6. Conclusions

This paper presents four approximation methods for estimating the density functions of hidden variables, referred to as VBA. We also consider three examples of the Normal-Inverse-Gamma, Normal-Inverse-Wishart, and linear inverse problems. We provide the details of the first model here and include the details for two other examples in the appendices. In all three models, the parameters are unobserved and estimated using recursive algorithms. We attempt to approximate the joint complex distribution by simplifying the margin factorials to resemble independent cases. We compare the performance and accuracy of VBA algorithms. The standard alternative analytic optimization algorithm demonstrates the highest robustness in minimizing KLD, particularly as the number of dimensions increases, and converges fairly quickly. The numerical computation cost is negligible in our examples because we find the explicit form for each parameter, and the algorithms are well-formulated. The main difference in algorithms lies in the accuracy of the results. They estimate the complex joint distribution using separable ones. In the linear inverse problem, the standard alternative algorithm yields lower $KL(·)$ values, while the parametric gradient algorithm with $\gamma ={\parallel KL(·)\parallel }^{-1}$ produces variance-covariance matrices that are closest to the real ones. The overall performance of the alternative is most satisfactory, especially in high dimensions.