Dynamics of Coordinate Ascent Variational Inference: A Case Study in 2D Ising Models

Abstract

Variational algorithms have gained prominence over the past two decades as a scalable computational environment for Bayesian inference. In this article, we explore tools from the dynamical systems literature to study the convergence of coordinate ascent algorithms for mean field variational inference. Focusing on the Ising model defined on two nodes, we fully characterize the dynamics of the sequential coordinate ascent algorithm and its parallel version. We observe that in the regime where the objective function is convex, both the algorithms are stable and exhibit convergence to the unique fixed point. Our analyses reveal interesting discordances between these two versions of the algorithm in the region when the objective function is non-convex. In fact, the parallel version exhibits a periodic oscillatory behavior which is absent in the sequential version. Drawing intuition from the Markov chain Monte Carlo literature, we empirically show that a parameter expansion of the Ising model, popularly called the Edward–Sokal coupling, leads to an enlargement of the regime of convergence to the global optima.

1. Introduction

Variational Bayes (VB) is now a standard tool to approximate computationally intractable posterior densities. Traditionally this computational intractability has been circumvented using sampling techniques such as Markov chain Monte Carlo (MCMC). MCMC techniques are prone to be computationally expensive for high dimensional and complex hierarchical Bayesian models, which are prolific in modern applications. VB methods, on the other hand, typically provide answers orders of magnitude faster, as they are based on optimization. Introduction to VB can be found in chapter 10 of [1] and chapter 33 of [2]. Excellent recent surveys can be found in [3,4].

The objective of VB is to find the best approximation to the posterior distribution from a more tractable class of distributions on the latent variables that is well-suited to the problem at hand. The best approximation is found by minimizing a divergence between the posterior distribution of interest and a class of distributions that are computationally tractable. The most popular choices for the discrepancy and the approximating class are the Kullback–Leibler (KL) divergence and the class of product distributions, respectively. This combination is popularly known as mean field variational inference, originating from mean field theory in physics [5]. Mean-field inference has percolated through a wide variety of disciplines, including statistical mechanics, electrical engineering, information theory, neuroscience, cognitive sciences [6] and more recently deep neural networks [7]. While computing the KL divergence is intractable for a large class of distributions, reframing the minimization problem for maximizing the evidence lower bound (ELBO) leads to efficient algorithms. In particular, for conditionally conjugate-exponential family models, the optimal distribution for mean field variational inference can be computed by iteration of closed form updates. These updates form a coordinate ascent algorithm known as coordinate ascent variational inference (CAVI) [1].

Research into the theoretical properties of variational Bayes has exploded in the last few years. Recent theoretical work focuses on statistical risk bounds for variational estimate obtained from VB [8,9,10,11], asymptotic normality of VB posteriors [12] and extension to model misspecification [8,13]. While much of the recent theoretical work focuses on statistical optimality guarantees, there has been less work studying the convergence of the CAVI algorithms employed in practice. Convergence of CAVI to the global optima is only known in special cases that depend heavily on model structure for normal mixture models [14,15]; stochastic block models [16,17,18,19]; topic models [20]; and under special restrictions of the parameter regime, Ising models [21,22]. The convergence properties of the CAVI algorithm still largely constitute an open problem.

The goal of this work is to suggest a general systematic framework for studying convergence properties of CAVI algorithms. By viewing CAVI as a discrete time dynamical system, we can leverage dynamical systems theory to analyze the convergence behavior of the algorithm and bifurcation theory to study the types of changes that solutions can undergo as the various parameters are varied. For sake of concreteness, we focus on the 2D Ising model. While dynamical systems theory possesses the tools [23,24,25] necessary to analyze higher dimensional systems, they were mainly developed for non-sequential systems. The general theory for n-dimensional discrete dynamical systems is dependent on having the evolution function in the form $x_{n+1}=F\left(x_n\right)$. Deriving this F is typically not possible for densely connected higher dimensional sequential systems. The 2D Ising model has the special property that both the sequential and parallel updates in the two variables case can be written as two separate one variable dynamical systems, allowing for a simplified analysis. Our contributions to the literature are as follows: We provide a complete classification of the dynamical properties of the the traditional sequential update CAVI algorithm, and a parallelized version of the algorithm using dynamical systems and bifurcation theory on the Ising models. Our findings show that the sequential CAVI algorithm and the parallelized version have different convergence properties. Additionally, we numerically investigated the convergence of the CAVI algorithm on the Edward–Sokal coupling, a generalization of the Ising model. Our findings suggest that couplings/parameter expansion may provide a powerful way of controlling the convergence behavior of the CAVI algorithm, beyond the immediate example considered here.

2. Mean-Field Variational Inference and the Coordinate Ascent Algorithm

In this section, we briefly introduce mean-field variational inference for a target distribution in the form of a Boltzmann distribution with potential function $\mathsf{\Psi }$,

$$\begin{array}{c}\hfill p\left(\mathbf{x}\right)=\frac{exp\left\{\mathsf{\Psi }\right(\mathbf{x}\left)\right\}}{\mathcal{Z}},\mathbf{x}\in \mathcal{X},\end{array}$$

where $\mathcal{Z}$ denotes the intractable normalizing constant. The above representation encapsulates both posterior distributions that arise in Bayesian inference, where $\mathsf{\Psi }$ is the log-posterior up to constants, and probabilistic graphical models such as the Ising and Potts models. For instance, $\mathsf{\Psi }\left(x\right)=\beta {\sum }_{u\sim v}{J}_{uv}{x}_u{x}_v+\beta {\sum }_u{h}_u{x}_u$ for the Ising model; see the next section for more details. Many of the complications in inference arise from the intractability of the normalizing constant $\mathcal{Z}$, which is commonly referred to as the free energy in probabilistic graphical models, and the marginal likelihood or evidence in Bayesian statistics. Variational inference aims to mitigate this problem by using optimization to find the best approximation $q^{*}$ to the target density p from a class $\mathcal{F}$ of variational distributions over the parameter vector $\mathbf{x}$,

$$\begin{array}{c}\hfill q^{*}=arg\underset{q\in \mathcal{F}}{min}D\left(q\right|\left|p\right)\end{array}$$

(1)

where $D\left(q\right|\left|p\right)$ denotes the Kullback–Leibler (KL) divergence between q and p. The complexity of this optimization problem is largely determined by the choice of variational family $\mathcal{F}$. The objective function of the above optimization problem is intractable because it also involves the evidence $\mathcal{Z}$. We can work around this issue by rewriting the KL divergence as

$$\begin{array}{c}\hfill D\left(q\right|\left|p\right)={\mathbb{E}}_q[logq]-{\mathbb{E}}_q\left[\mathsf{\Psi }\right]+log\mathcal{Z}\end{array}$$

(2)

where ${\mathbb{E}}_q$ denotes the expectation with respect to $q\left(\mathbf{x}\right)$. Rearranging terms,

$$\begin{array}{cc}\hfill log\mathcal{Z}& =D\left(q\right|\left|p\right)+{\mathbb{E}}_q\left[\mathsf{\Psi }\right]-{\mathbb{E}}_q[logq]\end{array}$$

(3)

$$\begin{array}{cc}& \ge {\mathbb{E}}_q\left[\mathsf{\Psi }\right]-{\mathbb{E}}_q[logq]:=\mathrm{ELBO}\left(q\right).\end{array}$$

(4)

The acronym ELBO stands for evidence lower bound and the nomenclature is now apparent from the above inequality. Notice from Equation (2) that maximizing the ELBO is equivalent to minimizing the KL divergence. By maximizing the ELBO we can solve the original variational problem while by-passing the computational intractability of the evidence.

As mentioned above, the choice of variational family controls both the complexity and accuracy of approximation. Using a more flexible family achieves a tighter lower bound but at the cost of having to solve a more complex optimization problem. A popular choice of family that balances both flexibility and computability is the mean-field family. Mean-field variational inference refers to the situation when q is restricted to the product family of densities over the parameters,

$$\begin{array}{c}\hfill {\mathcal{F}}_{\mathrm{MF}}:=\left\{q\left(\mathbf{x}\right)=q_1\left(x_1\right)\otimes \cdots \otimes q_n\left(x_n\right)\mathrm{for}\mathrm{probability}\mathrm{measures}{q}_j,j=1,\dots ,n\right\},\end{array}$$

(5)

The coordinate ascent variational inference (CAVI) algorithm (refer to Algorithm 1) is a learning algorithm that optimizes the ELBO over the mean-field family ${\mathcal{F}}_{\mathrm{MF}}$. At each time step $t\ge 1$, the CAVI algorithm iteratively updates the current mean field marginal distribution $q_j^{\left(t\right)}\left(x_j\right)$ by maximizing the ELBO over that marginal while keeping the other marginals ${\left\{q_{\mathsf{\ell }}^{\left(t\right)}\left(x_{\mathsf{\ell }}\right)\right\}}_{\mathsf{\ell }\ne j}$ fixed at their current values. Formally, we update the current distribution $q^{\left(t\right)}\left(\mathbf{x}\right)$ to $q^{(t+1)}\left(\mathbf{x}\right)$ by the updates,

$$\begin{array}{ccc}\hfill q_1^{(t+1)}\left(x_1\right)& =& arg\underset{q_1}{max}\mathrm{ELBO}(q_1\otimes q_2^{\left(t\right)}\otimes \cdots \otimes q_n^{\left(t\right)})\hfill \\ \hfill q_2^{(t+1)}\left(x_2\right)& =& arg\underset{q_2}{max}\mathrm{ELBO}(q_1^{(t+1)}\otimes q_2\otimes q_3^{\left(t\right)}\otimes \cdots \otimes q_n^{\left(t\right)})\hfill \\ & ⋮& \\ \hfill q_n^{(t+1)}\left(x_n\right)& =& arg\underset{q_n}{max}\mathrm{ELBO}(q_1^{(t+1)}\otimes \cdots \otimes q_{n-1}^{(t+1)}\otimes q_n).\hfill \end{array}$$

Algorithm 1 Coordinate ascent variational inference (CAVI).

The objective function $\mathrm{ELBO}(q_1\otimes \cdots \otimes q_n)$ is concave in each of the arguments individually (although it is rarely jointly concave), so these individual maximization problems have unique solutions. The optimal update for the jth mean field variational component of the model has the closed form,

$$\begin{array}{ccc}\hfill q_j^{*}\left({\mathbf{x}}_j\right)& \propto & exp\left\{{\mathbb{E}}_{-j}\left[\mathsf{\Psi }\left(\mathbf{x}\right)\right]\right\}\hfill \end{array}$$

where the expectations ${\mathbb{E}}_{-j}$ are taken with respect to the distribution ${\prod }_{i\ne j}{q}_i\left({\mathbf{x}}_i\right)$. Furthermore, the update step of the algorithm is monotonous, as each step of the CAVI increases the objective function

$$\begin{array}{cc}\hfill \mathrm{ELBO}(q_1^{(t+1)}\otimes q_2^{(t+1)}\otimes \cdots \otimes q_n^{(t+1)})& \ge \mathrm{ELBO}(q_1^{(t+1)}\otimes q_2^{(t+1)}\otimes \cdots \otimes q_{n-1}^{(t+1)}\otimes q_n^{\left(t\right)})\ge \cdots \ge \\ & \mathrm{ELBO}(q_1^{\left(t\right)}\otimes q_2^{\left(t\right)}\otimes \cdots \otimes q_n^{\left(t\right)}).\end{array}$$

For parametric models, the sequential updates of the variational marginal distributions in the CAVI algorithm is done by a sequential update of the variational parameters of these distributions. The CAVI algorithm updates for parametric models induce a discrete time dynamical system of the parameters. Clearly, convergence of the CAVI algorithm can be framed in terms of this induced discrete time dynamical system. As discussed before, the ELBO is generally a non-convex function, and hence the CAVI algorithm is only guaranteed to converge to a local optimum of the system. It is also not clear how many local optima (or fixed points) the system has, nor whether the algorithm always settles on a single fixed point, diverges away from the fixed point or cycles between multiple fixed points. These questions translate to questions about the existence and stability of fixed points of the induced dynamical system. We are also interested in how the behavior of the CAVI algorithm could possibly change as we vary the parameters of the model. This translates to questions about the possible bifurcations of the induced dynamical system. In Section 3, we formally introduce the Ising model and its mean-field variational inference.

3. CAVI in Ising Model

We first briefly review the definition of an Ising model. The Ising model was first introduced as a model for magnetization in statistical physics, but has found many applications in other fields; see [26] and references therein. The Ising model is a probability distribution on the hypercube ${\{\pm 1\}}^n$ given by

$$\begin{array}{ccc}\hfill p\left(\mathbf{x}\right)& \propto & exp\left[\beta \sum _{u\sim v}{J}_{uv}{x}_u{x}_v+\beta \sum _u{h}_u{x}_u\right],\hfill \end{array}$$

(6)

where the interaction matrix J is a symmetric real $n\times n$ matrix with zeros on the diagonal, h is a real n-vector that represents the external magnetic field, and $\beta$ is the inverse temperature parameter. The model is said to be ferromagnetic if $J_{uv}\ge 0$ for all $u,v$ and anti-ferromagnetic if $J_{uv}<0$ for all $u,v$. The normalizing constant or the partition function of the Ising model is

$$\begin{array}{ccc}\hfill \mathcal{Z}& =& \sum _{\mathbf{x}\in {\{\pm 1\}}^n}exp\left[\beta \sum _{u\sim v}{J}_{uv}{x}_u{x}_v+\beta \sum _u{h}_u{x}_u\right].\hfill \end{array}$$

Refer to Chapter 31 of [2] for an excellent review of Ising models.

Mean Field Variational Inference in Ising Model

Here we provide a derivation of the CAVI update function for the Ising model, focusing on the two nodes ($n=2$) case for simplicity and analytic tractability.

Notice $logp\left(\mathbf{x}\right):=\beta \mathit{H}\left(\mathbf{x}\right)=\beta {\sum }_{u\sim v}{J}_{uv}{x}_u{x}_v+\beta {\sum }_u{h}_u{x}_u$. In this case, we have the Ising model on two spins with $\mathbf{x}=(x_1,x_2)$ and influence matrix J with off diagonal term $J_{12}$ and external magnetic field $h=(h_1,h_2)=(0,0)$. From the general framework in Section 2, the CAVI updates are given by,

$$q_j^{*}\left(x_j\right)\propto exp\left\{{\mathbb{E}}_{-j}\left[\beta (J_{12}{x}_1{x}_2+h_1{x}_1+h_2{x}_2)\right]\right\}.$$

Equivalently, the same updates are obtained by setting the gradient of the ELBO as a function of $(x_1,x_2)$ equal to the ${(0,0)}^{\prime }$ vector. Illustrations of the ELBO and the gradient functions for various values of $\beta$ are in Figure 1 and Figure 2 respectively.

Since $q_1^{*}$ and $q_2^{*}$ are two point distributions, it is sufficient to keep track of the mass assigned to 1. Simplifying,

$$\begin{array}{ccc}\hfill q_1^{*}\left(x_1\right)& \propto & exp\left\{{\mathbb{E}}_2\left[logp(x_1,x_2)\right]\right\}\hfill \\ & =& exp\left\{\beta \mathit{H}(x_1,x_2=1)q_2(x_2=1)+\beta \mathit{H}(x_1,x_2=-1)q_2(x_2=-1)\right\}\hfill \\ & =& exp\left\{(\beta J_{12}{x}_1+\beta h_1{x}_1+\beta h_2)\xi +(-\beta J_{12}{x}_1+\beta h_1{x}_1-\beta h_2)(1-\xi )\right\}\hfill \\ & =& exp\left\{(2\xi -1)(\beta J_{12}{x}_1+\beta h_2)+\beta h_1{x}_1\right\},\hfill \end{array}$$

where $\xi =q_2(x_2=1)$. Therefore

$$\begin{array}{ccc}\hfill q_1^{*}(x_1=1)& =& \frac{exp\left\{(2\xi -1)(\beta J_{12}+\beta h_2)+\beta h_1\right\}}{exp\left\{(2\xi -1)(\beta J_{12}+\beta h_2)+\beta h_1\right\}+exp\left\{(2\xi -1)(-\beta J_{12}+\beta h_2)-\beta h_1\right\}}\hfill \\ & =& \frac{1}{1+exp\left\{-2\beta J_{12}(2\xi -1)-2\beta h_1\right\}}.\hfill \end{array}$$

Similarly denoting $\zeta =q_1(x_1=1)$,

$$\begin{array}{ccc}\hfill q_2^{*}(x_2=1)& =& \frac{exp\left\{(2\zeta -1)(\beta J_{12}+\beta h_1)+\beta h_2\right\}}{exp\left\{(2\zeta -1)(\beta J_{12}+\beta h_1)+\beta h_2\right\}+exp\left\{(2\zeta -1)(-\beta J_{12}+\beta h_1)-\beta h_2\right\}}\hfill \\ & =& \frac{1}{1+exp\left\{-2\beta J_{12}(2\zeta -1)-2\beta h_2\right\}}.\hfill \end{array}$$

Let ${\zeta }_k$ (resp. ${\xi }_k$) denote the kth iterate of $q_1(x_1=1)$ (resp. $q_2(x_2=1)$) from the CAVI algorithm. To succinctly represent these updates, define the logistic sigmoid function

$$\begin{array}{c}\hfill \sigma (u,\beta )=\frac{1}{1+e^{-\beta u}},u\in [0,1],\beta \in \mathbb{R}.\end{array}$$

(7)

With this notation, we have, for any $k\in {\mathbb{Z}}_{+}$,

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\zeta }_{k+1}& =\sigma (J_{12}(2{\xi }_k-1)+h_1,2\beta )\hfill \\ \hfill {\xi }_{k+1}& =\sigma (J_{12}(2{\zeta }_{k+1}-1)+h_2,2\beta ).\hfill \end{array}\end{array}$$

(8)

Without loss of generality we henceforth set $J_{12}=1$. Under this choice the model is in the ferromagnetic regime for $\beta >0$ and the anti-ferromagnetic regime for $\beta <0$.

4. Why the Ising Model: A Summary of Our Contributions

There are exactly two cases of the Ising model that have a full analytic solution for the free energy. They are (i) the one dimensional line graph solved by Ernst Ising in his thesis [27] and (ii) the two dimensional case on the anisotropic square lattice when the magnetic field $h=0$ by [28]. Comparison with the mean field solution for the same models highlights the poor approximation quality of the mean field solution in low dimensions. To the best knowledge of the authors, there are no results in the literature detailing the properties of the mean field solution to the anti-ferromagnetic Ising model. Readers not familiar with the physics may wonder why this is the case. To explain this, there are two cases in the anti-ferromagnetic regime: one of the two regions is equivalent to the ferromagnetic case and in the other the mean field approximation is not a good approximation of the system. The first case occurs in a bipartite graph where a transformation of variables makes the antiferromagnetic regime equivalent to the ferromagnetic one [29]. The other case can be seen on the triangle graph. By fixing the spin of one vertex as 1 and the other as $-1$, the third vertex becomes geometrically frustrated and neither choice of spin lowers the energy level of the system and the two configurations are equivalent [30]. In this case the mean field approximation gives a completely incorrect answer and does not merit further investigation from a qualitative point of view. The physics literature is primarily concerned with using the mean field solutions to the Ising model to estimate important physical constants of the systems. These constants are only meaningful when the mean field solution provides a good approximation to the behavior of the system in large dimensions. It is known, however, that under certain conditions the mean field approximation does indeed converge to the true free energy of the system as the dimension increases [21,31].

Our work is focused on providing a rigorous methodology to analyze dynamics of the CAVI algorithm that can be applied to any model structure. All of the interesting behaviors exhibited by the CAVI algorithm fit into the classical mathematical framework of discrete dynamical systems and bifurcation theory. Specifically, we use the Ising model as a simple and yet rich example to illustrate the potential of dynamical systems theory to analyze CAVI updates for mean field variational inference. The bifurcation of the ferromagnetic Ising model at the boundary of the Dobrushin regime is known [2,26]; however, a rigorous proof in terms of dynamical systems theory is missing in the literature.

There are several features that make the CAVI algorithm on the Ising model a nontrivial example worth investigating. The optimization problem arising from mean field variational inference on the Ising model is, in general, non-convex [21]. However, it is straightforward to obtain sufficient conditions to guarantee the existence of a global optima. One such condition is that the inverse temperature $\beta$ is inside the Dobrushin regime, $\left|\beta \right|<1$ [21]. Inside the Dobrushin regime, the CAVI update equations form a contraction mapping guaranteeing a unique global optima [21]. Outside of this regime the behavior of the CAVI algorithm is nontrivial. The CAVI solution to the Ising model with zero external magnetic field exhibits multiple local optima outside of the Dobrushin regime [2].

Our contributions to the literature are as follows. We utilize tools from dynamical systems theory to rigorously classify the full behavior of Ising model for the full parameter regime in dimension $n=2$ for both the sequential and parallel versions of CAVI algorithm. We show that the dynamical behavior of the sequential CAVI is not equivalent to the behavior of the parallel CAVI. Lastly we derive a variational approximation to the Edward-Sokal parameter expansion of the Potts and Random Cluster models and numerically study its convergence behavior under the CAVI algorithm. Our numerical results reveal that the parameter expansion leads to an enlargement of the regime of convergence to the global optima. In particular the Dobrushin regime is strictly contained in the expanded regime. This is compatible with the analogous results in Markov chain literature. See the introduction of [32] for a well written summary of Markov chain mixing in the Ising model.

Statistical Significance of Our Results

Although mean-field variational inference has been routinely used in applications [3] for computational efficiency, it may not yield statistically optimal estimators. A statistically optimal estimator should correctly recover the statistical properties of the true distribution. Ideally, we would like the estimate to recover the true mean and true covariance of the distribution. It is well known that mean-field variational inference produces estimators that underestimate the posterior covariance [14]. More recently, it was shown that the mean-field estimators for certain topic models and stochastic block models may not even be correlated with the true distribution [17,20]. For these reasons, it is important to see if the mean field estimators can at least recover the true mean for various $\beta \in \mathbb{R}$.

Mean field inference approximates the joint probability mass function in (6) for $n=2$ by product of two distributions on $\{-1,1\}$ in the sense of Kullback–Leibler divergence. As discussed in Section 3, minimizing this divergence is equivalent to maximizing an objective function, called the Evidence Lower Bound (ELBO). Our objective is to better understand the relation between the CAVI estimate and the global maximum of ELBO in (6) when $n=2$ and $h=0$. Ideally, we want the global maximum of the ELBO to be a statistically reliable estimate. To understand this, let us denote $2\times \mathrm{Bernoulli}\left(p\right)-1$ by $\langle 1,-1;p\rangle$. As the marginal distributions of (6) are both equal to $\langle 1,-1;0.5\rangle$, we want the ELBO to be maximized at this value. From an algorithmic perspective, we would like to ensure that the CAVI iterates converge to this global maximum. The synergy of these two phenomena leads to a successful variational inference method. We show in this article that both these conditions can be violated in a certain regime of the parameter space in the context of Ising model on two nodes. Inside the Dobrushin regime ($-1\le \beta \le 1$), the global optima of the ELBO obtained from a mean field inference occurs at $(\langle 1,-1;0.5\rangle ,\langle 1,-1;0.5\rangle )$ which is qualitatively the optimal solution. In this regime, the CAVI system converges to this global optimum irrespective of where the system is initialized. Thus, in the Dobrushin regime, the mean field inference yields the statistically optimal estimate. Additionally, the CAVI algorithm is stable and convergent at this value. Unfortunately, this property deteriorates outside of the Dobrushin regime. Outside of the regime, the global maxima occur at two symmetric points which are different from $(\langle 1,-1;0.5\rangle ,\langle 1,-1;0.5\rangle )$. These two symmetric points are equivalent under label switching. For example, when $\beta =1.2$ one of the optima is $(\langle 1,-1;0.17071\rangle ,\langle 1,-1;0.17071\rangle )$ and the other is $(\langle 1,-1;0.82928\rangle ,\langle 1,-1;0.82928\rangle )$. Notice this second optima is equivalent to the sign swapped version $(\langle -1,1;0.17071\rangle ,\langle -1,1;0.17071\rangle )$.

The original optima $(\langle 1,-1;0.5\rangle ,\langle 1,-1;0.5\rangle )$ is actually a local minimum of the ELBO outside the Dobrushin regime. We illustrate in our theory that the CAVI system returns one of two global maxima of the objective function depending on the initialization of the algorithm. Although it is widely known that the statistical quality of the mean field inference is poor outside the regime, we show in addition that the algorithm itself exhibits erratic behavior and may not converge to the global maximizer of the ELBO for all initializations. Interestingly, outside the Dobrushin regime, the statistically optimal solution $(\langle 1,-1;0.5\rangle ,\langle 1,-1;0.5\rangle )$ is a repelling fixed point of the CAVI system. This means that as the system is iterated, the current value of the system is pulled away from $(\langle 1,-1;0.5\rangle ,\langle 1,-1;0.5\rangle )$ to the global maximum.

A common technique to further improve computational time is the use of block updates in the CAVI algorithm, meaning groups of parameters are updated simultaneously. We refer to this as the parallelized CAVI algorithm. This has been shown to work well in certain models [17], but has not been investigated in a general setting. However, it turns out that block updating in the Ising model can lead to new problematic behaviors. Outside the Dobrushin regime, block updates can exhibit non-convergence in the form of cycling. As the system updates, it eventually switches back and forth between two points that yield the same value in the objective function.

Parameter expansions (coupling) is another method of improving the convergence properties of algorithms. In the Markov chain theory for Ising models, it is well-known that mixing and convergence time are typically improved by using the Edward–Sokal coupling, a parameter expansion of the ferromagnetic Ising model [33]. Our preliminary investigation reveals that the convergence properties of the CAVI algorithm also exhibit a similar phenomenon.

5. Main Results

In this section, we analyze the behavior of the dynamical systems that one can form using the CAVI update equations and show that the behaviors of the systems differ. Our results are heavily dependent on well-known techniques in dynamical systems. For readers unfamiliar with some of technical terminology below, we have included a primer on the basics of dynamical systems in Appendix A.

Recall the system of sequential updates, which are the updates used in CAVI:

$$\begin{array}{c}\hfill {\zeta }_{k+1}=\sigma (2{\xi }_k-1,2\beta ),{\xi }_{k+1}=\sigma (2{\zeta }_{k+1}-1,2\beta ),\end{array}$$

(9)

and the parallel updates:

$$\begin{array}{c}\hfill {\zeta }_{k+1}=\sigma (2{\xi }_k-1,2\beta ),{\xi }_{k+1}=\sigma (2{\zeta }_k-1,2\beta ).\end{array}$$

(10)

We will show that these two systems are not topologically conjugate. We first state and prove some results on the dynamics of the sigmoid function (7). These results will be used as building blocks to study the dynamics of (9) and (10). Phase change behavior of dynamical systems using the sigmoid and RELU activation functions are known in the literature in the context of generalization performance of deep neural networks [34,35]. In this section we present a complete proof of the bifurcation analysis of non-linear dynamical systems involving sigmoid activation function despite its connections with [34,35]. Our results in Section 5.1 provide a more complete picture of the behavior of the dynamics in all regimes and can be readily exploited to analyze the dynamics of (9) and (10).

5.1. Sigmoid Function Dynamics

In this section we provide a full classification for the dynamics of the following sigmoid function and its second iterate,

$$\begin{array}{c}\hfill \sigma (2x-1,2\beta ),\end{array}$$

(11)

$$\begin{array}{c}\hfill \sigma \left(2\sigma \right(2x-1,2\beta )-1,2\beta ).\end{array}$$

(12)

To the best of our knowledge, we could not find a formal proof of the full classification of the dynamics of the sigmoid function (or its second iterate) for all $\beta \in \mathbb{R}$ in the literature. Additionally, it provides an introductory example to demonstrate the concepts and techniques of dynamical systems. We begin by using numerical techniques to determine the number of fixed points in the system and its possible periodic behavior. We then proceed by providing a formal proof of the full dynamical properties of (11) in Theorem 1 and the full dynamical properties of (12) in Theorem 2.

Using numerical techniques, we solve for the number of fixed points of the system. The number of fixed points the function (11) depends on the magnitude of the parameter. For $\beta >0$, there is no periodic behavior, so there are no additional fixed points in (12) that are not fixed points in (11). For $-1\le \beta \le 1$, there is a single fixed point at $x_{*}=1/2$ and for $\beta >1$, there are 3 fixed points $c_0\left(\beta \right),1/2,c_1\left(\beta \right)$ in the interval $[0,1]$. These fixed points satisfy $0\le c_0\left(\beta \right)<1/2<c_1\left(\beta \right)\le 1$, $c_0\left(\beta \right)\to 0$ and $c_1\left(\beta \right)\to 1$ as $\beta \to \infty$. For $\beta <0$, we see periodic behavior in the system; there are fixed points of (12) that are not fixed points of (11). For $\beta <-1$, the function (11) has one fixed point at $x_{*}=1/2$ and a periodic cycle $C=\{c_0\left(\beta \right),c_1\left(\beta \right)\}$. Both $c_0\left(\beta \right),c_1\left(\beta \right)$ are fixed points of (12) and these points are the same fixed points from the $\beta >0$ regime as (12) is an even function with respect to $\beta$.

Table 1. Partial derivatives of (11) and (12) at fixed point $x_{*}=1/2$ for parameter value $\beta =\pm 1$. The derivatives of the the function (11) are denoted using $\sigma$ and the derivatives for (12) are denoted using ${\sigma }^2$.

	${\mathit{\sigma }}_{\mathit{x}}$	${\mathit{\sigma }}_{\mathit{xx}}$	${\mathit{\sigma }}_{\mathit{xxx}}$	${\mathit{\sigma }}_{\mathit{\beta }}$	${\mathit{\sigma }}_{\mathit{\beta }\mathit{x}}$	${\mathit{\sigma }}_{\mathit{x}}^2$	${\mathit{\sigma }}_{\mathit{xx}}^2$	${\mathit{\sigma }}_{\mathit{xxx}}^2$	${\mathit{\sigma }}_{\mathit{\beta }}^2$	${\mathit{\sigma }}_{\mathit{\beta }\mathit{x}}^2$
$\beta =1$	1	0	−8	0	$1/2$	1	0	−16	0	1
$\beta =-1$	−1	0	8	0	$1/2$	1	0	−16	0	−1

denotes the values of the derivatives at the fixed point $1/2$ for $\beta =\pm 1$.

We now have enough information to provide a complete classification of the dynamics of the sigmoid function.

Theorem 1 

(Dynamics of sigmoid function). Consider the discrete dynamical system generated by (11)

$$\begin{array}{c}\hfill x\mapsto \sigma (2x-1,2\beta )=\frac{1}{1+e^{-2\beta (2x-1)}}.\end{array}$$

The full dynamics of the system (11) are as follows

1.

For $-1\le \beta \le 1$, the system has a single hyperbolic fixed point $x_{*}=1/2$ which is a global attractor and there are no p-periodic points for $p\ge 2$.

2.

For $\beta >1$, the system has one repelling hyperbolic fixed point $x_{*}=1/2$ and two hyperbolic stable fixed points $c_0$, $c_1$, with $0<c_0<1/2<c_1<1$, and stable sets $W^s\left(c_0\right)=[0,1/2)$, $W^s\left(c_1\right)=(1/2,1]$. There are no p-periodic points for $p\ge 2$.

3.

For $\beta <-1$, the system has one unstable hyperbolic fixed point $x_{*}=1/2$, and a stable 2-cycle $\mathcal{C}=\{c_0,c_1\}$ with stable set $W^s\left(\mathcal{C}\right)=[0,1/2)\cup (1/2,1]$, with $0<c_0<1/2<c_1<1$. There are no p-periodic points for $p>2$.

4.

For $\left|\beta \right|=1$, the system has one non-hyperbolic fixed point at $x_{*}=1/2$ which is asymptotically stable and attracting.

The system undergoes a PD bifurcation at $\beta =-1$ and a pitchfork bifurcation at $\beta =1$.

Proof. 

We will break the proof up into three parts. The first part of the proof is a linear stability analysis of the system, the second part is a stability analysis of the periodic points in the system and the third part is an analysis of the bifurcations of the system. We begin with a linear stability analysis of the system at each fixed point. For $\beta \le 1$ the system has one fixed point $x_{*}=1/2$ and for $\beta >1$ the system has three fixed points $c_0,1/2,c_1$. The derivative of $\sigma (2x-1,2\beta )$ is ${\sigma }_x(2x-1,2\beta )=-4\beta \sigma (2x-1,2\beta )(1-\sigma (2x-1,2\beta ))$.

Fixed point $x_{*}=1/2$: The Jacobian of the system at the fixed point $x_{*}=1/2$ is ${\sigma }_x(2x_{*}-1,2\beta )=\beta$. For $\beta \ne 1$, the fixed point $x_{*}=1/2$ is hyperbolic and for $\beta =\pm 1$ the fixed point is non-hyperbolic. We classify the stability of the hyperbolic fixed point $x_{*}=1/2$ using Theorem A2. For $\left|\beta \right|<1$ the fixed point $x_{*}=1/2$ is globally attracting as $|{\sigma }_x(2x_{*}-1,2\beta )|<1$ and for $\left|\beta \right|>1$ the fixed point $x_{*}=1/2$ is globally repelling as $|{\sigma }_x(2x_{*}-1,2{\beta }_{*})|>1$. For $\beta =\pm 1$ we invoke Theorem A3 to check for stability of the fixed point. At $\beta =-1$ we have ${\sigma }_x(2x_{*}-1,2\beta )=-1$ and we need to check the Schwarzian derivative. The fixed point $x_{*}=1/2$ is asymptotically stable for $\beta =-1$ by Theorem A3, as $\mathit{S}\sigma (2\sigma (2x-1,2\beta )-1,2\beta ){\mid }_{x=x_{*}}=-8$. For $\beta =1$ we have ${\sigma }_x(2x_{*}-1,2\beta )=1$ and we need to check the second and third derivatives at the fixed point. The fixed point $x_{*}=1/2$ is asymptotically stable when $\beta =1$ by Theorem A3 as ${\sigma }_{xx}(2x_{*}-1,2\beta )=0$ and ${\sigma }_{xxx}(2x_{*}-1,2\beta )=-8$.

Fixed points $c_0,c_1$: These fixed points have the same behavior so we have grouped them together in the analysis. When $\beta >1$ there are two additional fixed points $c_0,c_1$ of the system, both are attracting fixed points by Theorem A2 as $|{\sigma }_x(2c_i-1,2\beta )|<1$ for each $i=0,1$ and all $\beta >1$. The stable sets are $W^s\left(c_0\right)=[0,1/2)$ and $W^s\left(c_1\right)=(1/2,1]$.

Periodic points: For $\beta <-1$ we see the two cycle $\mathcal{C}=\{c_0,c_1\}$. Notice $\sigma (2c_0-1,2\beta )=c_1$ and $\sigma (2c_1-1,2\beta )=c_0$. This two cycle is stable since $c_0$ and $c_1$ are both stable fixed points of (12). The stable set is $W^s\left(\mathcal{C}\right)=[0,1/2)\cup (1/2,1]$, $0<c_0<1/2<c_1<1$.

At $(x_{*},{\beta }_{*})=(1/2,1)$ the system under goes a pitchfork bifurcation as it satisfies the conditions in Theorem A5:

$$\begin{array}{c}\hfill \sigma (2x_{*}-1,2{\beta }_{*})=1/2{\sigma }_x(2x_{*}-1,2{\beta }_{*})=1{\sigma }_{xx}(2x_{*}-1,2{\beta }_{*})=0,\\ \hfill {\sigma }_{\beta }(2x_{*}-1,2{\beta }_{*})=0{\sigma }_{x\beta }(2x_{*}-1,2{\beta }_{*})\ne 0{\sigma }_{xxx}(2x_{*}-1,2{\beta }_{*})\ne 0.\end{array}$$

Similarly at $(x_{*},{\beta }_{*})=(1/2,-1)$ the system under goes a period doubling bifurcation as it satisfies the conditions in Theorem A4

$$\begin{array}{c}\hfill \sigma (2x_{*}-1,2{\beta }_{*})=1/2{\sigma }_x(2x_{*}-1,2{\beta }_{*})=-1{\sigma }_{xx}(2x_{*}-1,2{\beta }_{*})=0,\\ \hfill {\sigma }_{\beta }(2x_{*}-1,2{\beta }_{*})=0{\sigma }_{x\beta }(2x_{*}-1,2{\beta }_{*})\ne 0{\sigma }_{xxx}(2x_{*}-1,2{\beta }_{*})\ne 0.\end{array}$$

 □

We can fully classify the dynamics of (12) using the above theorem. We omit the proof as it is similar to the proof of Theorem 1.

Theorem 2. 

The full dynamics of the system (12) are as follows

1.

For $-1\le \beta \le 1$, the system has a single hyperbolic fixed point $x_{*}=1/2$ which is a global attractor and there are no p-periodic points for $p\ge 2$.

2.

For $\left|\beta \right|>1$, the system has one repelling hyperbolic fixed point $x_{*}=1/2$ and two hyperbolic stable fixed points $c_0$, $c_1$, with $0<c_0<1/2<c_1<1$, and stable sets $W^s\left(c_0\right)=[0,1/2)$, $W^s\left(c_1\right)=(1/2,1]$.

3.

For $\left|\beta \right|=1$, the system has one non-hyperbolic fixed point at $x_{*}=1/2$ which is asymptotically stable and attracting.

The system undergoes a pitchfork bifurcation at $\beta =\pm 1$. There are no p-periodic points for $p\ge 2$.

5.2. Sequential Dynamics

To fully understand the dynamics of the equations defining the updates to $q_1^{*}$ and $q_2^{*}$ it suffices to track the evolution of the points $q_1^{*}\left(1\right)=\zeta$ and $q_2^{*}\left(1\right)=\xi$. The CAVI algorithm updates terms sequentially, using the new values of the variables to calculate the others. We initialize the CAVI algorithm at points ${\zeta }_0,{\xi }_0$. The CAVI algorithm is a dynamical system formed by sequential iterations of $\sigma (2x-1,2\beta )$ starting from ${\zeta }_0,{\xi }_0$. We can decouple the CAVI updates for ${\xi }_k$ and ${\zeta }_k$ by looking at the second iterations. This decoupling is visualized in the diagram (14) below. The system formed the sequential updates is equivalent to the following decoupled system

$$\begin{array}{ccc}\hfill {\zeta }_1& & =\sigma (2{\xi }_0-1,2\beta ),\hfill \\ \hfill {\zeta }_{k+1}& & =\sigma (2\sigma (2{\zeta }_k-1,2\beta )-1,2\beta ),k\ge 1,\hfill \\ \hfill {\xi }_{k+1}& & =\sigma (2\sigma (2{\xi }_k-1,2\beta )-1,2\beta ),k\ge 0.\hfill \end{array}$$

(13)

We propose to investigate the dynamics of the sequential system (9) by studying the dynamics of individual subsequences ${\zeta }_{k+1}$ and ${\xi }_{k+1}$ of the decoupled system (13). The dynamical properties of the individual subsequences follow from a combination of Theorem 1, Theorem 2 and other methods from Appendix A.

(14)

Illustrations of the evolution of the dynamics of the sequential updates for various initializations and values of $\beta$ are in Figure 3, Figure 4, Figure 5 and Figure 6.

Theorem 3 

(CAVI dynamics). The Dynamics of the CAVI System (9) Are Given by

1.

For $\beta <-1$, the system has the system has one locally asymptotically unstable fixed point $(1/2,1/2)$ and two locally asymptotically stable fixed points $(c_0,c_1)$ and $(c_1,c_0)$, with stable sets $W^s\left((c_0,c_1)\right)=[0,1]\times [0,1/2)$ and $W^s\left((c_1,c_0)\right)=[0,1]\times (1/2,1]$ respectively.

2.

For $\left|\beta \right|\le 1$, the system has a global asymptotically stable fixed point $(1/2,1/2)$.

3.

For $\beta >1$ the system has the system has one locally asymptotically unstable fixed point $(1/2,1/2)$ and two locally asymptotically stable fixed points $(c_0,c_0)$ and $(c_1,c_1)$, with domains of attraction $W^s\left((c_0,c_0)\right)=[0,1]\times [0,1/2)$ and $W^s\left((c_1,c_1)\right)=[0,1]\times (1/2,1]$ respectively.

where $0\le c_0<1/2<c_1\le 1$ are the fixed points of (11) in $[0,1]$. The system undergoes a super-critical pitchfork bifurcation at $\beta =-1$ and again at $\beta =1$. Furthermore the system has no p-periodic points for $p\ge 2$.

Proof. 

We will proceed to construct the dynamics of the system (9) by tracing the behavior of the dynamics in the equivalent system (13). The dynamics of each of these subsequences is governed by the Functions (11) and (12) and dependent on the initialization ${\xi }_0$. The behavior for each of the subsequence ${\xi }_{k+1}$, for $k\ge 0$ is governed by Theorem 2. Similarly the behavior of the subsequence ${\zeta }_{k+1}$, for $k\ge 1$ is governed by Theorem 2 with the additional point ${\zeta }_1=\sigma (2{\xi }_0-1,2\beta )$ dependent on Theorem 1. For $\left|\beta \right|<1$, (11) has a globally stable fixed point at $x_{*}=1/2$ and thus for all ${\xi }_0$, ${\zeta }_1=\sigma (2{\xi }_0-1,2\beta )\in W^s(1/2)$. It now follows from Theorem 2 that the only fixed point in the sequential system is $(1/2,1/2)$ which must be globally stable. For $\beta =\pm 1$, the fixed point $x_0=1/2$ is asymptotically stable by Theorem A3. The system undergoes a super-critical pitchfork bifurcation at $\beta =-1$ and again at $\beta =1$ as a consequnece from its relation to (12). For $\beta >1$, (11) bifurcates. We have the unstable fixed point $x_{*}=1/2$, and the two locally stable fixed points, $c_0$ with stable set $W^s\left(c_0\right)=[0,1/2)$, and $c_1$ with stable set $W^s\left(c_1\right)=(1/2,1]$. For ${\xi }_0\in W^s\left(c_0\right)$ we have ${\zeta }_1\in W^s\left(c_0\right)$ and ${\xi }_1\in W^s\left(c_0\right)$. It now follows from Theorem 2 that the system will converge to $(c_0,c_0)$ and that $W^s\left((c_0,c_0)\right)=[0,1]\times [0,1/2)$. A similar argument shows the system converges to $(c_1,c_1)$ for ${\xi }_0\in W^s\left(c_1\right)$ and $W^s\left((c_1,c_1)\right)=[0,1]\times (1/2,1]$. Lastly, $(1/2,1/2)$ is a repelling fixed point of the systems since $x_{*}=1/2$ is a repelling fixed point for both (11) and (12). For $\beta <-1$, (11) bifurcates. We have the unstable fixed point $x_{*}=1/2$, and the stable two cycle, $\mathcal{C}=\{c_0,c_1\}$ with stable set $W^s\left(\mathcal{C}\right)=[0,1/2)\cup (1/2,1]$. For any ${\xi }_0<1/2$ we have, ${\zeta }_1>1/2$ and ${\xi }_1<1/2$. It now follows from Theorem 2 that the system will converge to $(c_1,c_0)$ and that $W^s\left((c_1,c_0)\right)=[0,1]\times [0,1/2)$. A similar argument shows the system converges to $(c_0,c_1)$ for ${\xi }_0>1/2$ and $W^s\left((c_0,c_1)\right)=[0,1]\times [0,1/2)$. Lastly, $(1/2,1/2)$ is a repelling fixed point of the systems since $x_{*}=1/2$ is a repelling fixed point for both (11) and (12). The dynamics of (13) lack any p-period point and cycles for $p>2$ as a consequence of its construction from (12). □

5.3. Parallel Updates

The system of parallel updates is defined by the one-step map $F:{\mathbb{R}}^2\to {\mathbb{R}}^2$

$$\begin{array}{c}\hfill \left(\begin{array}{c}\zeta \\ \xi \end{array}\right)\mapsto F(\zeta ,\xi )=\left(\begin{array}{c}\sigma (2\xi -1,2\beta )\\ \sigma (2\zeta -1,2\beta ).\end{array}\right)\end{array}$$

(15)

The dynamics of the parallel system are similar to the system studied in [36]. As we shall show below, the parallel system exhibits periodic behavior that the sequential system does not and it follows as a corollary that the systems are not locally topologically conjugate.

The parallelized CAVI algorithm is a dynamical system formed by iterations of F defined in (15). We shall decouple the parallelized CAVI updates for sequences ${\xi }_k$ and ${\zeta }_k$ by looking at iterations of (12) acting on the sequences individually. This decoupling is visualized in diagram form

(16)

where each cross is an application of F. The system formed the parallel updates is equivalent to the following decoupled systems of even subsequences and odd subsequences. The even subsequences are

$$\begin{array}{ccc}\hfill {\zeta }_{2k}& & =\sigma (2\sigma (2{\zeta }_{2(k-1)}-1,2\beta )-1,2\beta ),k\ge 1\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill {\xi }_{2k}& & =\sigma (2\sigma (2{\xi }_{2(k-1)}-1,2\beta )-1,2\beta ),k\ge 1.\hfill \end{array}$$

(18)

The odd subsequences are

$$\begin{array}{ccc}\hfill {\zeta }_{2k+1}& & =\left\{\begin{array}{cc}\sigma (2{\xi }_0,2\beta )\hfill & k=0\hfill \\ \sigma (2\sigma (2{\zeta }_{2k-1},2\beta ),2\beta )\hfill & k\ge 1\hfill \end{array}\right.\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill {\xi }_{2k+1}& & =\left\{\begin{array}{cc}\sigma (2{\zeta }_0,2\beta )\hfill & k=0\hfill \\ \sigma (2\sigma (2{\xi }_{2k-1},2\beta ),2\beta )\hfill & k\ge 1.\hfill \end{array}\right.\hfill \end{array}$$

(20)

Following a similar approach to the one used to study the sequential dynamics, we investigate the dynamics of the parallel system (15) by studying the dynamics of four individual subsequences (17)–(20) of the decoupled system given by diagram (16). The dynamical properties of the individual subsequences follow from a combination of Theorem 1, Theorem 2 and other methods from Appendix A. Illustrations of the evolution of the dynamics of the parallel updates for various initializations and values of $\beta$ are in Figure 7, Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12.

We now present the main result for the parallel dynamics.

Theorem 4 

(Parallel Dynamics). The Dynamics of the Parallel System (10) Are As Follows

1.

For $\beta <-1$, the system has two locally asymptotically stable fixed points $(c_1,c_0)$ and $(c_0,c_1)$, and one locally asymptotically unstable fixed point $(1/2,1/2)$, where $c_0$ and $c_1$ are the fixed points of (11). Furthermore the system exhibits periodic behavior in the form of 2-cycles. The asymptotically stable 2-cycle, ${\mathcal{C}}_1=\{(c_0,c_0),(c_1,c_1)\}$ and asymptotically unstable 2-cycles,

$$\begin{array}{c}\hfill {\mathcal{C}}_2=\{(1/2,c_1),(c_0,1/2)\}and{\mathcal{C}}_3=\{(1/2,c_0),(c_1,1/2)\}.\end{array}$$

The stable sets are

$$\begin{array}{ccc}\hfill W^s(c_0,c_1)& & =[0,1/2)\times (1/2,1]\hfill \\ \hfill W^s(c_1,c_0)& & =(1/2,1]\times [0,1/2)\hfill \\ \hfill W^s\left({\mathcal{C}}_1\right)& & =\left([0,1/2)\times [0,1/2)\right)\cup \left((1/2,1]\times (1/2,1]\right)\hfill \\ \hfill W^s\left({\mathcal{C}}_2\right)& & =\left([0,1/2)\times \{1/2\}\right)\cup \left(\{1/2\}\times (1/2,1]\right)\hfill \\ \hfill W^s\left({\mathcal{C}}_3\right)& & =\left([0,1/2)\times \{1/2\}\right)\cup \left(\{1/2\}\times (1/2,1]\right).\hfill \end{array}$$

2.

For $-1\le \beta \le 1$, the system has a global attracting fixed point $(1/2,1/2)$.

3.

For $\beta >1$, the system has two locally asymptotically stable fixed points $(c_0,c_0)$ and $(c_1,c_1)$, and one locally asymptotically unstable fixed point $(1/2,1/2)$, where $c_0$ and $c_1$ are the fixed points of (11). Furthermore the system exhibits periodic behavior in the form of 2-cycles. The asymptotically stable 2-cycle, ${\mathcal{C}}_3=\{(c_0,c_0),(c_1,c_1)\}$ and asymptotically unstable 2-cycles, ${\mathcal{C}}_4=\{(1/2,c_0),(c_1,1/2)\}$ and ${\mathcal{C}}_5=\{(1/2,c_1),(c_1,1/2)\}$. The stable sets are

$$\begin{array}{ccc}\hfill W^s(c_0,c_1)& & =[0,1/2)\times (1/2,1]\hfill \\ \hfill W^s(c_1,c_0)& & =(1/2,1]\times [0,1/2)\hfill \\ \hfill W^s\left({\mathcal{C}}_3\right)& & =\left([0,1/2)\times [0,1/2)\right)\cup \left((1/2,1]\times (1/2,1]\right)\hfill \\ \hfill W^s\left({\mathcal{C}}_4\right)& & =\left([0,1/2)\times \{1/2\}\right)\cup \left(\{1/2\}\times [0,1/2)\right)\hfill \\ \hfill W^s\left({\mathcal{C}}_5\right)& & =\left(\{1/2\}\times (1/2,1]\right)\cup \left((1/2,1]\times \{1/2\}\right).\hfill \end{array}$$

The system has no p-periodic points for $p>2$. The system under goes a PD bifurcation at $\beta =-1$ and a pitchfork bifurcation at $\beta =1$.

Proof. 

The dynamics of the system defined by F in (15) are equivalent to the dynamics of the system generated by the subsequences (17)–(20). The dynamics of each of these subsequences are governed by the functions (11) and (12). By Theorem 1, we have the behavior for each of the subsequences (17)–(20). For $\left|\beta \right|<1$, (11) has a globally stable fixed point at $x_{*}=1/2$ and thus the only fixed point in the parallel system is $(1/2,1/2)$ which must be globally stable. For $\beta =\pm 1$, the fixed point $x_0=1/2$ is asymptotically stable by Theorem A3.

For $\beta >1$, (11) bifurcates. We have the unstable fixed point $x_{*}=1/2$, and the two locally stable fixed points, $c_0$ with stable set $W^s\left(c_0\right)=[0,1/2)$, and $c_1$ with stable set $W^s\left(c_1\right)=(1/2,1]$. Returning to the system generated by F, if we consider the initialization $({\zeta }_0,{\xi }_0)=(c_0,c_0)$ then by the sequence construction of ${\zeta }_n$, given in (19) and (21), we see that ${\zeta }_n=c_0$ for $n\ge 1$, as $c_0$ is a fixed point of (11) for $\beta >1$. Similarly, using the sequence construction of ${\xi }_n$, given in (18) and (20), we see that ${\xi }_n=c_0$ for $n\ge 1$, as $c_0$ is a fixed point of (11) for $\beta >1$. Therefore, $(c_0,c_0)$ is a fixed point. An analogous argument shows that $(c_1,c_1)$ is also a fixed point. The parallel system has the stable fixed points $(c_0,c_0)$ with stable set $W^s(c_0,c_0)=W^s\left(c_0\right)\times W^s\left(c_0\right)$ and $(c_1,c_1)$ with stable set $W^s(c_1,c_1)=W^s\left(c_1\right)\times W^s\left(c_1\right)$. After the bifurcation at $\beta =1$ the parallel system also contains 2-cycles. Using the sequence construction we see that ${\mathcal{C}}_3=\{(c_1,c_0),(c_0,c_1)\}$ is an asymptotically stable 2-cycle in the parallel system, with stable subspace $W^s\left({\mathcal{C}}_3\right)=(1/2,1]\times [0,1/2)\cup [0,1/2)\times (1/2,1]$. Additionally, we have two asymptotically unstable 2-cycles ${\mathcal{C}}_4=\{(c_0,1/2),(1/2,c_0)\}$ and ${\mathcal{C}}_5=\{(c_1,1/2),(1/2,c_1)\}$. Perturbing the $1/2$ coordinate in the unstable cycle pushes it into the basin of attraction for one of the fixed points or the asymptotically stable 2-cycle. The stable sets are $W^s\left({\mathcal{C}}_4\right)=\left([0,1/2)\times \{1/2\}\right)\cup \left(\{1/2,1\}\times [0,1/2)\right)$, $W^s\left({\mathcal{C}}_5\right)=\left(\{1/2\}\times (1/2,1]\right)\cup \left((1/2,1]\times \{1/2\}\right)$. The dynamics of F lack any p-period point and cycles for $p>2$ as a consequence of its construction from (12).

For $\beta <-1$, (11) bifurcates. We have the unstable fixed point $x_{*}=1/2$, and the stable two cycle, $\mathcal{C}=\{c_0,c_1\}$ with stable set $W^s\left(\mathcal{C}\right)=[0,1/2)\cup (1/2,1]$. Returning to the system generated by F, if we consider the initialization $({\zeta }_0,{\xi }_0)=(c_0,c_1)$ then by the sequence construction of ${\zeta }_n$, given in (17) and (19), we see that ${\zeta }_n=c_0$ for $n\ge 1$, as $\mathcal{C}$ is a 2-cycle of (11) for $\beta <-1$. Similarly, using the sequence construction of ${\xi }_n$, given in (18) and (20) we see that ${\xi }_n=c_1$ for $n\ge 1$, as $\mathcal{C}$ is a 2-cycle of (11) for $\beta <-1$. Therefore, $(c_0,c_1)$ is a fixed point. An analogous argument shows that $(c_1,c_0)$ is also a fixed point. The parallel system has the stable fixed points $(c_0,c_1)$ with stable set $W^s(c_0,c_1)=W^s\left(c_0\right)\times W^s\left(c_1\right)$ and $(c_1,c_0)$ with stable set $W^s(c_1,c_0)=W^s\left(c_1\right)\times W^s\left(c_0\right)$, where $W^s\left(c_0\right)=[0,1/2)$ and $W^s\left(c_1\right)=(1/2,1]$. After the bifurcation at $\beta =-1$ the parallel system also contains 2-cycles. Using the sequence construction we see that ${\mathcal{C}}_1=\{(c_0,c_0),(c_1,c_1)\}$ is an asymptotically stable 2-cycle in the parallel system, with stable subspace $W^s\left({\mathcal{C}}_1\right)=W^s\left(c_0\right)\times W^s\left(c_0\right)\cup W^s\left(c_1\right)\times W^s\left(c_1\right)$. Additional we have two asymptotically unstable 2-cycles ${\mathcal{C}}_2=\{(c_0,1/2),(1/2,c_1)\}$ and ${\mathcal{C}}_3=\{(c_1,1/2),(1/2,c_0)\}$. Perturbing the $1/2$ coordinate in the unstable cycle pushes it into the basin of attraction for one of the fixed points or the asymptotically stable 2-cycle. The stable sets are $W^s\left({\mathcal{C}}_3\right)=\left([0,1/2)\times [0,1/2)\right)\cup \left((1/2,1]\times (1/2,1]\right)$, $W^s\left({\mathcal{C}}_4\right)=\left([0,1/2)\times \{1/2\}\right)\cup \left(\{1/2\}\times [0,1/2)\right)$, $W^s\left({\mathcal{C}}_5\right)=\left(\{1/2\}\times (1/2,1]\right)\cup \left((1/2,1]\times \{1/2\}\right)$. The dynamics of F lack any p-period point and cycles for $p>2$ as a consequence of its construction from (12).

This completes the characterization of the dynamics of F for $\beta \in \mathbb{R}$. □

5.4. A Comparison of the Dynamics

We end the section by providing a comparison of the dynamical properties of the sequential system in Theorem 3 and the parallel system in Theorem 4. The main difference between the sequential system and the parallel system is the presence of two-cycles that can be found in the parallel system when $\left|\beta \right|>1$. This behavior stems from the difference between the sequential and parallel implementations of the CAVI. Looking closely at the update diagrams for the two systems reveals the key difference that produces these two-cycles. The decoupled sequential system is

and the decoupled parallel system is

The major difference between these diagrams is how the individual update sequences begin. Notice ${\zeta }_0$ plays no role in updating the sequential system as both the ${\zeta }_k$ update sequence and the ${\xi }_k$ update sequence are dependent only on the choice of ${\xi }_0$. Even after rewriting the sequential updates in terms of individual sequences the system is not truly decoupled as both sequences depend on a common starting point. This precisely prescribes the behavior that we see in the system relative to the sigmoid function dynamics in Theorem 1 and Theorem 2. Compare this to the parallel system. Here ${\zeta }_0$ is involved in updating both the odd ${\xi }_{2k+1}$ subsequence and the even ${\zeta }_{2k}$ subsequence. Furthermore, ${\xi }_0$ remains involved by controlling the updates for the even ${\xi }_{2k}$ subsequence and the odd ${\zeta }_{2k+1}$ subsequence. This additional flexibility allows the parallel system to develop periodic behavior outside of the Dobrushin regime $(1\le \beta \le 1)$.

As an example, we will consider initializing the sequential algorithm to the parallel algorithm for $\beta =1.2$. We begin with the sequential algorithm. For $\beta =1.2$, consider initializing the sequential system at $({\zeta }_0,{\xi }_0)=(0.7,0.3)$. The sequential system updates are fully determined by ${\xi }_0$, so for ${\xi }_0=0.3$ it follows from Theorem 1 that an application of the function (11) will cause ${\zeta }_1\in W^s\left(c_0\right)$. At this point, the system can be evolved by applying (12) to the independent sequences for $\zeta$ and $\xi$ as given in (13). The dynamics of the system are now controlled by the function (12). From this initialization the system will converge to the fixed point $(c_0,c_0)=(0.17071,0.17071)$ as shown in Figure 6.

Contrast this with the behavior of the parallel system in which the updates are determined by both ${\xi }_0$ and ${\zeta }_0$. For $\beta =1.2$, consider initializing the parallel system at $({\zeta }_0,{\xi }_0)=(0.7,0.3)$. It follows from Theorem 1 that an application of the function (11) will cause ${\zeta }_1\in W^s\left(c_0\right)$ and ${\xi }_1\in W^s\left(c_1\right)$. Successive updates will cause the sequences ${\zeta }_k$ and ${\xi }_k$ to flip back and forth between the domains $W^s\left(c_0\right)$ and $W^s\left(c_1\right)$, until the system settles into the two cycle ${\mathcal{C}}_1=\{(c_0,c_1),(c_1,c_0)\}=\{(0.17071,0.82928),(0.82928,0.17071)\}$ as seen in Figure 10.

This simple example highlights the danger of naively parallelizing the CAVI algorithm. The convergence properties of a parallel version of the CAVI algorithm will heavily depend on the models CAVI update equations. In the case of the Ising model we have demonstrated that for certain parameter regimes the parallel implementation of the algorithm can fail to converge due to the dependence of the algorithm on both ${\zeta }_0$ and ${\xi }_0$.

6. Edward–Sokal Coupling

One method of improving convergence in Markov chains is through the use of probabilistic couplings. The Edward–Sokal (ES) coupling is a coupling of two statistical physics models, the random cluster model and the Potts model (a generalization of the Ising model) [37]. Running a Markov chain on the ES coupling leads to improved mixing properties compared to the equivalent Potts model and random cluster models [33]. Motivated by these findings in the Markov chain literature, we ask a similar question: Can the convergence properties of mean-field VI be improved by using the ES coupling in place of the Ising model? In this section we investigate this idea numerically. We first introduce the Edward–Sokal coupling following [37]. We introduce a variational family for the Edward–Sokal coupling and derive the variational updates for this model. Our findings suggests the variational updates converge to a unique solution in a larger range than the equivalent Dobrushin regime for the corresponding Ising measure.

6.1. Random Cluster Model

Let $G=(V,E)$ be a finite graph. Let $e=\langle x,y\rangle \in E$ denote an edge in G with endpoints $x,y\in V$. $\mathsf{\Sigma }={\{1,2,\dots ,q\}}^V$, $\mathsf{\Omega }={\{0,1\}}^E$ and $\mathcal{F}$ denotes the powerset of $\mathsf{\Omega }$. The random cluster model is a 2 parameter probability measure with an edge weight parameter $p\in [0,1]$ and a cluster weight parameter $q\in \{2,3,\dots \}$ on $(\mathsf{\Omega },\mathcal{F})$ given by

$${\varphi }_{p,q}\left(\omega \right)\propto \left\{\prod _{e\in E}{p}^{\omega \left(e\right)}{(1-p)}^{(1-\omega (e\left)\right)}\right\}{q}^{\kappa \left(\omega \right)},$$

where $\kappa \left(\omega \right)$ denoted the number of connected components in the subgraph corresponding to $\omega$. The partition function for the random cluster model is

$${\mathcal{Z}}_{RC}=\sum _{\omega \in \mathsf{\Omega }}\left\{\prod _{e\in E}{p}^{\omega \left(e\right)}{(1-p)}^{(1-\omega (e\left)\right)}\right\}{q}^{\kappa \left(\omega \right)}.$$

For $q=2$ the the random cluster model reduces to the Ising model on G.

The Edward–Sokal Coupling is a probability measure $\mu$ on $\mathsf{\Sigma }\times \mathsf{\Omega }$ given by

$$\mu (\sigma ,\omega )\propto \prod _{e\in E}\left[(1-p){\delta }_{\omega \left(e\right),0}+p{\delta }_{\omega \left(e\right),1}{\delta }_e\left(\sigma \right)\right],$$

(21)

where ${\delta }_{a,b}=1(a=b)$, and ${\delta }_e\left(\sigma \right)=1({\sigma }_x={\sigma }_y)$, for $e=(x,y)\in E$.

It is well known that in the special case, $p=1-e^{-\beta }$ and $q=2$ the $\mathsf{\Sigma }$-marginal of the ES coupling is the Ising model, the $\mathsf{\Omega }$-marginal is the random cluster model [37]. We are interested in better understanding how the convergence of the CAVI algorithm on the ES coupling compares to the convergence of the CAVI algorithm on the Ising model.

6.2. VI Objective Function

To calculate the VI updates for each variable we may need to make use of the alternative characterization of the ES coupling

$$\mu (\sigma ,\omega )\propto \psi \left(\sigma \right){\varphi }_{p,1}\left(\omega \right)1_F(\sigma ,\omega )$$

where $\psi$ is uniform measure on $\mathsf{\Sigma }$ and ${\varphi }_{p,1}\left(\omega \right)$ is a product measure on $\mathsf{\Omega }$

$${\varphi }_{p,1}\left(\omega \right)=\prod _{e\in E}{p}^{\omega \left(e\right)}{(1-p)}^{(1-\omega (e\left)\right)}$$

(22)

and

$$F=\left\{(\sigma ,\omega ):{\delta }_{\omega }\left(e\right)=1\Rightarrow {\delta }_e\left(\sigma \right)=1\right\}$$

(23)

The variational family that we will be optimizing over is

$$\begin{array}{c}\hfill q(\sigma ,\omega )=q_1\left({\sigma }_1\right)q_2\left({\sigma }_2\right)q_0\left(\omega \right)1_F(\sigma ,\omega ).\end{array}$$

(24)

We have added the indicator on the set F to eliminate the configurations $(\sigma ,\omega )$ that are not well defined in the variational objective. We will use the convention that $0log\left(0\right)=0$.

6.3. VI Updates

The ELBO that corresponds to the variational family (24) is

$$\begin{array}{ccc}\hfill \mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}(x_1,x_2,y,p)& =& x_1{x}_{2}ylog\left(x_1{x}_{2}y\right)-x_1{x}_{2}ylog(1-p)\hfill \\ & +& (1-x_1)x_{2}ylog\left((1-x_1)x_{2}y\right)-(1-x_1)x_{2}ylog(1-p)\hfill \\ & +& x_1(1-x_2)ylog\left(x_1(1-x_2)y\right)-x_1(1-x_2)ylog(1-p)\hfill \\ & +& (1-x_1)(1-x_2)ylog\left((1-x_1)(1-x_2)y\right)-(1-x_1)(1-x_2)ylog(1-p)\hfill \\ & +& x_1{x}_2(1-y)log\left(x_1{x}_2(1-y)\right)-x_1{x}_2(1-y)log\left(p\right)\hfill \\ & +& (1-x_1)(1-x_2)(1-y)log\left((1-x_1)(1-x_2)(1-y)\right)-(1-x_1)(1-x_2)(1-y)log\left(p\right).\hfill \end{array}$$

Taking the derivative with respect to $x_1$ and simplifying gives us

$$\begin{array}{ccc}\hfill {\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}}_1(x_1,x_2,y,p)& =& ylog\left(\frac{x_1}{1-x_1}\right)+(1-y)log\left(\frac{1}{1-x_1}\right)\hfill \\ & +& x_2(1-y)log\left(x_1(1-x_1)\right)+x_2(1-y)log\left(\frac{x_2(1-x_2){(1-y)}^2}{p^2}\right)\hfill \\ & +& log\left(\frac{p}{(1-x_2)(1-y)}\right)+(2x_2-1)(1-y).\hfill \end{array}$$

Taking the derivative with respect to $x_2$ and simplifying gives us

$$\begin{array}{ccc}\hfill {\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}}_2(x_1,x_2,y,p)& =& ylog\left(\frac{x_2}{1-x_2}\right)+(1-y)log\left(\frac{1}{1-x_2}\right)\hfill \\ & +& x_1(1-y)log\left(x_2(1-x_2)\right)+x_1(1-y)log\left(\frac{x_1(1-x_1){(1-y)}^2}{p^2}\right)\hfill \\ & +& log\left(\frac{p}{(1-x_1)(1-y)}\right)+(2x_1-1)(1-y).\hfill \end{array}$$

Taking the derivative with respect to y and simplifying gives us

$$\begin{array}{ccc}\hfill {\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}}_y(x_1,x_2,y,p)& =& x_1{x}_{2}log\left(\frac{y}{1-y}\right)+x_1{x}_{2}log\left(\frac{p}{1-p}\right)\hfill \\ & +& (1-x_1)(1-x_2)log\left(\frac{y}{1-y}\right)+(1-x_1)(1-x_2)log\left(\frac{p}{1-p}\right)\hfill \\ & +& (1-x_1)x_{2}log\left(\frac{(1-x_1)x_{2}y}{1-p}\right)+x_1(1-x_2)log\left(\frac{x_1(1-x_2)y}{1-p}\right)\hfill \\ & +& (1-x_1)x_2+x_1(1-x_2).\hfill \end{array}$$

Absence of closed form updates for any of the variables limits our ability to study the convergence of the system with classical dynamical systems techniques. Instead we look at the long evolution behavior of the system by plotting 100 iterations of the CAVI updates which are generated from the following system

$$\begin{array}{ccc}\hfill x_1(t+1)& =& {\mathrm{argmin}}_{z\in (0,1)}\left|{\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}}_1(z,x_2\left(t\right),y\left(t\right),p)\right|,\hfill \\ \hfill x_2(t+1)& =& {\mathrm{argmin}}_{z\in (0,1)}\left|{\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}}_2(x_1(t+1),z,y\left(t\right),p)\right|,\hfill \\ \hfill y(t+1)& =& {\mathrm{argmin}}_{z\in (0,1)}\left|{\mathrm{E}\mathrm{L}\mathrm{B}\mathrm{O}}_y(x_1(t+1),x_2(t+1),z,p)\right|.\hfill \end{array}$$

We generate the argmin of the free variable z from a line search with a step size of $\Delta ={10}^{-6}$. Running these simulations we find that the iterations of $x_1\left(t\right),x_2\left(t\right),y\left(t\right)$ converge to a global solution within about $T=20$ time steps from any initialization $x_1\left(0\right),x_2\left(0\right),y\left(0\right)\in (0,1)$ and any $\beta >0$. It is evident that using the ES coupling, we get global convergence of the algorithm outside of the Dobrushin regime of the corresponding paramagnetic Ising model. The figures depicting the simulation results of convergence of the variational inference algorithm in the Edward–Sokal coupling can be found below in Figure 13, Figure 14, Figure 15 and Figure 16.

7. Conclusions

This paper demonstrates the use of classical dynamical systems and bifurcation theory to study the convergence properties of the CAVI algorithm of the Ising model on two nodes. In our simple model we are able to provide the complete dynamical behavior for the Ising model on two nodes. Interestingly, we find that the sequential CAVI algorithm and parallelized CAVI algorithm are not topologically conjugate owing to the presence of periodic behavior in the parallelized CAVI. This behavior originates from the added flexibility of the initialization in the parallelized CAVI when compared to the sequential CAVI. The erratic behavior we see in the Ising model for $\left|\beta \right|>1$ is due to a combination of the existence of multiple fixed points of the systems update function and the instability of these fixed points. In this parameter regime, the fixed point that produces the optimal solution (0.5, 0.5) is a repelling fixed point. Unless we initialize the algorithm exactly at (0.5, 0.5), the CAVI system cannot converge to this point. The other two suboptimal fixed points are both asymptotically stable. This suggests that the main problem that the CAVI algorithm experiences is centered around the existence of multiple fixed points. Recent work on stochastic block models (SBM) and topic models (TM) models shows that mean field VI leads to suboptimal estimators [17,18,19,20]. It is not clear if this property comes from the mean field variational inferences construction using product distributions or if this is a consequence of structure among latent variables. A minor difference of the stochastic block model (SBM) or topic model (TM) with the Ising model is that the former contain parameters (e.g., the cluster labels) that are identifiable only up to permutations. That being said, in the SBM or TM, if the cluster means are not well-separated, then it is not possible to identify the labels even up to permutations. This is somewhat related to having multiple fixed points of the objective function and we conjecture similar behavior to what we have found in the Ising model will be exhibited in the SBM or TM outside the Dobrushin regime. Interestingly, a close look at the BCAVI updates in [17,18] reveals a similar sigmoid update function $1(1+e^{-x})$. Applying the tools and techniques from dynamical systems theory to study the CAVI algorithm in the SBM, TM and other models will provide a better understanding of the issues that come with using mean field variational inference and is important to developing better variational inference techniques.

Most of the research into the theoretical properties of variational inference has focused on the mean field family due to its computational simplicity. This computational simplicity comes at the cost of limited expressive power. Can we make due with this limited expressive power in practical applications? More specifically, is there an equivalent parameter regime to the Dobrushin regime ($1\le \beta \le 1$) for other similar models like the SBM and TM inside which the CAVI produces statistically optimal estimators? The answer to this question provides researchers with stable parameter regimes for the model. The non-existenceof such a region would indicate the need for more expressive variational methods for the model beyond mean field methods. Recent work [19,20] suggests that this adding some structure to algorithms may fix the problems that arise from mean field VI. How much structure is needed to recover statistically optimal estimators? Could adding in a simple structure of pair-wise dependence to the mean field VI in the Ising model, similarly to [19], be enough to recover the optimal estimator outside of the Dobrushin regime? Is the amount of additional structure that is needed somehow related to the latent structure of the models? Tools from dynamical systems theory can be used to study these questions.

Using dynamical systems to study the convergence properties of the CAVI algorithm is not without its challenges. While dynamical systems theory can provide the answers to many of the above questions, applying these tools to higher dimensional sequential systems is a challenging problem. As mentioned previously, the general theory for n-dimensional discrete dynamical systems is dependent on writing the evolution function in the form $x_{n+1}=F\left(x_n\right)$. Deriving this F is typically not possible for densely connected higher dimensional sequential systems like the n-dimensional Ising model CAVI. This is not the only challenging aspect to the problem. These systems typically possess multiple fixed points which can only be found numerically. Multiple fixed points will lead to more complicated partitions of the space into domains of attraction. Furthermore, higher dimensional systems can possess bifurcations of multiple codimensions, which as significantly more difficult to study. Bifurcations of codimension 3 are so exotic that they are not well studied [23,24]. Software to handle such calculations has only recently been developed [24]. In practical terms this means that the convergence properties can only be studied numerically for models with a small number of parameters. Furthermore, most of the numerical techniques work under the assumption of differentiability of the evolution operator and will fail to be applicable to many systems of practical interest in statistics such as the Edward–Sokal CAVI. Applying tools from dynamical systems to the study of variational inference algorithms will require developing new theory for high dimensional and well connected sequential dynamical systems.