Simultaneous Bayesian Clustering and Model Selection with Mixture of Robust Factor Analyzers

Abstract

Finite Gaussian mixture models are powerful tools for modeling distributions of random phenomena and are widely used for clustering tasks. However, their interpretability and efficiency are often degraded by the impact of redundancy and noise, especially on high-dimensional datasets. In this work, we propose a generative graphical model for parsimonious modeling of the Gaussian mixtures and robust unsupervised learning. The model assumes that the data are generated independently and identically from a finite mixture of robust factor analyzers, where the features’ salience is adjusted by an active set of latent factors to allow a violation of the local independence assumption. For the model inference, we propose a structured variational Bayes inference framework to realize simultaneous clustering, model selection and outlier processing. Performance of the proposed algorithm is evaluated by conducting experiments on artificial and real-world datasets. Moreover, an application on the high-dimensional machine learning task of handwritten alphabet recognition is introduced.

1. Introduction

Finite Gaussian mixture models are powerful tools for modeling distributions of random phenomena. They are widely used for unsupervised classification tasks and lay the foundation for many deep learning-based clustering algorithms, e.g., [1,2]. However, competitive performance of the Gaussian mixture model cannot be expected on high-dimensional datasets due to the curse of dimensionality [3]. The impact of redundancy and noise can degrade the model’s interpretability and efficiency, which is crucial in many application fields such as molecular biology and the clinical medicine [4]. Since the intrinsic dimensions of high-dimensional data are usually much less than their original feature space, it is possible to improve the clustering performance via dimension-reduction methods [5].

Feature-selection approaches are designed to retain a subset of features that are informative and discriminant for clustering. The two-stage methods implement the feature selection and clustering separately, which consider the preselected features as input without regarding the subsequent clustering algorithms [6,7]. But as choosing the feature subset and clustering are highly dependent problems, to circumvent the loss of information, incorporating the feature selection in the clustering algorithms and constructing an integrated objective function is suggested [3,4]. Pan and Shen [8] proposed a penalized likelihood approach for unsupervised feature selection where they used an $L_1$ penalty to shrink the component means. A similar approach was also suggested in [9], where the feature-selection consistency via the penalization was further studied. However, as both approaches are highly dependent on the choosing of penalization parameters, cross-validation or criterion-based model selection is required to tune the parameters.

A different stream of research casts the feature selection as parameter-estimation problems [4,10,11,12], where the random variable “feature saliency” is introduced to quantify the relevance of features to class assignment. This approach is efficient as neither combinatorial search through the feature subsets nor tuning of parameters is required. The feature selection and the clustering can be performed simultaneously in a principled and automatic way. Zhang et al. [13] extended this method to the Student’s t mixture model, which has higher tolerance to outliers and therefore is more robust for clustering and feature selection. As an extension to the work of Zhang et al., Sun and Zhou [14] made a full-Bayesian treatment for the model and proposed the structured variational Bayesian (VB) approach, which takes into consideration the estimation uncertainty of all model parameters and can deliver a tighter bound to the marginal likelihood than the mean-field approximated VB algorithms. Their model was extended further in [3] to consider the class-specific feature saliency for Bayesian feature selection.

Feature-selection approaches commonly assume that the features are conditionally independent given the latent class variable, which is equivalent to adopting a diagonal component covariance matrix structure in the Gaussian mixture model. While this assumption greatly facilitates computational efficiency, it can be easily violated in real-world datasets [15]. For the linear regression analysis, Fan et al. [16] conducted a synthetic study indicating that when the covariates are highly correlated exact recovery of the active set from the solution path of LASSO will be difficult. For classification, as has been mentioned in [17], ignoring the dependence relationship across features may undermine the reliability of the algorithms and lead to misleading conclusions about the features’ salience.

In [18], the local independence assumption for the Gaussian mixture model was relaxed by a block-diagonal specification of the component covariance matrix, where the features are partitioned into several disconnected groups in each class. However, as the total number of block-diagonal structures increases as the Bell number [19], searching for the optimal model can be difficult, especially in the high-dimensional cases. Galimberti and Soffritti [18] proposed a hierarchical aggregative strategy based on the BIC criterion to perform a nonexhaustive search of the structures. But this method cannot promise to find the optimal model. Ruan et al. [20] extended the graphical LASSO method to the context of the Gaussian mixture model and proposed a penalized likelihood approach for a sparse solution of the component covariance matrices. But the penalization parameters still need to be selected.

Different to the block-diagonal specification, the model of mixture of factor analyzers assumes a factor analysis-based decomposition structure for the component covariance matrices, where the local dependence between features is explained by a few latent factors [21,22,23,24]. Typically, the number of factors for each component needs to be specified in advance of the model fitting or the model-selection criterion used to select the optimal number of factors. However, while presetting of the number may lead to over-fitted or over-simplified models, conducting exhaustive searches over the model space is computationally expensive. The shrinkage prior methods were proposed to achieve an automatic latent dimension reduction. Wang and Lan [25] imposed automatic relevance determination prior [26] on the factor loading matrices. Multiplicative Gamma process shrinkage priors [27] in the infinite factor analysis was used by Murphy et al. [28]. They also suggested an adaptive Gibbs sampling algorithm where the factors with negligible loadings are removed gradually during the iterations. Therefore, the computational efficiency can be improved.

In this paper, we develop further the Student’s t mixture model of Sun and Zhou [14] for Bayesian clustering and feature selection to tackle the cases where features are correlated in the mixture component. While the model in [14] defines the feature saliency under the local independence assumption, we introduce the factor-adjusted feature saliency, where the salience of each feature is evaluated by conditioning on the latent factors. Taken as a whole, the extension produces parsimonious, flexible and robust modeling for the mixture of factor analyzers. Moreover, motivated by the Bayesian model selection method in [29] for the linear regression analysis, instead of using the shrinkage priors, we propose an automatic inference scheme for the number of factors by introducing the random variable of factor activity. Then, the problems of feature selection, latent dimension reduction, outlier processing and clustering can be integrated together as the inference for a Bayesian hierarchical latent variable model. We continue the work in [14] to adopt a full Bayesian treatment, where proper prior distributions are assumed for the model parameters. The structured VB inference framework that improves the evidence lower bound (ELBO) for the proposed model is presented, where a “drop-out” sampling technique [30] can be applied immediately to ease the computation.

The rest of this paper is organized as follows: In Section 2, we introduce the Student’s t mixture model proposed by Sun and Zhou [14], which has provided the base of our study. In Section 3, we develop the proposed mixture of robust factor analyzers, which is present as a hierarchical latent variable model for Bayesian inference. In Section 4, the structured VB inference framework for the proposed model is established. Section 5 justifies the performance of the developed model and algorithm on the synthetic data and presents the evaluation results based on some real-world datasets. Section 6 concludes this paper, points out the limitations and suggests future research directions.

2. The Student’s t Mixture Model for Feature Selection

We present the proposed hierarchical latent variable model starting from the Student’s t mixture model defined in [14]. Denote the set of i.i.d. observations as $\mathbf{Y}={\left\{{\mathit{y}}_n\right\}}_{n=1}^N$, where ${\mathit{y}}_n={(y_{n1},y_{n2},\dots ,y_{nd})}^T\in {\mathbb{R}}^d$ is the d-dimensional feature data for the nth individual. The finite mixture model for clustering assumes that the data for each individual are generated from a class-specific distribution but with the class label missing, then it marginally follows a finite mixture distribution. Throughout the paper, we denote the number of mixture components or equally the number of classes as K. The latent class label for the nth individual is denoted as $z_n$ which takes value in $\{1,2,\dots ,K\}$. The clustering is realized by assigning each individual the class label where it has the highest posterior probability of belonging.

The Student’s t mixture model given in [14] assumes that the features are conditionally independent given the hidden class label and each follows a Student’s t distribution. Moreover, the relevance or irrelevance of feature to data separation is taken into account by introducing the Bernoulli latent variables ${\mathit{\varphi }}_n={({\varphi }_{n1},{\varphi }_{n2},\dots ,{\varphi }_{nd})}^T$, which gives the mixture density of ${\mathit{y}}_n$ as

$$p\left({\mathit{y}}_n\right|{\mathit{\varphi }}_n,{\Theta }_1)=\sum _{k=1}^K{\pi }_k\prod _{l=1}^d\left[{\mathcal{S}}_t{\left(y_{nl}\right|{\mu }_{kl},{\sigma }_{kl},v_{kl})}^{{\varphi }_{nl}}{\mathcal{S}}_t{\left(y_{nl}\right|{\mu }_{0l},{\sigma }_{0l},v_{0l})}^{1-{\varphi }_{nl}}\right].$$

(1)

${\mathcal{S}}_t\left(y\right|\mu ,\sigma ,v)$ is the density function of the Student’s t distribution with mean, precision and degrees of freedom as $\mu$, $\sigma$ and v, respectively. For $l=1,2,\dots ,d$, ${\varphi }_{nl}\in \{0,1\}$, if ${\varphi }_{nl}=1$, then the lth feature is relevant to the class assignment; if ${\varphi }_{nl}=0$, then the lth feature is irrelevant and follows a common distribution independent of the class assignment. For $k=1,2,\dots ,K$, the parameter ${\pi }_k$ (${\pi }_k>0$ and ${\sum }_{k=1}^K{\pi }_k=1$) is the mixing proportion of class k. Let ${\Theta }_1=\{\mathit{\pi },\mathit{\mu },\mathit{\sigma },\mathit{v}\}$ denote the set of unknown parameters in model (1), where $\mathit{\pi }={({\pi }_1,{\pi }_2,\dots ,{\pi }_K)}^T$, $\mathit{\mu }={\left\{{\mu }_{0l},{\left\{{\mu }_{kl}\right\}}_{k=1}^K\right\}}_{l=1}^d$, $\mathit{\sigma }={\left\{{\sigma }_{0l},{\left\{{\sigma }_{kl}\right\}}_{k=1}^K\right\}}_{l=1}^d$ and $\mathit{v}={\left\{v_{0l},{\left\{v_{kl}\right\}}_{k=1}^K\right\}}_{l=1}^d$.

The prior distribution of ${\mathit{\varphi }}_n$ is given by

$$p\left({\mathit{\varphi }}_n\right|\mathit{\beta })=\prod _{l=1}^{d}p\left({\varphi }_{nl}\right|{\beta }_l)=\prod _{l=1}^d{{\beta }_l}^{{\varphi }_{nl}}{(1-{\beta }_l)}^{1-{\varphi }_{nl}},$$

(2)

where the ${\varphi }_{nl}$’s are assumed to be mutually independent. The parameter ${\beta }_l$ for the Bernoulli distribution of ${\varphi }_{nl}$ is called the feature saliency [4] of the lth feature. It measures the importance of the feature for class assignment and is estimated to realize a “soft” feature selection. Denote $\mathit{\beta }={\left\{{\beta }_l\right\}}_{l=1}^d$.

The observed-data likelihood function can be obtained by integrating over the latent variables ${\mathit{\varphi }}_n$ in model (1), which gives

$$p\left({\mathit{y}}_n\right|{\Theta }_2)=\sum _{k=1}^K{\pi }_k\prod _{l=1}^d\left[{\beta }_l{\mathcal{S}}_t\left(y_{nl}\right|{\mu }_{kl},{\sigma }_{kl},v_{kl})+(1-{\beta }_l){\mathcal{S}}_t\left(y_{nl}\right|{\mu }_{0l},{\sigma }_{0l},v_{0l})\right],$$

(3)

where ${\Theta }_2=\{\mathit{\pi },\mathit{\mu },\mathit{\sigma },\mathit{v},\mathit{\beta }\}$. Statistical inference directly on the observed-data likelihood is difficult. In [14], the VB inference method was adopted where the complete-data likelihood is given by

$$p({\mathit{y}}_n,{\mathit{u}}_n,{\mathit{\varphi }}_n,z_n|{\Theta }_2)=\prod _{k=1}^K{\left[{\pi }_{k}p\left({\mathit{y}}_n\right|{\mathit{u}}_n,{\mathit{\varphi }}_n,z_n=k,\mathit{\mu },\mathit{\sigma })p\left({\mathit{u}}_n\right|{\mathit{\varphi }}_n,z_n=k,\mathit{v})p\left({\mathit{\varphi }}_n\right|\mathit{\beta })\right]}^{{\delta }_{z_n,k}}.$$

(4)

At the right-hand side of (4), ${\delta }_{z_n,k}$ is the Kronecker delta function. The latent variables ${\mathit{u}}_{\mathit{n}}={(u_{n1},u_{n2},\dots ,u_{nd})}^T$ are introduced by noting that the Student’s t distribution can be written as a convolution of a Gaussian and a Gamma distribution [3,14]. It follows that

$$\begin{array}{cc}\hfill p\left({\mathit{y}}_n\right|{\mathit{u}}_n,{\mathit{\varphi }}_n,z_n=k,\mathit{\mu },\mathit{\sigma })& =\prod _{l=1}^{d}p\left(y_{nl}\right|u_{nl},{\varphi }_{nl},z_n=k,\mathit{\mu },\mathit{\sigma })\hfill \\ \hfill & =\prod _{l=1}^d\left[\mathcal{N}{\left(y_{nl}\right|{\mu }_{kl},{\sigma }_{kl}{u}_{nl})}^{{\varphi }_{nl}}\mathcal{N}{\left(y_{nl}\right|{\mu }_{0l},{\sigma }_{0l}{u}_{nl})}^{1-{\varphi }_{nl}}\right],\hfill \end{array}$$

(5)

and

$$\begin{array}{cc}\hfill p\left({\mathit{u}}_n\right|{\mathit{\varphi }}_n,z_n=k,\mathit{v})& =\prod _{l=1}^{d}p\left(u_{nl}\right|{\varphi }_{nl},z_n=k,\mathit{v})\hfill \\ \hfill & =\prod _{l=1}^d\left[\mathcal{G}{\left(u_{nl}|\frac{v_{kl}}{2},\frac{v_{kl}}{2}\right)}^{{\varphi }_{nl}}\mathcal{G}{\left(u_{nl}|\frac{v_{0l}}{2},\frac{v_{0l}}{2}\right)}^{1-{\varphi }_{nl}}\right].\hfill \end{array}$$

(6)

$\mathcal{N}\left(y\right|\mu ,\sigma )$ represents the Gaussian density function with mean $\mu$ and precision $\sigma$ and $\mathcal{G}\left(u\right|a,b)$ is the Gamma density function

$$\mathcal{G}\left(u\right|a,b)=\frac{b^a{u}^{a-1}}{\Gamma \left(a\right)}exp(-bu).$$

(7)

Note that by integrating over ${\mathit{u}}_n$, ${\mathit{\varphi }}_n$ and $z_n$ in (4), the observed-data likelihood (3) can be recovered.

3. Towards the Mixture of Robust Factor Analyzers

To tackle the cases where features are correlated in the mixture component, we relax the local independence assumption in [14] by specifying for each class a latent factor model. Specifically, for class k, we introduce the latent factors ${\mathit{x}}_{nk}={(x_{nk1},x_{nk2},\dots ,x_{nk{p}_k})}^T$, where $x_{nkj}$’s are i.i.d. from the distribution $\mathcal{N}(0,1)$ and $p_k$ is the number of latent factors. After conditioning on ${\mathit{x}}_{nk}$, the features are assumed mutually independent within the class, which corresponds to a modification of model (1) as follows:

$$p\left({\mathit{y}}_n\right|{\mathit{\varphi }}_n,{\mathit{x}}_n,{\Theta }_3)=\sum _{k=1}^K{\pi }_k\prod _{l=1}^d\left[{\mathcal{S}}_t{\left(y_{nl}\right|{\mathit{w}}_{kl}^T{\mathit{x}}_{nk}+{\mu }_{kl},{\sigma }_{kl},v_{kl})}^{{\varphi }_{nl}}{\mathcal{S}}_t{\left(y_{nl}\right|{\mathit{w}}_{kl}^T{\mathit{x}}_{nk}+{\mu }_{0l},{\sigma }_{0l},v_{0l})}^{1-{\varphi }_{nl}}\right],$$

(8)

where ${\mathit{w}}_{kl}\in {\mathbb{R}}^{p_k}$ are the factor loadings for the lth feature in class k. Denote ${\mathit{x}}_n={\left\{{\mathit{x}}_{nk}\right\}}_{k=1}^K$ and ${\Theta }_3=\{\mathit{\pi },\mathit{\mu },\mathit{\sigma },\mathit{v},\mathit{w}\}$ where $\mathit{w}={\left\{{\left\{{\mathit{w}}_{kl}\right\}}_{l=1}^d\right\}}_{k=1}^K$. In model (8), ${\varphi }_{nl}$ indicates the relevance of the lth feature to class assignment after adjustment by the latent factors. Correspondingly, ${\beta }_l$ that defines the distribution of ${\varphi }_{nl}$ in (2) represents the factor-adjusted feature saliency.

Typically, in each local factor model, the latent dimensions $p_k$ need to be specified. With overly high dimensions, the model may over-fit the data, yielding poor interpretations and hardening the computation, while with low and inadequate dimensions, the model may not be flexible enough to capture the correlations between features in each class. To enable an automatic determination, we treat the problem as another feature-selection task, but now the “features” become the latent factors. Starting from a sufficiently large $p_k$, we introduce in class k the Bernoulli latent variables ${\mathit{r}}_{nk}={(r_{nk1},r_{nk2},\dots ,r_{nk{p}_k})}^T$, where $r_{nkj}\in \{0,1\}$ with $r_{nkj}=1$ indicating that the factor $x_{nkj}$ is active and $r_{nkj}=0$ inactive. Model (8) then becomes

$$p\left({\mathit{y}}_n\right|{\mathit{\varphi }}_n,{\mathit{x}}_n,{\mathit{r}}_n,{\Theta }_3)=\sum _{k=1}^K{\pi }_k\prod _{l=1}^d\left[{\mathcal{S}}_t{\left(y_{nl}\right|{\mathit{w}}_{kl}^T{\mathbf{R}}_{nk}{\mathit{x}}_{nk}+{\mu }_{kl},{\sigma }_{kl},v_{kl})}^{{\varphi }_{nl}}{\mathcal{S}}_t{\left(y_{nl}\right|{\mathit{w}}_{kl}^T{\mathbf{R}}_{nk}{\mathit{x}}_{nk}+{\mu }_{0l},{\sigma }_{0l},v_{0l})}^{1-{\varphi }_{nl}}\right],$$

(9)

where ${\mathbf{R}}_{nk}=\mathrm{diag}\left({\mathit{r}}_{nk}\right)$ and we denote ${\mathit{r}}_n={\left\{{\mathit{r}}_{nk}\right\}}_{k=1}^K$. When $r_{nkj}$’s all equal zero, the model reduces to the Student’s t mixtures of model (1).

The prior distribution of ${\mathit{r}}_{nk}$ is given by

$$p\left({\mathit{r}}_{nk}\right|{\mathit{\rho }}_k)=\prod _{j=1}^{p_k}p\left(r_{nkj}\right|{\rho }_{kj})=\prod _{j=1}^{p_k}{{\rho }_{kj}}^{r_{nkj}}{(1-{\rho }_{kj})}^{1-r_{nkj}},$$

(10)

where we have assumed prior independence between the entries of ${\mathit{r}}_{nk}$. Denote ${\mathit{\rho }}_k={\left\{{\rho }_{kj}\right\}}_{j=1}^{p_k}$ and $\mathit{\rho }={\left\{{\mathit{\rho }}_k\right\}}_{k=1}^K$. In accordance with the concept of feature saliency, we call ${\rho }_{kj}$ the factor activity. It is the probability that the jth factor in class k is active. The problem of finding latent dimensions then can be cast as a parameter-estimation problem, i.e., the estimation of $\mathit{\rho }$.

Our modeling of ${\mathit{r}}_{nk},k=1,2,\dots ,K$ to select the active factors in each class is inspired by the normal-zero model proposed in [29], which introduces the indicators to select automatically the important covariates in linear regression. The difference is that we have defined the indicators as latent variables for each individual, while the normal-zero model introduces the indicators as model parameters. In our model, the parameters $\mathit{\rho }$ that define the Bernoulli distributions of the indicators are the key quantities for model selection and will be inferred under a Bayesian inference framework, while following the normal-zero model $\mathit{\rho }$ are treated as hyper-parameters and typically need to be specified.

Note that the conditional probability of (9) defines a mixture of robust factor analyzers. The factor model for class k can be written as

$${\mathit{y}}_n={\mathbf{W}}_k{\mathbf{R}}_{nk}{\mathit{x}}_{nk}+{\mathsf{\Phi }}_n{\mathit{\mu }}_k+(\mathbf{I}-{\mathsf{\Phi }}_n){\mathit{\mu }}_0+{\mathit{\epsilon }}_n,$$

(11)

where ${\mathbf{W}}_k={[{\mathit{w}}_{k1},{\mathit{w}}_{k2},\dots ,{\mathit{w}}_{kd}]}^T$ is the $d\times p_k$ factor loading matrix. Denote ${\mathit{\mu }}_k=({\mu }_{k1},{\mu }_{k2},\dots ,$${\mu }_{kd}{)}^T$, ${\mathit{\mu }}_0={\left({\mu }_{01},{\mu }_{02},\dots ,{\mu }_{0d}\right)}^T$ and ${\mathsf{\Phi }}_n=\mathrm{diag}\left({\mathit{\varphi }}_n\right)$. The latent factors ${\mathit{x}}_{nk}$ follow the distribution of $\mathcal{N}(\mathbf{0},{\mathbf{I}}_{p_k})$, where ${\mathbf{I}}_{p_k}$ is the identity matrix of order $p_k$. The distributions for ${\mathit{\varphi }}_n$ and ${\mathit{r}}_{nk}$ are defined in (2) and (10), separately. ${\mathit{\epsilon }}_n={({\epsilon }_{n1},{\epsilon }_{n2},\dots ,{\epsilon }_{nd})}^T$, where ${\epsilon }_{nl}$’s are mutually independent given $z_n$ and

$$p\left({\epsilon }_{nl}\right|{\varphi }_{nl},z_n=k,\mathit{\sigma },\mathit{v})={\mathcal{S}}_t{\left({\epsilon }_{nl}\right|0,{\sigma }_{kl},v_{kl})}^{{\varphi }_{nl}}{\mathcal{S}}_t{\left({\epsilon }_{nl}\right|0,{\sigma }_{0l},v_{0l})}^{1-{\varphi }_{nl}}.$$

(12)

By introducing the latent variable $u_{nl}$ distributed according to (6), we have

$$p\left({\epsilon }_{nl}\right|u_{nl},{\varphi }_{nl},z_n=k,\mathit{\sigma })=\mathcal{N}{\left({\epsilon }_{nl}\right|0,{\sigma }_{kl}{u}_{nl})}^{{\varphi }_{nl}}\mathcal{N}{\left({\epsilon }_{nl}\right|0,{\sigma }_{0l}{u}_{nl})}^{1-{\varphi }_{nl}}.$$

(13)

The complete-data likelihood $p({\mathit{y}}_n,{\mathit{u}}_n,{\mathit{\varphi }}_n,{\mathit{x}}_n,{\mathit{r}}_n,z_n|\Theta )$ for the proposed hierarchical latent variable model, where $\Theta =\{\mathit{\pi },\mathit{\mu },\mathit{\sigma },\mathit{v},\mathit{\beta },\mathit{w},\mathit{\rho }\}$ can be factorized as

$$\begin{array}{cc}\hfill p({\mathit{y}}_n,{\mathit{u}}_n,{\mathit{\varphi }}_n,{\mathit{x}}_n,{\mathit{r}}_n,z_n|\Theta )& =\prod _{k=1}^K[{\pi }_{k}p\left({\mathit{y}}_n\right|{\mathit{u}}_n,{\mathit{\varphi }}_n,{\mathit{x}}_{nk},{\mathit{r}}_{nk},z_n=k,\mathit{\mu },\mathit{\sigma },\mathit{w})\hfill \\ \hfill & \times p\left({\mathit{u}}_n\right|{\mathit{\varphi }}_n,z_n=k,\mathit{v})p\left({\mathit{\varphi }}_n\right|\mathit{\beta })p\left({\mathit{x}}_{nk}\right)p\left({\mathit{r}}_{nk}\right|{\mathit{\rho }}_k){]}^{{\delta }_{z_n,k}},\hfill \end{array}$$

(14)

where

$$\begin{array}{cc}\hfill p\left({\mathit{y}}_n\right|{\mathit{u}}_n,{\mathit{\varphi }}_n,{\mathit{x}}_{nk},{\mathit{r}}_{nk},z_n=k,\mathit{\mu },\mathit{\sigma },\mathit{w})& =\prod _{l=1}^{d}p\left(y_{nl}\right|u_{nl},{\varphi }_{nl},{\mathit{x}}_{nk},{\mathit{r}}_{nk},z_n=k,\mathit{\mu },\mathit{\sigma },\mathit{w})\hfill \\ \hfill & =\prod _{l=1}^d\left[\mathcal{N}{\left(y_{nl}\right|{\mathit{w}}_{kl}^T{\mathbf{R}}_{nk}{\mathit{x}}_{nk}+{\mu }_{kl},{\sigma }_{kl}{u}_{nl})}^{{\varphi }_{nl}}\mathcal{N}{\left(y_{nl}\right|{\mathit{w}}_{kl}^T{\mathbf{R}}_{nk}{\mathit{x}}_{nk}+{\mu }_{0l},{\sigma }_{0l}{u}_{nl})}^{1-{\varphi }_{nl}}\right],\hfill \end{array}$$

(15)

corresponding to a modification of conditional probability (5) for the Student’s t mixture model.

In the following, we denote the set of latent variables as $\mathcal{H}={\left\{{\mathit{h}}_n\right\}}_{n=1}^N$ where ${\mathit{h}}_n=\{{\mathit{u}}_n,{\mathit{\varphi }}_n,{\mathit{x}}_n,{\mathit{r}}_n,z_n\}$. Then, the complete-data likelihood for the whole dataset can be written as

$$p(\mathbf{Y},\mathcal{H}|\Theta )=\prod _{n=1}^{N}p({\mathit{y}}_n,{\mathit{h}}_n|\Theta ).$$

(16)

Full Bayesian treatment to the latent variable model requires specification of the prior distributions associated with the model parameters. We assume that

$$p(\Theta )=p\left(\mathit{\pi }\right)p\left(\mathit{\mu }\right)p\left(\mathit{\sigma }\right)p\left(\mathit{\beta }\right)p\left(\mathit{w}\right)p\left(\mathit{\rho }\right),$$

(17)

and

$$\begin{array}{cc}\hfill p\left(\mathit{\pi }\right)& =\mathcal{D}ir\left(\mathit{\pi }\right|{\mathit{\alpha }}_0),\hfill \\ \hfill p\left(\mathit{\mu }\right)& =\prod _{l=1}^d\left[p\left({\mu }_{0l}\right)\prod _{k=1}^{K}p\left({\mu }_{kl}\right)\right]=\prod _{l=1}^d\left[\mathcal{N}\left({\mu }_{0l}\right|s_{0l},{\lambda }_0)\prod _{k=1}^K\mathcal{N}\left({\mu }_{kl}\right|s_{0l},{\lambda }_0)\right],\hfill \\ \hfill p\left(\mathit{\sigma }\right)& =\prod _{l=1}^d\left[p\left({\sigma }_{0l}\right)\prod _{k=1}^{K}p\left({\sigma }_{kl}\right)\right]=\prod _{l=1}^d\left[\mathcal{G}\left({\sigma }_{0l}|\frac{{\eta }_0}{2},\frac{{\xi }_0}{2}\right)\prod _{k=1}^K\mathcal{G}\left({\sigma }_{kl}|\frac{{\eta }_0}{2},\frac{{\xi }_0}{2}\right)\right],\hfill \\ \hfill p\left(\mathit{\beta }\right)& =\prod _{l=1}^d\mathcal{B}eta\left({\beta }_l\right|{\kappa }_1,{\kappa }_2),\hfill \\ \hfill p\left(\mathit{w}\right)& =\prod _{k=1}^K\prod _{l=1}^{d}p\left({\mathit{w}}_{kl}\right)=\prod _{k=1}^K\prod _{l=1}^d\mathcal{N}\left({\mathit{w}}_{kl}\right|\mathbf{0},m_0{\mathbf{I}}_{p_k}),\hfill \\ \hfill p\left(\mathit{\rho }\right)& =\prod _{k=1}^K\prod _{j=1}^{p_k}\mathcal{B}eta\left({\rho }_{kj}\right|{\tau }_1,{\tau }_2),\hfill \end{array}$$

(18)

where $\mathcal{B}eta\left(\beta \right|a,b)$ represents the Beta density function

$$\mathcal{B}eta\left(\beta \right|a,b)=\frac{{\beta }^{a-1}{(1-\beta )}^{b-1}}{\mathcal{B}(a,b)},$$

(19)

and

$$\mathcal{D}ir\left(\mathit{\pi }\right|\mathit{\alpha })=\frac{\Gamma \left({\sum }_{k=1}^K{\alpha }_k\right)}{{\prod }_{k=1}^K\Gamma \left({\alpha }_k\right)}\prod _{k=1}^K{{\pi }_k}^{{\alpha }_k-1},$$

(20)

is the Dirichlet density. In the above specifications, the conjugate priors are used. The parameters in the priors, including ${\kappa }_1$, ${\kappa }_2$, ${\tau }_1$, ${\tau }_2$, $m_0$, ${\lambda }_0$, ${\eta }_0$, ${\xi }_0$, ${\mathit{s}}_0$ and ${\mathit{\alpha }}_0$ where ${\mathit{s}}_0={\left\{s_{0l}\right\}}_{l=1}^d$ and ${\mathit{\alpha }}_0={({\alpha }_{01},{\alpha }_{02},\dots ,{\alpha }_{0K})}^T$, are considered as hyperparameters. It is noticeable that we do not assume any prior for the degrees of freedom $v_{0l}$’s and $v_{kl}$’s. Since there are no conjugate priors, we follow the practice in [3,14] to seek for the point estimates for them.

The plate diagram of the proposed hierarchical latent variable model is shown in Figure 1. The arrows in the diagram indicate the dependencies. The original model of Sun and Zhou [14] is depicted in blue.

4. Inference on the Model

4.1. Brief Introduction to VB Method

To infer from the posterior distribution of the latent variables and the parameters, computation of the evidence $p\left(\mathbf{Y}\right)$ is required. However, the computation involves integration over the latent variables and the parameters, which is intractable for our model. In this paper, we resort to the VB method for model inference. It is designed to maximize a lower bound of $logp\left(\mathbf{Y}\right)$. Assuming posterior independence between the latent variables and the parameters, the evidence lower bound (ELBO) is defined by

$$\mathcal{L}(q\left(\mathcal{H}\right),q(\Theta ),\mathbf{Y})={\mathbb{E}}_{q\left(\mathcal{H}\right)q(\Theta )}\left[log\frac{p(\mathbf{Y},\mathcal{H}|\Theta \left)p\right(\Theta )}{q\left(\mathcal{H}\right)q(\Theta )}\right]\le logp\left(\mathbf{Y}\right),$$

(21)

where $q\left(\mathcal{H}\right)$ and $q(\Theta )$ are auxiliary posteriors for the latent variables and the parameters, respectively. A coordinate ascent search method [31] can be applied to iteratively maximize the ELBO. At the tth iteration, it implements the VB expectation (VB-E) step and the VB maximization (VB-M) step as follows:

$$\begin{array}{cc}\hfill \mathrm{VB}-\mathrm{E}\mathrm{step}:& q^{(t+1)}\left(\mathcal{H}\right)=\underset{q\left(\mathcal{H}\right)}{argmax}\mathcal{L}(q\left(\mathcal{H}\right),q^{\left(t\right)}(\Theta ),\mathbf{Y});\hfill \\ \hfill \mathrm{VB}-\mathrm{M}\mathrm{step}:& q^{(t+1)}(\Theta )=\underset{q(\Theta )}{argmax}\mathcal{L}(q^{(t+1)}\left(\mathcal{H}\right),q(\Theta ),\mathbf{Y}).\hfill \end{array}$$

(22)

4.2. Tree-Like Factorization of the Auxiliary Posterior

In this paper, we apply the tree-like factorization proposed in [3,14] to the auxiliary posterior of the latent variables. The resultant structured VB method can be viewed as a partially collapsed VB [32], which can reach a tighter lower bound for $logp\left(\mathbf{Y}\right)$ than the mean-filed approximated VB method [13].

As the observations are mutually independent, $q\left(\mathcal{H}\right)$ has the form

$$q\left(\mathcal{H}\right)=\prod _{n=1}^{N}q\left({\mathit{h}}_n\right).$$

(23)

Tree-like factorization assumes that the auxiliary posterior $q\left({\mathit{h}}_n\right)$ can be factorized as

$$q\left({\mathit{h}}_n\right)=\prod _{k=1}^K{\left\{q\left({\mathit{u}}_n\right|{\mathit{\varphi }}_n,z_n=k)q\left({\mathit{\varphi }}_n\right)q\left({\mathit{x}}_{nk}\right|{\mathit{r}}_{nk})q\left({\mathit{r}}_{nk}\right)q(z_n=k)\right\}}^{{\delta }_{z_n,k}}.$$

(24)

As entries of the noise term in the local factor model are assumed to be mutually independent, $q\left({\mathit{h}}_n\right)$ can be further factorized as

$$\begin{array}{cc}\hfill q\left({\mathit{h}}_n\right)=& \prod _{k=1}^K\{\prod _{l=1}^d\left[{\left(q\left(u_{nl}\right|{\varphi }_{nl}=1,z_n=k)q({\varphi }_{nl}=1)\right)}^{{\varphi }_{nl}}{\left(q\left(u_{nl}\right|{\varphi }_{nl}=0)q({\varphi }_{nl}=0)\right)}^{1-{\varphi }_{nl}}\right]\hfill \\ \hfill & \times q\left({\mathit{x}}_{nk}\right|{\mathit{r}}_{nk})q\left({\mathit{r}}_{nk}\right)q(z_n=k){\}}^{{\delta }_{z_n,k}}.\hfill \end{array}$$

(25)

Different from [3,14], we do not keep the posterior dependence of ${\varphi }_{nl}$ on $z_n$ and when ${\varphi }_{nl}=0$ the auxiliary posterior of $u_{nl}$ is assumed to be independent of $z_n$, though the closed forms of the posteriors are available when retaining the dependencies. We found that the above specifications lead to more robust inference results. As in [29,33], we assume a full factorization for $q\left({\mathit{r}}_{nk}\right)$, i.e.,

$$q\left({\mathit{r}}_{nk}\right)=\prod _{j=1}^{p_k}q\left(r_{nkj}\right)=\prod _{j=1}^{p_k}q{(r_{nkj}=1)}^{r_{nkj}}q{(r_{nkj}=0)}^{1-r_{nkj}}.$$

(26)

Additionally, the auxiliary posterior $q(\Theta )$ is assumed to be its full factorized form

$$\begin{array}{cc}\hfill q(\Theta )& =q\left(\mathit{\pi }\right)q\left(\mathit{\mu }\right)q\left(\mathit{\sigma }\right)q\left(\mathit{\beta }\right)q\left(\mathit{w}\right)q\left(\mathit{\rho }\right)\hfill \\ \hfill & =q\left(\mathit{\pi }\right)·\prod _{l=1}^d\left[q\left({\mu }_{0l}\right)\prod _{k=1}^{K}q\left({\mu }_{kl}\right)\right]·\prod _{l=1}^d\left[q\left({\sigma }_{0l}\right)\prod _{k=1}^{K}q\left({\sigma }_{kl}\right)\right]·\prod _{l=1}^{d}q\left({\beta }_l\right)·\prod _{k=1}^K\prod _{l=1}^{d}q\left({\mathit{w}}_{kl}\right)·\prod _{k=1}^K\prod _{j=1}^{p_k}q\left({\rho }_{kj}\right).\hfill \end{array}$$

(27)

For ease of exposition, we use n, l, j and k in the following to denote the index of the individual, the feature, the latent factor and the class, respectively. We omit the iteration indexes $\left(t\right)$ and $(t+1)$ and without loss of generosity deliver the update during one iteration of the algorithm. We use $\langle ·\rangle$ to denote the expectation operation with respect to the current auxiliary posteriors.

4.3. Auxiliary Posteriors of the Latent Variables: VB-E Step

The VB-E step updates the auxiliary posterior $q\left({\mathit{h}}_n\right)$ of the latent variables following the factorizations of (25) and (26).

(i) $q\left(u_{nl}\right|{\varphi }_{nl},z_n)$: Through some mathematical manipulations (see the Supplementary Materials for the details), we obtain

$$\begin{array}{cc}\hfill q(u_{nl}|{\varphi }_{nl}=1,z_n=k)& =\mathcal{G}\left(u_{nl}\right|{\widehat{a}}_{kl},{\widehat{b}}_{nl}^k),\hfill \\ \hfill q(u_{nl}|{\varphi }_{nl}=0)& =\mathcal{G}\left(u_{nl}\right|{\widehat{a}}_{0l},{\widehat{b}}_{nl}^0),\hfill \end{array}$$

(28)

where

$$\begin{array}{cc}\hfill & {\widehat{a}}_{kl}=\frac{v_{kl}+1}{2},{\widehat{b}}_{nl}^k=\frac{v_{kl}+\langle {\sigma }_{kl}\rangle \langle {({\tilde{y}}_{nl}^k-{\mu }_{kl})}^2\rangle }{2},\hfill \\ \hfill & {\widehat{a}}_{0l}=\frac{v_{0l}+1}{2},{\widehat{b}}_{nl}^0=\frac{v_{0l}+\langle {\sigma }_{0l}\rangle {\sum }_k\langle {\delta }_{z_n,k}\rangle \langle {({\tilde{y}}_{nl}^k-{\mu }_{0l})}^2\rangle }{2},\hfill \end{array}$$

(29)

and ${\tilde{y}}_{nl}^k=y_{nl}-{\mathit{w}}_{kl}^T{\mathbf{R}}_{nk}{\mathit{x}}_{nk}$. Note that

$$\begin{array}{cc}\hfill \langle {({\tilde{y}}_{nl}^k-{\mu }_{kl})}^2\rangle =& {\left(y_{nl}-\langle {\mu }_{kl}\rangle \right)}^2+{\widehat{\lambda }}_{kl}^{-1}-2\left(y_{nl}-\langle {\mu }_{kl}\rangle \right){\langle {\mathit{w}}_{kl}\rangle }^T\langle {\mathbf{R}}_{nk}{\mathit{x}}_{nk}\rangle \hfill \\ \hfill & +\mathrm{tr}\left(\langle {\mathit{w}}_{kl}{{\mathit{w}}_{kl}}^T\rangle \langle {\mathit{x}}_{nk}{{\mathit{x}}_{nk}}^T\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle \right),\hfill \\ \hfill \langle {({\tilde{y}}_{nl}^k-{\mu }_{0l})}^2\rangle =& {\left(y_{nl}-\langle {\mu }_{0l}\rangle \right)}^2+{\widehat{\lambda }}_{0l}^{-1}-2\left(y_{nl}-\langle {\mu }_{0l}\rangle \right){\langle {\mathit{w}}_{kl}\rangle }^T\langle {\mathbf{R}}_{nk}{\mathit{x}}_{nk}\rangle \hfill \\ \hfill & +\mathrm{tr}\left(\langle {\mathit{w}}_{kl}{{\mathit{w}}_{kl}}^T\rangle \langle {\mathit{x}}_{nk}{{\mathit{x}}_{nk}}^T\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle \right),\hfill \end{array}$$

(30)

where ${\widehat{\lambda }}_{kl}$ is the precision of posterior $q\left({\mu }_{kl}\right)$ and ${\widehat{\lambda }}_{0l}$ is the precision of $q\left({\mu }_{0l}\right)$. We denote the trace operator as $\mathrm{tr}(·)$ and the Hadamard product operator between two matrices as ⊙.

In the sequel, we use ${\langle ·\rangle }_k^1$ and ${\langle ·\rangle }^0$ to distinguish between the expectations regarding $q\left(u_{nl}\right|{\varphi }_{nl}=1,z_n=k)$ and $q\left(u_{nl}\right|{\varphi }_{nl}=0)$. As with the property of Gamma distribution, we obtain

$$\begin{array}{c}\hfill {\langle u_{nl}\rangle }_k^1=\frac{{\widehat{a}}_{kl}}{{\widehat{b}}_{nl}^k},{\langle log{u}_{nl}\rangle }_k^1=\psi \left({\widehat{a}}_{kl}\right)-log\left({\widehat{b}}_{nl}^k\right),\\ \hfill {\langle u_{nl}\rangle }^0=\frac{{\widehat{a}}_{0l}}{{\widehat{b}}_{nl}^0},{\langle log{u}_{nl}\rangle }^0=\psi \left({\widehat{a}}_{0l}\right)-log\left({\widehat{b}}_{nl}^0\right),\end{array}$$

(31)

where $\psi (·)$ is the digamma function.

(ii) $q\left({\varphi }_{nl}\right)$: Define

$$\begin{array}{cc}\hfill \overline{q}({\varphi }_{nl}=1)& =exp\left\{\sum _k\langle {\delta }_{z_n,k}\rangle \left[\frac{1}{2}\langle log{\sigma }_{kl}\rangle +\frac{v_{kl}}{2}log\frac{v_{kl}}{2}-log\Gamma \left(\frac{v_{kl}}{2}\right)-{\widehat{a}}_{kl}log{\widehat{b}}_{nl}^k+log\Gamma \left({\widehat{a}}_{kl}\right)\right]+\langle log{\beta }_l\rangle \right\},\hfill \\ \hfill \overline{q}({\varphi }_{nl}=0)& =exp\left\{\frac{1}{2}\langle log{\sigma }_{0l}\rangle +\frac{v_{0l}}{2}log\frac{v_{0l}}{2}-log\Gamma \left(\frac{v_{0l}}{2}\right)-{\widehat{a}}_{0l}log{\widehat{b}}_{nl}^0+log\Gamma \left({\widehat{a}}_{0l}\right)+\langle log(1-{\beta }_l)\rangle \right\}.\hfill \end{array}$$

(32)

Then, $q\left({\varphi }_{nl}\right)$ can be obtained by

$$q({\varphi }_{nl}=1)=\frac{\overline{q}({\varphi }_{nl}=1)}{\overline{q}({\varphi }_{nl}=1)+\overline{q}({\varphi }_{nl}=0)},$$

(33)

and $q({\varphi }_{nl}=0)=1-q({\varphi }_{nl}=1)$. Denote $\langle {\varphi }_{nl}\rangle =q({\varphi }_{nl}=1)$ and $\langle 1-{\varphi }_{nl}\rangle =q({\varphi }_{nl}=0)$.

(iii) $q\left({\mathit{x}}_{nk}\right|{\mathit{r}}_{nk})$: The posterior $q\left({\mathit{x}}_{nk}\right|{\mathit{r}}_{nk})$ is multivariate Gaussian with precision matrix and mean vector as

$${\widehat{\mathbf{C}}}_n^k\left({\mathit{r}}_{nk}\right)=\mathbf{I}+{\mathbf{A}}_{nk}\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T,{\widehat{\mathit{f}}}_n^k\left({\mathit{r}}_{nk}\right)={\widehat{\mathbf{C}}}_n^k{\left({\mathit{r}}_{nk}\right)}^{-1}{\mathbf{R}}_{nk}{\mathit{t}}_{nk},$$

(34)

where

$$\begin{array}{cc}\hfill {\mathbf{A}}_{nk}& =\sum _l\left[\langle {\varphi }_{nl}\rangle \langle {\sigma }_{kl}\rangle {\langle u_{nl}\rangle }_k^1+\langle 1-{\varphi }_{nl}\rangle \langle {\sigma }_{0l}\rangle {\langle u_{nl}\rangle }^0\right]\langle {\mathit{w}}_{kl}{\mathit{w}}_{kl}^T\rangle ,\hfill \\ \hfill {\mathit{t}}_{nk}& =\sum _l\left[\langle {\varphi }_{nl}\rangle \langle {\sigma }_{kl}\rangle {\langle u_{nl}\rangle }_k^1\left(y_{nl}-\langle {\mu }_{kl}\rangle \right)+\langle 1-{\varphi }_{nl}\rangle \langle {\sigma }_{0l}\rangle {\langle u_{nl}\rangle }^0\left(y_{nl}-\langle {\mu }_{0l}\rangle \right)\right]\langle {\mathit{w}}_{kl}\rangle .\hfill \end{array}$$

(35)

(iv) $q\left({\mathit{r}}_{nk}\right)$: The posterior $q\left(r_{nkj}\right)$ can be obtained by

$$q(r_{nkj}=1)=\frac{\overline{q}(r_{nkj}=1)}{\overline{q}(r_{nkj}=1)+\overline{q}(r_{nkj}=0)},$$

(36)

and $q(r_{nkj}=0)=1-q(r_{nkj}=1)$, where

$$\begin{array}{cc}\hfill \overline{q}(r_{nkj}=c)=exp[& -\frac{1}{2}\langle log|\mathbf{I}+{\mathbf{A}}_{nk}\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T|\rangle \hfill \\ \hfill & +\frac{1}{2}\mathrm{tr}\langle {(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T)}^{-1}\left({\mathbf{R}}_{nk}{\mathit{t}}_{nk}\right){\left({\mathbf{R}}_{nk}{\mathit{t}}_{nk}\right)}^T\rangle \hfill \\ \hfill & +c\langle log{\rho }_{kj}\rangle +(1-c)\langle log(1-{\rho }_{kj})\rangle ],\hfill \end{array}$$

(37)

with $c\in \{0,1\}$. The expectations in (37) are taken by fixing $r_{nkj}=c$. Denote $\langle r_{nkj}\rangle =q(r_{nkj}=1)$ and $\langle 1-r_{nkj}\rangle =q(r_{nkj}=0)$.

When posterior independence is assumed between ${\mathit{x}}_{nk}$ and ${\mathit{r}}_{nk}$ or ${\mathit{x}}_{nk}$ is observable as in the regression models of [29,33], $q\left(r_{nkj}\right)$ can be derived analytically and the expectations in (30) regarding $q({\mathit{x}}_{nk},{\mathit{r}}_{nk})$ can be obtained in closed form using the results:

$$\begin{array}{cc}\hfill \langle {\mathbf{R}}_{nk}\rangle & =\mathrm{diag}\left(\langle {\mathit{r}}_{nk}\rangle \right),\hfill \\ \hfill \langle {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle & =\langle {\mathbf{R}}_{nk}\rangle \odot \langle \mathbf{I}-{\mathbf{R}}_{nk}\rangle +\langle {\mathit{r}}_{nk}\rangle {\langle {\mathit{r}}_{nk}\rangle }^T,\hfill \\ \hfill \langle {\mathit{x}}_{nk}\rangle & ={\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot \langle {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle \right)}^{-1}\langle {\mathbf{R}}_{nk}\rangle {\mathit{t}}_{nk},\hfill \\ \hfill \langle {\mathit{x}}_{nk}{\mathit{x}}_{nk}^T\rangle & ={\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot \langle {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle \right)}^{-1}+\langle {\mathit{x}}_{nk}\rangle {\langle {\mathit{x}}_{nk}\rangle }^T.\hfill \end{array}$$

(38)

However, the computation in high-dimensional cases is obstructed as it involves multiplication and inversion of large-scale matrices.

The sparse property of the indicator vector ${\mathit{r}}_{nk}$ motivates us to resort to a “drop-out” sampling scheme [30], where the conditioning of ${\mathit{x}}_{nk}$ on ${\mathit{r}}_{nk}$ does not influence the efficiency of the algorithm. Specifically, we keep a random sample ${\widehat{\mathit{r}}}_{nk}$ from $q\left({\mathit{r}}_{nk}\right)$ at each iteration of the algorithm and use it as an imputation for ${\mathit{r}}_{nk}$ to update the remaining auxiliary posteriors. During this process, the connections to the latent factor with smaller $q(r_{nkj}=1)$ have higher chance of drop out. Simplification of the computation can be realized, for example, in Equation (30),

$$\begin{array}{cc}\hfill {\langle {\mathit{w}}_{kl}\rangle }^T\langle {\mathbf{R}}_{nk}{\mathit{x}}_{nk}\rangle & ={\left({\widehat{\mathbf{R}}}_{nk}\langle {\mathit{w}}_{kl}\rangle \right)}^T{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}{\widehat{\mathbf{R}}}_{nk}{\mathit{t}}_{nk},\hfill \\ \hfill \langle {\mathit{w}}_{kl}{{\mathit{w}}_{kl}}^T\rangle \langle {\mathit{x}}_{nk}{\mathit{x}}_{nk}^T\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle & =\left(\langle {\mathit{w}}_{kl}{\mathit{w}}_{kl}^T\rangle \odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)[{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}\hfill \\ \hfill & +{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}\left({\widehat{\mathbf{R}}}_{nk}{\mathit{t}}_{nk}\right){\left({\widehat{\mathbf{R}}}_{nk}{\mathit{t}}_{nk}\right)}^T{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}],\hfill \end{array}$$

(39)

where the latent dimensions to be tackled are reduced due to the sparse property of the random sample ${\widehat{\mathit{r}}}_{nk}$. For the multiplication and inversion computations, only the entries of vector or matrix corresponding to ${\widehat{r}}_{nkj}\ne 0$ need to be involved.

To obtain a random sample from $q\left({\mathit{r}}_{nk}\right)$, we update the entries of ${\widehat{\mathit{r}}}_{nk}$ one by one through a single turn of Gibbs sampling, where the sampling probability for ${\widehat{r}}_{nkj}$ has the form in (37) but with the expectation replaced by the current imputation of ${\widehat{\mathit{r}}}_{nk,-j}$.

(v) $q\left(z_n\right)$: To update $q(z_n=k)$, we define the quantity

$$\begin{array}{cc}\hfill \overline{q}(z_n=k)=& exp\{\sum _l\langle {\varphi }_{nl}\rangle \left[\frac{1}{2}\langle log{\sigma }_{kl}\rangle +\frac{v_{kl}}{2}log\frac{v_{kl}}{2}-log\Gamma \left(\frac{v_{kl}}{2}\right)-{\widehat{a}}_{kl}log{\widehat{b}}_{nl}^k+log\Gamma \left({\widehat{a}}_{kl}\right)\right]\hfill \\ \hfill & -\frac{1}{2}\sum _l\langle 1-{\varphi }_{nl}\rangle \langle {\sigma }_{0l}\rangle {\langle u_{nl}\rangle }^0\langle {({\tilde{y}}_{nl}^k-{\mu }_{0l})}^2\rangle -\frac{1}{2}log|\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T|\hfill \\ \hfill & -\frac{1}{2}\mathrm{tr}\left[{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}+{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}\left({\widehat{\mathbf{R}}}_{nk}{\mathit{t}}_{nk}\right){\left({\widehat{\mathbf{R}}}_{nk}{\mathit{t}}_{nk}\right)}^T{\left(\mathbf{I}+{\mathbf{A}}_{nk}\odot {\widehat{\mathit{r}}}_{nk}{\widehat{\mathit{r}}}_{nk}^T\right)}^{-1}\right]\hfill \\ \hfill & +\sum _j\left[\langle r_{nkj}\rangle \left(\langle log{\rho }_{kj}\rangle -log\langle r_{nkj}\rangle \right)+\langle 1-r_{nkj}\rangle \left(\langle log(1-{\rho }_{kj})\rangle -log\langle 1-r_{nkj}\rangle \right)\right]+\langle log{\pi }_k\rangle \}.\hfill \end{array}$$

(40)

Then,

$$q(z_n=k)=\frac{\overline{q}(z_n=k)}{{\sum }_{k^{\prime }}\overline{q}(z_n=k^{\prime })}.$$

(41)

4.4. Auxiliary Posteriors of the Parameters: VB-M Step

The VB-M step updates the posterior $q(\Theta )$ for the parameters following the factorization in (27). Through mathematical manipulation (see the Supplementary Materials for the details), we have

$$\begin{array}{cc}\hfill & q\left(\mathit{\pi }\right)=\mathcal{D}ir\left(\mathit{\pi }\right|\widehat{\mathit{\alpha }}),\hfill \\ \hfill & q\left({\beta }_l\right)=\mathcal{B}eta\left({\beta }_l\right|{\widehat{\kappa }}_{1l},{\widehat{\kappa }}_{2l}),\hfill \\ \hfill & q\left({\rho }_{kj}\right)=\mathcal{B}eta\left({\rho }_{kj}\right|{\widehat{\tau }}_{1kj},{\widehat{\tau }}_{2kj}),\hfill \end{array}$$

(42)

where

$$\begin{array}{cc}\hfill & \widehat{\mathit{\alpha }}={({\widehat{\alpha }}_1,{\widehat{\alpha }}_2,\dots ,{\widehat{\alpha }}_K)}^T,{\widehat{\alpha }}_k={\alpha }_{0k}+\sum _n\langle {\delta }_{z_n,k}\rangle ,\hfill \\ \hfill & {\widehat{\kappa }}_{1l}={\kappa }_1+\sum _n\langle {\varphi }_{nl}\rangle ,{\widehat{\kappa }}_{2l}={\kappa }_2+\sum _n\langle 1-{\varphi }_{nl}\rangle ,\hfill \\ \hfill & {\widehat{\tau }}_{1kj}={\tau }_1+\sum _n\langle {\delta }_{z_n,k}\rangle \langle r_{nkj}\rangle ,{\widehat{\tau }}_{2kj}={\tau }_2+\sum _n\langle {\delta }_{z_n,k}\rangle \langle 1-r_{nkj}\rangle .\hfill \end{array}$$

(43)

The posterior $q\left({\mathit{w}}_{kl}\right)$ is given by

$$q\left({\mathit{w}}_{kl}\right)=\mathcal{N}\left({\mathit{w}}_{kl}\right|{\widehat{\mathit{m}}}_{kl},{\widehat{\mathbf{M}}}_{kl}),$$

(44)

where

$$\begin{array}{cc}\hfill {\widehat{\mathbf{M}}}_{kl}& =m_0\mathbf{I}+\sum _n\langle {\delta }_{z_n,k}\rangle \left(\langle {\sigma }_{kl}\rangle \langle {\varphi }_{nl}\rangle {\langle u_{nl}\rangle }_k^1+\langle {\sigma }_{0l}\rangle \langle 1-{\varphi }_{nl}\rangle {\langle u_{nl}\rangle }^0\right)\langle {\mathit{x}}_{nk}{\mathit{x}}_{nk}^T\odot {\mathit{r}}_{nk}{\mathit{r}}_{nk}^T\rangle ,\hfill \\ \hfill {\widehat{\mathit{m}}}_{kl}& ={{\widehat{\mathbf{M}}}_{kl}}^{-1}\sum _n\langle {\delta }_{z_n,k}\rangle \left[\langle {\sigma }_{kl}\rangle \langle {\varphi }_{nl}\rangle {\langle u_{nl}\rangle }_k^1\left(y_{nl}-\langle {\mu }_{kl}\rangle \right)+\langle {\sigma }_{0l}\rangle \langle 1-{\varphi }_{nl}\rangle {\langle u_{nl}\rangle }^0\left(y_{nl}-\langle {\mu }_{0l}\rangle \right)\right]\langle {\mathbf{R}}_{nk}{\mathit{x}}_{nk}\rangle .\hfill \end{array}$$

(45)

The posteriors $q\left({\mu }_{0l}\right)$ and $q\left({\mu }_{kl}\right)$ are given by

$$q\left({\mu }_{0l}\right)=\mathcal{N}\left({\mu }_{0l}\right|{\widehat{s}}_{0l},{\widehat{\lambda }}_{0l}),q\left({\mu }_{kl}\right)=\mathcal{N}\left({\mu }_{kl}\right|{\widehat{s}}_{kl},{\widehat{\lambda }}_{kl}),$$

(46)

where

$$\begin{array}{cc}\hfill {\widehat{\lambda }}_{0l}& ={\lambda }_0+\langle {\sigma }_{0l}\rangle \sum _n\langle 1-{\varphi }_{nl}\rangle {\langle u_{nl}\rangle }^0,\hfill \\ \hfill {\widehat{s}}_{0l}& ={{\widehat{\lambda }}_{0l}}^{-1}\left({\lambda }_0{s}_{0l}+\langle {\sigma }_{0l}\rangle \sum _{n,k}\langle {\delta }_{z_n,k}\rangle \langle 1-{\varphi }_{nl}\rangle {\langle u_{nl}\rangle }^0\langle {\tilde{y}}_{nl}^k\rangle \right),\hfill \\ \hfill {\widehat{\lambda }}_{kl}& ={\lambda }_0+\langle {\sigma }_{kl}\rangle \sum _n\langle {\delta }_{z_n,k}\rangle \langle {\varphi }_{nl}\rangle {\langle u_{nl}\rangle }_k^1,\hfill \\ \hfill {\widehat{s}}_{kl}& ={{\widehat{\lambda }}_{kl}}^{-1}\left({\lambda }_0{s}_{0l}+\langle {\sigma }_{kl}\rangle \sum _n\langle {\delta }_{z_n,k}\rangle \langle {\varphi }_{nl}\rangle {\langle u_{nl}\rangle }_k^1\langle {\tilde{y}}_{nl}^k\rangle \right).\hfill \end{array}$$

(47)

In addition, the posteriors $q\left({\sigma }_{0l}\right)$ and $q\left({\sigma }_{kl}\right)$ are updated as

$$q\left({\sigma }_{0l}\right)=\mathcal{G}({\sigma }_{0l}|\frac{{\widehat{\eta }}_{0l}}{2},\frac{{\widehat{\xi }}_{0l}}{2}),q\left({\sigma }_{kl}\right)=\mathcal{G}({\sigma }_{kl}|\frac{{\widehat{\eta }}_{kl}}{2},\frac{{\widehat{\xi }}_{kl}}{2}),$$

(48)

where

$$\begin{array}{cc}\hfill {\widehat{\eta }}_{0l}& ={\eta }_0+\sum _n\langle 1-{\varphi }_{nl}\rangle ,\hfill \\ \hfill {\widehat{\xi }}_{0l}& ={\xi }_0+\sum _{n,k}\langle {\delta }_{z_n,k}\rangle \langle 1-{\varphi }_{nl}\rangle {\langle u_{nl}\rangle }^0\langle {({\tilde{y}}_{nl}^k-{\mu }_{0l})}^2\rangle ,\hfill \\ \hfill {\widehat{\eta }}_{kl}& ={\eta }_0+\sum _n\langle {\delta }_{z_n,k}\rangle \langle {\varphi }_{nl}\rangle ,\hfill \\ \hfill {\widehat{\xi }}_{kl}& ={\xi }_0+\sum _n\langle {\delta }_{z_n,k}\rangle \langle {\varphi }_{nl}\rangle {\langle u_{nl}\rangle }_k^1\langle {({\tilde{y}}_{nl}^k-{\mu }_{kl})}^2\rangle .\hfill \end{array}$$

(49)

The degrees of freedom $v_{0l}$ can be obtained by solving the nonlinear equation

$$\sum _n\langle 1-{\varphi }_{nl}\rangle \left[1+log\left(\frac{v_{0l}}{2}\right)-\psi \left(\frac{v_{0l}}{2}\right)+{\langle log{u}_{nl}\rangle }^0-{\langle u_{nl}\rangle }^0\right]=0.$$

(50)

Similarly, $v_{kl}$ can be obtained by solving

$$\sum _n\langle {\delta }_{z_n,k}\rangle \langle {\varphi }_{nl}\rangle \left[1+log\left(\frac{v_{kl}}{2}\right)-\psi \left(\frac{v_{kl}}{2}\right)+{\langle log{u}_{nl}\rangle }_k^1-{\langle u_{nl}\rangle }_k^1\right]=0.$$

(51)

4.5. Algorithm

The developed structured VB algorithm is summarized in Algorithm 1. The optimization process can be monitored via the ELBO (21). The computation of the ELBO is detailed in Appendix A.

Algorithm 1 Proposed Structured VB Algorithm for Robust Clustering and Model Selection

Require:  training data ${\mathit{y}}_n$, $1\le n\le N$, the number of clusters K; Ensure:  the response probabilities, the centroids, the saliency of features, the factor loading matrices, the activity of factors;   1: while the evidence lower bound $\mathcal{L}$ increases more than $ϵ$ and the number of iteration is less than $IterMax$ do   2:     VB-E step   3:     Update $q\left(u_{nl}\right|{\varphi }_{nl},z_n)$ according to (28) for $1\le l\le d$ and $1\le n\le N$;   4:     Update $q\left({\varphi }_{nl}\right)$ according to (33) for $1\le l\le d$ and $1\le n\le N$;   5:     Update $q\left({\mathit{x}}_{nk}\right|{\mathit{r}}_{nk})$ according to (34) for $1\le k\le K$ and $1\le n\le N$;   6:     Update $q\left({\mathit{r}}_{nk}\right)$ according to (36) for $1\le k\le K$ and $1\le n\le N$: run a single turn of Gibbs sampling for a sample from $q\left({\mathit{r}}_{nk}\right)$;   7:     Update $q\left(z_n\right)$ according to (41) for $1\le n\le N$;   8:     VB-M step   9:     Update $q\left(\mathit{\pi }\right)$, $q\left({\beta }_l\right)$ and $q\left({\rho }_{kj}\right)$ according to (42) for $1\le l\le d$, $1\le j\le p_k$ and $1\le k\le K$; 10:     Update $q\left({\mathit{w}}_{kl}\right)$ according to (44) for $1\le l\le d$ and $1\le k\le K$; 11:     Update $q\left({\mu }_{0l}\right)$ and $q\left({\mu }_{kl}\right)$ according to (46) for $1\le l\le d$ and $1\le k\le K$; 12:     Update $q\left({\sigma }_{0l}\right)$ and $q\left({\sigma }_{kl}\right)$ according to (48) for $1\le l\le d$ and $1\le k\le K$; 13:     Update $v_{0l}$ and $v_{kl}$ according to (50) and (51) for $1\le l\le d$ and $1\le k\le K$; 14: end while

We apply K-mean clustering for initialization of the VB algorithm and initialize a large p$(p<min(d,N\left)\right)$ for the latent dimensions of the K local factor models. At each iteration, we randomize the updating order of ${\widehat{r}}_{nkj}$’s in the Gibbs sampling step to avoid co-adaptation. To further accelerate the algorithm, we make the number of factors adaptive. The empirical estimator of factor activity, i.e.,

$${\widehat{\rho }}_{kj}=\frac{{\sum }_n\langle {\delta }_{z_n,k}\rangle \langle r_{nkj}\rangle }{{\sum }_n\langle {\delta }_{z_n,k}\rangle },$$

(52)

is computed at the end of each iteration. If ${\widehat{\rho }}_{kj}=0$, then we remove the jth latent factor from the kth local factor model. The pruning is carried out after a burn-in period of the algorithm.

4.6. Interpreting the Model

The expectation of feature saliency ${\beta }_l$ can be used to show the informative degree of features after being adjusted by latent factors, which is given by

$$\langle {\beta }_l\rangle =\frac{{\widehat{\kappa }}_{1l}}{{\widehat{\kappa }}_{1l}+{\widehat{\kappa }}_{2l}}.$$

(53)

In addition, the expectation of factor activity ${\rho }_{kj}$ can be applied to evaluate the explanatory power of latent factors in each class, which can be obtained as

$$\langle {\rho }_{kj}\rangle =\frac{{\widehat{\tau }}_{1kj}}{{\widehat{\tau }}_{1kj}+{\widehat{\tau }}_{2kj}}.$$

(54)

We also consider the reconstruction performance of the proposed algorithm. The centroid of each class is estimated by

$$\langle {\tilde{\mathit{\mu }}}_k\rangle =\langle \mathbf{B}\rangle \langle {\mathit{\mu }}_k\rangle +\langle \mathbf{I}-\mathbf{B}\rangle \langle {\mathit{\mu }}_0\rangle ,$$

(55)

where $\langle \mathbf{B}\rangle =\mathrm{diag}(\langle {\beta }_1\rangle ,\langle {\beta }_2\rangle ,\dots ,\langle {\beta }_d\rangle )$, $\langle {\mathit{\mu }}_k\rangle ={({\widehat{s}}_{k1},{\widehat{s}}_{k2},\dots ,{\widehat{s}}_{kd})}^T$ and $\langle {\mathit{\mu }}_0\rangle ={({\widehat{s}}_{01},{\widehat{s}}_{02},\dots ,{\widehat{s}}_{0d})}^T$. Then, reconstruction for the nth individual in class k can be computed as

$${\widehat{\mathit{y}}}_n=\langle {\mathbf{W}}_k\rangle \langle {\mathbf{R}}_{nk}{\mathit{x}}_{nk}\rangle +\langle {\tilde{\mathit{\mu }}}_k\rangle ,$$

(56)

where $\langle {\mathbf{W}}_k\rangle ={[{\widehat{\mathit{m}}}_{k1},{\widehat{\mathit{m}}}_{k2},\dots ,{\widehat{\mathit{m}}}_{kd}]}^T$ and $\langle {\mathbf{R}}_{nk}{\mathit{x}}_{nk}\rangle$ is obtained from the VB-E step after the algorithm converges.

5. Experiment Study

5.1. Experiments on Synthetic Data

In this section, we justify the developed model and the structured VB algorithm using controlled experiments. We continue the experiments in [14] with the same synthetic data where the features were generated independently in each component. An additional set of data was generated where we imposed correlation between features within the mixture component. The proposed model and algorithm was compared with the semi-Bayesian clustering model and algorithm in [10], called varFnMS, in which a finite mixture of Gaussian is adopted and a mean-field VB is applied and compared with the full-Bayesian model and algorithm in [14], denoted varFnMS-T, which is based on the mixture of Student’s t distribution and uses the structured VB algorithm.

The synthetic data in [14] contain 800 data points from four well-separated classes. The data are 10-dimensional with two influential features located around the class centers $(0,3)$, $(1,9)$, $(6,4)$ and $(7,10)$ with identity covariance matrices in each class. The remaining eight “noisy” features were sampled from $\mathcal{N}(0,1)$. We made randomly $1\%$ of the data outliers by adding noises sampled uniformly from ${[-10,10]}^{10}$. The features are mutually independent in each class, which is consistent with the assumption underlying varFnMS and varFnMS-T. In the additional set of data, the local independence assumption is violated. We assigned a four-factor model for class 1, a two-factor model for class 2, a one-factor model for class 3 and no factor in class 4. The mean vector of each factor model remained the same as that in the “locally independent” data. The factor loading matrices were generated randomly with each entry from $\mathcal{N}(0,1)$. The noise term in each class was generated from $\mathcal{N}(\mathbf{0},{\mathbf{I}}_{10})$.

The proposed algorithm, denoted as varFnMS-TFA, the varFnMS and the varFnMS-T were carried out twenty times, separately. The number of clusters K was set as four. The K-mean clustering algorithm was used to initialize the posterior $q\left(z_n\right)$. The feature saliency and factor activity were both initialized as $0.5$. The hyperparameters ${\alpha }_{0k},{\kappa }_1,{\kappa }_2,{\tau }_1,{\tau }_2,{\lambda }_0,m_0$, ${\eta }_0$ and ${\xi }_0$ were set to be ${10}^{-5}$ and ${\mathit{s}}_0$ was set as the empirical mean of the feature data. We assumed a nine-factor model for each class at the beginning and initialized the posterior means of the latent factors by sampling from $\mathcal{N}(\mathbf{0},{\mathbf{I}}_9)$. The algorithm terminates when the difference of the ELBO between two consecutive iterations is less than ${10}^{-7}$ or the maximum number of iterations ($IterMax=500$) is reached. To avoid the “label switching” problems, we labeled the obtained clusters from the twenty repeated experiments by matching with the true classification of the data.

The ELBO reached and the classification error rate comparing the clustering with the original grouping of the data via the three algorithms with the two synthetic datasets are presented in

Table 1. Evidence lower bound (ELBO) and classification error obtained via varFnMS, varFnMS-T and varFnMS-TFA for the two synthetic datasets where the features are locally independent and correlated, separately.

	Independent		Correlated
Algorithm	ELBO	Error	ELBO	Error
varFnMS	−13,320.803	0.076	−16,868.400	0.351
	(136.822)	(0.118)	(113.338)	(0.071)
varFnMS-T	−13,140.178	0.073	−16,468.433	0.302
	(763.027)	(0.111)	(135.973)	(0.083)
varFnMS-TFA	−9082.876	0.071	−8854.751	0.051
	(3320.315)	(0.109)	(2696.577)	(0.053)

. For the dataset where features are mutually independent within class, the classification accuracy of varFnMS-TFA is slightly higher than the other two algorithms, but the difference is not evident. For the dataset generated with correlated features, the proposed algorithm shows significantly higher accuracy than the other two algorithms under the local independence assumption. Moreover, the ELBO reached via varFnMS-TFA is the highest on average in both datasets and the discrepancy is enlarged where the features are locally correlated. As seen in Figure 2, it successfully captures the correlation across features through the latent factors.

The estimated factor activity in each class by the proposed algorithm (averaged over the twenty repeats) for the two synthetic datasets is presented in Figure 2. Generally, the algorithm recovers the ground truth in both datasets. It can be seen in subplot (a) that there is no significantly active factor across the four classes for the “locally independent” data and the true pattern of factor activity in the “locally correlated” data is recovered as shown in subplot (b).

Figure 3 compares the estimated feature saliency for the two synthetic datasets. As shown in subplot (a), the three algorithms make the same good estimation on the feature saliency when the features are independent within class. But when the dependence relationship is imposed, the proposed algorithm shows apparently different behavior from the other two algorithms as shown in subplot (b). While varFnMS and varFnMS-T estimate the salience of a feature that could be confounded by the other features, varFnMS-TFA gives the factor-adjusted feature saliency, where the confounding effects are resolved by the latent factors.

In

Table 2. The centroids estimated via varFnMS, varFnMS-T and varFnMS-TFA for the two synthetic datasets where the features are locally independent and correlated, separately.

	Independent			Correlated
Class 1	varFnMS	varFnMS-T	varFnMS-TFA	varFnMS	varFnMS-T	varFnMS-TFA
${\mu }_{11}=0$	0.405 (0.791)	0.399 (0.910)	0.392 (0.906)	0.739 (1.056)	0.064 (0.569)	0.019 (0.140)
${\mu }_{12}=3$	3.289 (0.664)	3.178 (0.471)	3.223 (0.670)	3.922 (1.405)	1.771 (1.270)	3.093 (0.201)
${\mu }_{13}=0$	0.000 (0.037)	−0.107 (0.497)	−0.116 (0.490)	−0.314 (0.760)	−1.730 (0.952)	−0.001 (0.121)
${\mu }_{14}=0$	−0.007 (0.049)	0.041 (0.178)	−0.027 (0.096)	−0.009 (0.776)	−1.053 (0.814)	−0.025 (0.078)
${\mu }_{15}=0$	0.002 (0.043)	−0.025 (0.098)	0.000 (0.044)	−0.271 (0.915)	−1.748 (0.869)	−0.020 (0.041)
${\mu }_{16}=0$	0.005 (0.044)	−0.086 (0.393)	−0.031 (0.121)	0.011 (0.210)	−0.390 (0.428)	0.000 (0.000)
${\mu }_{17}=0$	−0.016 (0.043)	−0.073 (0.292)	−0.017 (0.046)	−0.185 (0.543)	0.060 (0.537)	−0.014 (0.046)
${\mu }_{18}=0$	−0.007 (0.039)	−0.051 (0.194)	−0.032 (0.119)	−0.004 (0.448)	0.340 (0.670)	−0.052 (0.180)
${\mu }_{19}=0$	0.034 (0.050)	0.075 (0.228)	0.015 (0.032)	0.019 (0.866)	1.410 (0.979)	−0.012 (0.028)
${\mu }_{1,10}=0$	−0.021 (0.040)	−0.168 (0.662)	−0.034 (0.125)	0.026 (0.422)	0.262 (0.237)	−0.004 (0.030)
Class 2	varFnMS	varFnMS-T	varFnMS-TFA	varFnMS	varFnMS-T	varFnMS-TFA
${\mu }_{21}=1$	1.332 (0.650)	1.096 (0.447)	1.058 (0.282)	2.000 (0.939)	1.129 (0.678)	1.049 (0.151)
${\mu }_{22}=9$	8.798 (0.802)	8.875 (0.478)	8.793 (0.862)	6.712 (1.229)	6.762 (0.863)	9.012 (0.101)
${\mu }_{23}=0$	0.100 (0.472)	0.136 (0.622)	0.040 (0.209)	−0.024 (0.984)	0.340 (0.315)	0.008 (0.066)
${\mu }_{24}=0$	0.002 (0.057)	0.015 (0.047)	0.008 (0.035)	0.071 (0.738)	0.487 (0.792)	0.042 (0.091)
${\mu }_{25}=0$	−0.207 (0.926)	−0.228 (1.006)	−0.192 (0.865)	−0.028 (0.647)	0.394 (0.321)	−0.011 (0.028)
${\mu }_{26}=0$	−0.049 (0.138)	−0.049 (0.167)	−0.012 (0.037)	−0.107 (0.566)	0.124 (0.305)	0.004 (0.019)
${\mu }_{27}=0$	−0.034 (0.200)	−0.053 (0.231)	−0.001 (0.027)	−0.293 (0.578)	−0.468 (0.615)	−0.007 (0.031)
${\mu }_{28}=0$	−0.083 (0.367)	−0.094 (0.423)	−0.093 (0.417)	−0.052 (0.684)	−0.213 (0.314)	0.008 (0.067)
${\mu }_{29}=0$	0.036 (0.189)	0.039 (0.202)	0.004 (0.033)	−0.340 (0.611)	−0.702 (0.637)	−0.009 (0.027)
${\mu }_{2,10}=0$	0.009 (0.129)	0.018 (0.193)	0.040 (0.245)	−0.099 (0.550)	−0.147 (0.160)	0.002 (0.032)
Class 3
${\mu }_{31}=6$	5.684 (1.136)	5.691 (0.998)	5.758 (0.767)	4.773 (1.507)	5.313 (1.603)	6.013 (0.158)
${\mu }_{32}=4$	4.392 (0.843)	4.514 (1.246)	4.553 (1.464)	3.962 (0.782)	3.644 (0.726)	4.174 (0.656)
${\mu }_{33}=0$	−0.082 (0.504)	0.056 (0.402)	0.064 (0.215)	0.206 (0.502)	0.127 (0.454)	0.018 (0.049)
${\mu }_{34}=0$	0.114 (0.570)	−0.015 (0.722)	−0.063 (0.631)	0.053 (0.466)	−0.204 (0.516)	0.006 (0.037)
${\mu }_{35}=0$	−0.025 (0.289)	−0.015 (0.330)	0.062 (0.305)	0.197 (0.517)	0.131 (0.428)	−0.011 (0.019)
${\mu }_{36}=0$	−0.181 (0.557)	−0.032 (0.632)	−0.087 (0.854)	0.076 (0.390)	0.064 (0.196)	0.002 (0.008)
${\mu }_{37}=0$	−0.137 (0.646)	−0.176 (0.761)	−0.213 (0.649)	0.269 (0.495)	0.703 (0.619)	−0.005 (0.051)
${\mu }_{38}=0$	−0.241 (0.744)	−0.332 (0.946)	−0.141 (0.465)	−0.170 (0.599)	0.216 (0.427)	0.010 (0.034)
${\mu }_{39}=0$	0.030 (0.270)	−0.030 (0.196)	−0.017 (0.158)	−0.023 (0.711)	0.468 (0.629)	−0.012 (0.033)
${\mu }_{3,10}=0$	−0.309 (0.771)	−0.304 (0.767)	−0.142 (0.416)	−0.017 (0.512)	0.068 (0.156)	−0.001 (0.030)
Class 4
${\mu }_{41}=7$	6.792 (0.752)	6.966 (0.130)	6.960 (0.123)	5.443 (1.236)	6.329 (0.899)	6.709 (1.289)
${\mu }_{42}=10$	9.857 (0.521)	9.830 (0.689)	9.831 (0.688)	9.039 (1.183)	9.649 (0.623)	9.741 (0.995)
${\mu }_{43}=0$	0.002 (0.051)	−0.007 (0.060)	−0.005 (0.045)	0.075 (0.100)	0.034 (0.079)	−0.006 (0.046)
${\mu }_{44}=0$	0.000 (0.053)	0.005 (0.036)	0.000 (0.021)	−0.072 (0.111)	−0.061 (0.129)	0.380 (1.725)
${\mu }_{45}=0$	0.008 (0.043)	0.009 (0.041)	0.002 (0.032)	0.043 (0.175)	−0.013 (0.096)	−0.015 (0.027)
${\mu }_{46}=0$	−0.003 (0.041)	−0.003 (0.037)	−0.006 (0.025)	−0.017 (0.095)	−0.044 (0.084)	0.002 (0.008)
${\mu }_{47}=0$	−0.025 (0.063)	−0.011 (0.058)	0.002 (0.026)	0.111 (0.117)	0.079 (0.098)	−0.097 (0.425)
${\mu }_{48}=0$	−0.003 (0.051)	−0.005 (0.045)	−0.004 (0.031)	−0.053 (0.106)	−0.026 (0.077)	0.081 (0.434)
${\mu }_{49}=0$	0.029 (0.055)	0.021 (0.051)	0.016 (0.032)	0.071 (0.118)	0.050 (0.106)	−0.011 (0.035)
${\mu }_{4,10}=0$	−0.026 (0.048)	−0.027 (0.049)	−0.011 (0.036)	−0.001 (0.059)	0.015 (0.053)	−0.001 (0.031)

, the estimated class centroids in the two synthetic datasets (averaged over the twenty repeats) are present. When features are generated independently within class, the three algorithms exhibit comparable performance and recover the real centroids approximately. But with correlations imposed, the estimation accuracy by varFnMS or varFnMS-T is apparently degraded. In class 1, 2 and 3, they misestimate the means of the first two features that are salient and the variations of estimation are significantly enlarged compared with the results of varFnMS-TFA. In comparison, the varFnMS-TFA algorithm that considers the local dependence relationships gives more accurate and stable estimation results.

5.2. Experiments on Real Datasets

In this section, we apply the proposed model on the benchmark datasets: Iris, Olive, Wine and WDBC. The Iris dataset is obtained from the R package “datasets”. Olive is the Italian olive oil dataset and Wine the Italian wine dataset. They are both obtained from the R package “pgmm”. WDBC is the Wisconsin diagnostic breast cancer dataset downloaded from the UCI machine learning repository (https://doi.org/10.24432/C5DW2B; accessed on 26 May 2023). For each dataset, we repeated each algorithm ten times and retrieved the result with the highest value on ELBO. We set the initial dimensions of latent factors for each dataset as $d-1$, where d is the number of features in the data.

Table 3. Classification error obtained via varFnMS, varFnMS-T and varFnMFAS-T for the four benchmark datasets.

Dataset	N	d	K	varFnMS	varFnMS-T	varFnMS-TFA
Iris	150	4	3	0.093	0.093	0.020
Olive	572	8	3	0.203	0.199	0.042
Wine	178	27	3	0.079	0.062	0.073
WDBC	569	30	2	0.095	0.095	0.088

presents the basic information for the four datasets and the classification error obtained. There is a significant decrease on the classification error for the Olive data and a slight improvement on the results for Iris and WDBC when using the proposed algorithm. The exception goes to the Wine data, where the proposed algorithm gives results slightly inferior to varFnMS-T.

Figure 4 and Figure 5 show the factor activity and feature saliency for the four benchmark datasets. It can be seen from Figure 4 that strong factor activity is detected in Iris, Olive and WDBC data. As shown in Figure 5, the patterns of estimated feature saliency are noticeably changed when applying the proposed algorithm. The combined results indicate that the correlation between features could interfere with our decision about the features’ relevance and the classification of data.

5.3. Application on Handwritten Object Recognition

In this section, we apply the developed algorithm to the machine learning task of handwritten alphabet recognition. The handwritten alphabet dataset is obtained from the Kaggle webpage (https://www.kaggle.com/datasets/sachinpatel21/az-handwritten-alphabets-in-csv-format/data; accessed on 26 May 2023). It contains more than 370,000 images for the English alphabets (A–Z). The images are gray-scale in the size of $28\times 28$ pixels. We focus on separation of the handwritten alphabets A, B and C and reserve randomly 200 images for each of the alphabets. As the variability of some pixels in the image of an alphabet is exactly zero, we may encounter the singularity problem during iterations of the clustering algorithms. Therefore, pre-processing was implemented on the data as detailed in Appendix B. In the proposed algorithm, the initial number of latent factors was set as fifty for each class. The three algorithms attain the same classification error rate as 0.16. The patterns of feature saliency estimated via varFnMS, varFnMS-T and the proposed algorithm are compared in Figure 6. The pixels are arranged along the x-axis column by column in the $28\times 28$-pixel image. The saliences for the margin of the image with almost zero variability have been set as zero in the pre-processing stage. As can be seen from Figure 6, for the potentially discriminant part of the image, while the other two algorithms may have ambiguity concerning deciding the relevance of features, the evaluation based on the proposed algorithm is clearer which could be an improvement by extracting the confounding effects through latent factors.

The centroids estimated via the three algorithms are shown in Figure 7, where the reconstruction process of images via the proposed algorithm is also illustrated. The estimated centroid in each class can be calculated following (55), which is a mixing of the class-specific mean and the background. The calculation is outlined by the red box, where the centroid, the class-specific mean and the background are present from left to right, successively. It can be seen that all three algorithms exhibit good performance with respect to characterizing the alphabets. The estimated centroids can sketch the general appearance of the alphabets. But it is noticeable that varFnMS and varFnMS-T make some mistakes on estimation of the background. There should have been no handwritten stroke at the bottom of the background image, since this distinguishes the images of alphabet A. The proposed algorithm performs well with respect to reconstructing the images. An example of reconstruction is present in subplot (c). Additional examples are present in Appendix C. Generally, by adding the influence of latent factors, handwriting on the images becomes legible. The factor loadings on the two most active factors for each alphabet are shown at the right side of subplot (c). As can be seen, the information from latent factors plays an important role in refining the images.

6. Conclusions

In this paper, we developed a hierarchical latent variable model for robust clustering and model selection. We considered the cases where features are correlated within mixture components in a Student’s t mixture model. Factor-adjusted feature saliency was proposed to evaluate the relevance of features to data separation. Automatic latent dimension reduction was achieved by introducing the variables of factor activity. A full Bayesian treatment was adopted and a structured VB inference framework was developed that have enabled a tighter bond to the marginal likelihood and improved the inference accuracy. Controlled experiments on synthetic and real-world datasets showed that the proposed model is able to capture the correlation between features and shows better clustering performance than the models relying on the local independence assumption. Application of the developed algorithm on the high-dimensional handwritten alphabet data showed its applicability and usefulness for image recognition and reconstruction.

In the proposed model, we take the number of clusters (number of components in the mixture model) as fixed and given before inference. An ongoing work is to extend our model to realize automatic selection of the number of clusters. We imposed the Dirichlet prior on the mixing probabilities, which can act as a penalization to drive the mixing probabilities associated with unnecessary components towards extinction. We will also investigate the novel penalization methods proposed in [34] which result in continuous objective functions and can shrink the mixing weights to exactly zero. Other limitations include assuming that features are approximated Gaussian distributed in each component. This assumption can be violated when the features only take positive values or follow skewed distributions. Future work may consider extending the model to tackle these scenarios.