Clustering Mixed-Type Data via Dirichlet Process Mixture Model with Cluster-Specific Covariance Matrices

Abstract

Many studies have shown successful applications of the Dirichlet process mixture model (DPMM) for clustering continuous data. Beyond continuous data, in practice, one can expect to see different data types, including ordinal and nominal data. Existing DPMMs for clustering mixed-type data assume a strict covariance matrix structure, resulting in an overfit model. This article explores a DPMM for mixed-type data that allows the covariance matrix to differ from one cluster to another. We assume an underlying latent variable framework for ordinal and nominal data, which is then modeled jointly with the continuous data. The identifiability issue on the covariance matrix poses computational challenges, thus requiring a nonstandard inferential algorithm. The applicability and flexibility of the proposed model are illustrated through simulation examples and real data applications.

1. Introduction

Clustering analysis aims to partition a dataset into disjoint groups. Various clustering algorithms have been proposed, yet most of them are designed to work with only a single type of data at a time. For instance, in the context of model-based clustering, the multivariate normal distribution is usually employed as the component distribution of a mixture model for clustering continuous data [1,2,3]. However, datasets containing different types of variables frequently arise in practice. In the current literature, studies of the model-based clustering of mixed-type data are less explored [4].

The absence of a standard distribution to model the different data types simultaneously makes clustering mixed-type data challenging. A straightforward approach in clustering mixed-type data in the context of model-based clustering is by assuming local independence between the different types of data, in which the component of the mixture model is made of the product of the individual distributions of each data type, that is, normal distribution for continuous data and multinomial distribution for nominal data [5]. Nevertheless, this approach usually overestimates the number of clusters [6]. The location mixture model [7,8,9] is an alternative approach when considering the dependencies between different data types. These dependencies are captured by conditioning the normal distribution for continuous data on multinomial distribution for the nominal data. More recently, the location mixture model is revisited in [10] to accommodate missing data.

Another approach to dealing with mixed-type data is to use latent variables to underlie the non-continuous data [11]. In this approach, each ordinal variable is assumed to arise from thresholding a latent normal variable. Still, this approach is only feasible for up to two ordinal variables due to the computational complexity. The same approach has been used in [12] to cluster mixed continuous and binary data. A more recent attempt at leveraging the latent variables approach in clustering mixed-type data is reported in [13]. Their model can handle three data types: continuous, ordinal, and nominal, but allows a limited covariance structure.

Within the Bayesian literature, Murray and Reiter [14] integrate normal mixtures with multinomial mixtures through a prior tensor factorization in an attempt to capture the dependence between continuous and nominal data. Then, DeYoreo et al. [15] extend this approach by focusing on only certain variables to improve model fit. The use of latent variables in the Dirichlet process mixture model (DPMM) for clustering continuous and ordinal data has been proposed in [16]. In a similar line of research, the latent variable approach is also adopted in [17], but with the addition of a nominal data type. However, the model in [17] imposes a common covariance assumption with a complex prior update.

This article presents a model for the joint clustering of continuous, ordinal, and nominal data using DPMM. Our model utilizes a similar latent variable approach as in [17] to handle ordinal and nominal data, yet built upon different model construction. The significant differences in our proposed formulation are with regard to the Dirichlet process (DP) representation and the prior specification on the covariance matrix. Compared to [17], our prior specification is more straightforward to implement, with fewer prior hyperparameters to be tuned. In addition, our model offers a more flexible assumption, allowing for a cluster-specific covariance structure. The rest of this article is organized as follows. In Section 2, we briefly review the DPMM with multivariate normal components that outline the foundation of our model. The formulation of our proposed model, along with the algorithm involved, is introduced in Section 3. Then, the simulation examples and real data applications in Section 4 will illustrate how the covariance assumption affects the clustering performance. Finally, the article concludes with Section 5.

2. Dirichlet Process Multivariate Normal Mixture Model

Let $\mathbf{X}={({\mathbf{x}}_1,\dots ,{\mathbf{x}}_N)}^{\prime }$ be a continuous dataset of dimension $N\times Q$, where N denotes the sample size and Q denotes the number of variables. We assume that $\mathbf{X}$ arises from a Dirichlet process mixture model (DPMM) with multivariate normal components indexed by the mean $\mathsf{\mu }$ and covariance $\Sigma$ such that

$$\begin{array}{c}\hfill p\left({\mathbf{x}}_i\right)=\int {\mathcal{N}}_Q({\mathbf{x}}_i|\mathsf{\mu },\Sigma )dG(\mathsf{\mu },\Sigma ),i=1,\dots ,N,\end{array}$$

(1)

where G is a mixing measure. Then, G is assumed to follow the Dirichlet process (DP) with a concentration parameter $\alpha$ and base distribution $G_0$, or, in short, $G\sim \mathrm{DP}(\alpha ,G_0)$. The DP draws distributions around the $G_0$, which is usually set as normal-inverse Wishart distribution, a natural conjugate prior of the data likelihood. Since samples drawn from DP are almost surely discrete, this property causes a tie among the observed component parameter $(\mathsf{\mu },\Sigma )$s. Hence, the data instances that are indexed by the same $(\mathsf{\mu },\Sigma )$ are considered to belong to one cluster. Note that the parameter $\alpha$ controls the number of clusters produced. If $\alpha \to 0$, the DPMM reduces to a single component mixture model where the samples are i.i.d. from a fully Bayesian parametric model with ${\mathcal{N}}_Q({\mathbf{x}}_i|\mathsf{\mu },\Sigma )$ and $(\mathsf{\mu },\Sigma )\sim G_0$. On the contrary, for $\alpha \to \infty$, the model forms singleton clusters of each and every data instance, and we have $p\left({\mathbf{x}}_i\right)=\int {\mathcal{N}}_Q({\mathbf{x}}_i|\mathsf{\mu },\Sigma )d{G}_0(\mathsf{\mu },\Sigma )$. From this, it follows that the DPMM can be presented equivalently in hierarchical form as

$$\begin{array}{c}\hfill {\mathbf{x}}_i|\mathsf{\mu },\Sigma \sim p({\mathbf{x}}_i|{\mathsf{\mu }}_i,{\Sigma }_i),i=1,\dots ,N,\hfill \\ \hfill ({\mathsf{\mu }}_i,{\Sigma }_i)|G\sim G,i=1,\dots ,N,\hfill \\ \hfill G\sim \mathrm{DP}(\alpha ,G_0),\hfill \end{array}$$

(2)

and $G_0\sim \mathcal{N}I{W}_Q({\mathbf{b}}_{\mathbf{0}},{\kappa }_0,{\mathbf{S}}_0,v_0)$. Here, the $\mathcal{N}IW$ stands for the normal-inverse Wishart distribution specified by four parameters: ${\mathbf{b}}_{\mathbf{0}},{\kappa }_0,{\mathbf{S}}_0$, and $v_0$. The hierarchical structure in (2) highlights the grouping induced by the DPMM due to ties of the $(\mathsf{\mu },\Sigma )$s.

We use the stick-breaking construction of the Sethuraman [18] for the DP. This construction does not rely on being able to integrate out the infinite-dimensional G, and thus allowing the direct specification of the prior. Then, the model in (2) can be represented in stick-breaking construction as

$$\begin{array}{cc}\hfill {\mathbf{x}}_i|\mathsf{\mu },\Sigma ,\mathbf{c}& \sim {\mathcal{N}}_Q({\mathsf{\mu }}_{c_i},{\Sigma }_{c_i}),i=1,\dots ,N,\hfill \\ \hfill c_i|\mathbf{W}& \sim \sum _{k=1}^{\infty }{w}_k{\delta }_k(·),i=1,\dots ,N,\hfill \\ \hfill ({\mathsf{\mu }}_k,{\Sigma }_k)& \sim \mathcal{N}I{W}_Q({\mathbf{b}}_{\mathbf{0}},{\kappa }_0,{\mathbf{S}}_0,v_0),k=1,\dots ,\infty ,\hfill \end{array}$$

(3)

where ${\delta }_k$ denotes a Dirac measure with a point mass at k. The vector of a cluster assignment variable $\mathit{c}={(c_1,\dots ,c_N)}^{\prime }$ is introduced to index each ${\mathbf{x}}_i$ to $(\mathsf{\mu },\Sigma )$ such that $p({\mathbf{x}}_i|c_i=k)={\mathcal{N}}_Q({\mathbf{x}}_i|{\mathsf{\mu }}_k,{\Sigma }_k)$. Following [18], the weights $w_k$ are defined such that $w_1={\pi }_1$ and $w_k={\pi }_k{\prod }_{\ell =1}^{k-1}(1-{\pi }_{\ell })$ for $k=2,3,\dots$ with ${\pi }_k\sim Beta(1,\alpha )$. The representation in (3) shows that the DPMM can be viewed as an infinite extension of the standard multivariate normal mixture model. Even though the mixture model has infinitely many clusters, it should be noted that for any sample of size N, only N clusters are realized at most. Therefore, we do not need to specify the number of clusters K as it will be part of the outputs of the inferential algorithm. Compared to other clustering algorithms that require the K to be set in advance [19], the DPMM offers an advantage for cases where the K is not well defined.

3. Methodology

3.1. Model Formulation

Let $\mathbf{X}$ be the input dataset that contains three different types of data: continuous, ordinal, and nominal. We divide the data instance into three parts: ${\mathbf{x}}_i={({\mathbf{x}}_i^{\mathrm{c}\prime },{\mathbf{x}}_i^{\mathrm{o}\prime },{\mathbf{x}}_i^{\mathrm{n}\prime })}^{\prime }$ for $i=1,\dots ,N$, where each part corresponds to one data type.

Suppose that, for the continuous part, there are $M_{\mathrm{c}}$ continuous variable such that ${\mathbf{x}}_i^{\mathrm{c}}={(x_{i1}^{\mathrm{c}},\dots ,x_{i{M}_{\mathrm{c}}}^{\mathrm{c}})}^{\prime }$. We employ the following standard assumption: ${\mathbf{x}}_i^{\mathrm{c}}\sim {\mathcal{N}}_{M_{\mathrm{c}}}({\mathsf{\mu }}_k^{\mathrm{c}},{\Sigma }_k^{\mathrm{c}})$.

For ordinal data, we adopt the multivariate probit regression framework [20,21] that assumes each ordinal variable to be derived from an underlying normal latent variable. Let $j=1,\dots ,M_{\mathrm{o}}$ index the ordinal variable such that ${\mathbf{x}}_i^{\mathrm{o}}={(x_{i1}^{\mathrm{o}},\dots ,x_{i{M}_{\mathrm{o}}}^{\mathrm{o}})}^{\prime }$, each having $L_j^{\mathrm{o}}$ levels with $x_{ij}^{\mathrm{o}}\in \{0,\dots ,L_j^{\mathrm{o}}-1\}$. Then, we introduce latent variables ${\mathbf{z}}_i^{\mathrm{o}}={(z_{i1}^{\mathrm{o}},\dots ,z_{i{M}_{\mathrm{o}}}^{\mathrm{o}})}^{\prime }$ for $i=1,\dots ,N$ such that

$$\begin{array}{c}\hfill x_{ij}^{\mathrm{o}}=\ell \mathrm{iff}{\gamma }_{j,\ell }<z_{ij}^{\mathrm{o}}\le {\gamma }_{j,\ell +1},\end{array}$$

(4)

where ${\mathbf{z}}_i^{\mathrm{o}}\sim {\mathcal{N}}_{M_{\mathrm{o}}}({\mathsf{\mu }}_k^{\mathrm{o}},{\Sigma }_k^{\mathrm{o}})$ and ${\gamma }_{j,\ell }$s are the set of cut-offs for $j=1,\dots ,M_{\mathrm{o}}$ and $\ell =0,\dots ,L_j^{\mathrm{o}}-1$. If we let the cut-offs as unknown parameters, the posterior inference becomes challenging due to the extra steps required to update the ${\gamma }_{j,\ell }$s at each iteration. To avoid these complex updating steps, the ${\gamma }_{j,\ell }$s can be treated as constants, and that the DPMM is robust with respect to the choice of value for the ${\gamma }_{j,\ell }$s [22]. Hence, the cut-offs are set such that ${\gamma }_{j,\ell }={\Phi }^{-1}(\ell /L_j^{\mathrm{o}})$ for $\ell =0,\dots ,L_j^{\mathrm{o}}$ where ${\Phi }^{-1}(·)$ is the inverse of the standard univariate normal distribution function for computational practicality. Figure 1 shows the histogram of the generated underlying latent data of ordinal variables with four levels. From the figure, we can see that ordinal data are essentially the discretization of continuous data. Here, plots (a) and (b) illustrate two ordinal data of different proportions of instances between levels. Even for the same sets of cutoffs, the different levels of proportions will invoke different locations and shapes of the underlying data. These differences are the ones that contribute to cluster discrimination. Another issue that requires attention is the identifiability of the component parameters. If any of the $x_{ij}^{\mathrm{o}}$s is binary, that is, when $L_j^{\mathrm{o}}=2$, the covariance matrix is not fully identifiable by the data likelihood. To address this, we need to restrict the diagonal elements of the covariance matrix that corresponds to the binary $x_{ij}^{\mathrm{o}}$ to one, as demonstrated in [23].

To model nominal data, the multinomial probit regression framework [24,25] is used. Let $j=1,\dots ,M_{\mathrm{n}}$ index the nominal variable such that ${\mathbf{x}}_i^{\mathrm{n}}={(x_{i1}^{\mathrm{n}},\dots ,x_{i{M}_{\mathrm{n}}}^{\mathrm{n}})}^{\prime }$, each having $L_j^{\mathrm{n}}$ categories with $x_{ij}^{\mathrm{n}}\in \{0,\dots ,L_j^{\mathrm{n}}-1\}$. For each nominal instance $x_{ij}^{\mathrm{n}}$, $i=1,\dots ,N$, $j=1,\dots ,M_{\mathrm{n}}$, we introduce a $(L_j^{\mathrm{n}}-1)$-dimensional continuous latent vector ${\mathbf{z}}_{ij}^{\mathrm{n}}={(z_{ij1}^{\mathrm{n}},\dots ,z_{ij(L_j^{\mathrm{n}}-1)}^{\mathrm{n}})}^{\prime }$, defined as follows:

$$\begin{array}{c}\hfill x_{ij}^{\mathrm{n}}=\left\{\begin{array}{cc}\hfill 0\hfill & \mathrm{iff}max\{z_{ij\ell }^{\mathrm{n}},\ell =1,\dots ,L_j^{\mathrm{n}}-1\}<0\hfill \\ \hfill \ell \hfill & \mathrm{iff}max\{z_{ij\ell }^{\mathrm{n}},\ell =1,\dots ,L_j^{\mathrm{n}}-1\}=z_{ij\ell }^{\mathrm{n}}>0,\hfill \end{array}\right.\end{array}$$

(5)

where $x_{ij}^{\mathrm{n}}=0$ is the most frequent category. Then, we concatenate the latent vectors ${\mathbf{z}}_{ij}^{\mathrm{n}\prime }$ for $j=1,\dots ,M_{\mathrm{n}}$, forming a ${\sum }_{j=1}^{M_{\mathrm{n}}}(L_j^{\mathrm{n}}-1)$-dimensional vector ${\mathbf{z}}_i^{\mathrm{n}}={({\mathbf{z}}_{i1}^{\mathrm{n}\prime },\dots ,{\mathbf{z}}_{i{M}_{\mathrm{n}}}^{\mathrm{n}\prime })}^{\prime }\sim {\mathcal{N}}_{{\sum }_{j=1}^{M_{\mathrm{n}}}(L_j^{\mathrm{n}}-1)}({\mathsf{\mu }}_k^{\mathrm{n}},{\Sigma }_k^{\mathrm{n}})$. The scatter plot in Figure 2 depicts the generated underlying latent data of nominal variables with three categories. Here, plots (a) and (b) compare two nominal variables with different proportions of instances between categories. As we can see from the figure, even the underlying data of the first categories are constrained to take values below zero in both dimensions; the shape of the underlying data can vary due to the spread of the generated data instances from other categories. This can shift the overall location and shape of the data structure and thus contribute to cluster discrimination.

Further, observe that each $z_{ij\ell }^{\mathrm{n}}$ in (5) is equivalent to (4) with $L_j^{\mathrm{o}}=2$. Hence, in this case, the full covariance matrix is also not identifiable. For ease of notation and computation, we will treat all binary ordinal data as nominal data. Therefore, we can leave the covariance matrix unrestricted for the ordinal data, but the covariance matrix for nominal data is restricted to a correlation matrix with unit elements on its diagonal.

Combining all the vector representations of each data type, we arrive at one vector ${\mathbf{z}}_i$ of dimension $Q=M_{\mathrm{c}}+M_{\mathrm{o}}+{\sum }_{j=1}^{M_{\mathrm{n}}}(L_j^{\mathrm{n}}-1)$ such that ${\mathbf{z}}_i={({{\mathbf{x}}_i^{\mathrm{c}}}^{\prime },{{\mathbf{z}}_i^{\mathrm{o}}}^{\prime },{{\mathbf{z}}_i^{\mathrm{n}}}^{\prime })}^{\prime }\sim {\mathcal{N}}_Q({\mathsf{\mu }}_k,{\Sigma }_k^{\#})$, for $i=1,\dots ,N$ with mean and restricted covariance matrix of the forms:

$$\begin{array}{c}\hfill {\mathsf{\mu }}_k={({{\mathsf{\mu }}_k^{\mathrm{c}}}^{\prime },{{\mathsf{\mu }}_k^{\mathrm{o}}}^{\prime },{{\mathsf{\mu }}_k^{\mathrm{n}}}^{\prime })}^{\prime },{\Sigma }_k^{\#}=\left(\begin{array}{ccc}{\Sigma }_k^{\mathrm{c}}& {\Sigma }_k^{\mathrm{co}}& {\Sigma }_k^{\mathrm{cn}}\\ {{\Sigma }_k^{\mathrm{co}}}^{\prime }& {\Sigma }_k^{\mathrm{o}}& {\Sigma }_k^{\mathrm{on}}\\ {{\Sigma }_k^{\mathrm{cn}}}^{\prime }& {{\Sigma }_k^{\mathrm{on}}}^{\prime }& {\mathbf{R}}_k^{\mathrm{n}}\end{array}\right),\end{array}$$

where $\mathbf{R}$ is a correlation matrix. To complete the specification of our model, we require a prior for the component parameters $({\mathsf{\mu }}_k,{\Sigma }_k^{\#})$. However, owing to the restrictiveness of the component covariance matrix, we cannot use the standard conjugate normal-inverse Wishart prior. Instead, we adopt the following prior:

$$\begin{array}{c}\hfill ({\mathsf{\mu }}_k,{\Sigma }_k^{\#})\sim {\mathcal{N}}_Q({\mathbf{b}}_0,{\mathbf{S}}_0)\times {\mathrm{PXW}}_Q(v_0,{\Omega }_0).\end{array}$$

(6)

Here, we use the multivariate normal distribution with mean ${\mathbf{b}}_0$ and covariance ${\mathbf{S}}_0$ as a prior for the component mean. For the restricted covariance matrix, the PXW distribution [26] with degrees of freedom $v_0$ and scale ${\Omega }_0$ is used as a prior. Originally, the PXW distribution is introduced to sample a correlation matrix ${\mathbf{R}}_k$ by decomposition of ${\Sigma }_k={\mathbf{D}}_k^{1/2}{\mathbf{R}}_k{\mathbf{D}}_k^{1/2}$ where ${\Sigma }_k$ is a standard unrestricted covariance matrix and ${\mathbf{D}}_k$ is a diagonal matrix of variances. Thus, multiplying the density of ${\Sigma }_k$ by the Jacobian of the transformation from ${\Sigma }_k$ to $({\mathbf{D}}_k,\mathbf{R})$, we arrive at the prior density of the PXW distribution,

$$\begin{array}{cc}\hfill p({\mathbf{D}}_k,{\mathbf{R}}_k)& \hfill \propto |{\Sigma }_k{|}^{({\nu }_0-Q-1)/2}\mathrm{etr}\left(-\frac{1}{2}{{\Omega }_0}^{-1}{\Sigma }_k\right)J({\Sigma }_k\to {\mathbf{D}}_k,{\mathbf{R}}_k)\hfill \end{array}$$

(7)

where ${\Sigma }_k\sim {\mathrm{W}}_Q(v_0,{\Omega }_0)$ and etr stands for the expectation of the trace of a matrix. Here, $\mathrm{W}$ stands for Wishart distribution with degrees of freedom $v_0$ and scale ${\Omega }_0$. However, note that, in our case, we are interested in a restricted covariance matrix, not a correlation matrix. So, by setting some of the diagonal elements of ${\mathbf{D}}_k$ to be 1 such that ${\mathbf{D}}_k^{\#}=\mathrm{BD}({\mathbf{D}}_k^{\#\mathrm{c}},{\mathbf{D}}_k^{\#\mathrm{o}},{\mathbf{D}}_k^{\mathrm{n}})$ is a block diagonal matrix with ${\mathbf{D}}_k^{\#\mathrm{c}}={\mathbf{D}}_k^{\#\mathrm{o}}=\mathrm{diag}(1,\dots ,1)$ and ${\mathbf{D}}_k^{\mathrm{n}}$ is the diagonal matrix of variances for nominal part, we will have ${\Sigma }_k^{\#}$ in the desired form from the decomposition of ${\Sigma }_k={\mathbf{D}}_k^{\#1/2}{\Sigma }_k^{\#}{\mathbf{D}}_k^{\#1/2}$. Thus, using the same idea, the prior density of the PXW distribution for our case is defined as

$$\begin{array}{cc}\hfill p({\mathbf{D}}_k^{\#},{\Sigma }_k^{\#})& \hfill \propto |{\Sigma }_k{|}^{({\nu }_0-Q-1)/2}\mathrm{etr}\left(-\frac{1}{2}{{\Omega }_0}^{-1}{\Sigma }_k\right)J({\Sigma }_k\to {\mathbf{D}}_k^{\#},{\Sigma }_k^{\#}).\hfill \end{array}$$

(8)

The PXW distribution is easy to implement as we do not need to check the positive definiteness of the resulting matrix in each iteration as in [17].

Finally, the full hierarchical model can be written as:

$$\begin{array}{cc}\hfill {\mathbf{z}}_i={({{\mathbf{x}}_i^{\mathrm{c}}}^{\prime },{{\mathbf{z}}_i^{\mathrm{o}}}^{\prime },{{\mathbf{z}}_i^{\mathrm{n}}}^{\prime })}^{\prime }|\mathsf{\mu },{\Sigma }^{\#},\mathbf{c}& \sim {\mathcal{N}}_Q({\mathsf{\mu }}_{c_i},{\Sigma }_{c_i}^{\#}),i=1,\dots ,N,\hfill \\ \hfill c_i|\mathbf{W}& \sim \sum _{k=1}^{\infty }{w}_k{\delta }_k(·),i=1,\dots ,N,\hfill \\ \hfill ({\mathsf{\mu }}_k,{\Sigma }_k^{\#})& \sim {\mathcal{N}}_Q({\mathbf{b}}_0,{\mathbf{S}}_0)\times {\mathrm{PXW}}_Q(v_0,{\Omega }_0),k=1,\dots ,\infty .\hfill \end{array}$$

(9)

The model specification is concluded with the setting of parameter $\alpha$, ${\mathbf{b}}_0$, ${\mathbf{S}}_0,v_0$, and ${\Omega }_0$. We assign Gamma$(2,4)$ prior on $\alpha$, following [27]. Next, the parameter ${\mathbf{b}}_0$ is set to be the approximate center of the data. In particular, the part of ${\mathbf{b}}_0$ that belongs to the continuous data is set to be the sample mean, whereas those that belong to ordinal and nominal data are set to be zero. The matrix ${\mathbf{S}}_0$ is set such that ${\mathbf{S}}_0=$diag$\{{(r_1/3)}^2,\dots ,{(r_Q/3)}^2\}$ where $r_j$ denotes the range of the data. The $r_j$s for ordinal and nominal parts are set arbitrarily as 6 [28]. Lastly, we set ${\nu }_0=Q+2$ and ${\Omega }_0={\mathbf{S}}_0^{*}/{\nu }_0$ with ${\mathbf{S}}_0^{*}$ being the matrix ${\mathbf{S}}_0$ where the entries for the nominal part are replaced by one.

3.2. Inferential Algorithm

We use a Markov chain Monte Carlo (MCMC) method to approximate the posterior distribution for inferential purposes. To avoid the need for a finite truncation of the DP, the slice sampler [29] is adapted to our formulation. The idea behind the slice sampler is to introduce an auxiliary variable that makes the infinite-dimensional G conditionally finite. Let $\mathbf{u}={(u_1,\dots ,u_N)}^{\prime }$ be the auxiliary variable such that $0<u_i<1$. By augmenting the auxiliary variable into the model in (9), we have the following joint probability density function:

$$\begin{array}{c}\hfill p({\mathbf{z}}_i,u_i,c_i)=I(u_i<w_{c_i}){\mathcal{N}}_Q({\mathbf{z}}_i;{\mathsf{\mu }}_{c_i},{\Sigma }_{c_i}^{\#}).\end{array}$$

Then, the rest of the sampler follows the framework of the standard slice sampler where the component parameters ${\mathsf{\mu }}_k$ and ${\Sigma }_k^{\#}$, including the concentration parameter $\alpha$, are updated successively according to the prior specification in Section 3.1. Since the latent variable for ordinal and nominal portions are conditional to the component parameters, the corresponding portions of the variable $\mathbf{Z}$ also need to be updated. In addition, two label-switching moves proposed in [30] are added as the last step of the sampler to help the MCMC chain mix better. The full MCMC sampler is implemented in Algorithm 1 as follows:

Algorithm 1 MCMC Algorithm

The state of the MCMC sampler consists of the $\mathbf{w}={(w_1,\dots ,w_K)}^{\prime }$, $\mathit{u}={(u_1,\dots ,u_N)}^{\prime }$, $\mathit{c}={(c_1,\dots ,c_N)}^{\prime }$, ${\mathsf{\mu }}_k:k\in \{c_1,\dots ,c_N\}$, ${\Sigma }_k^{\#}:k\in \{c_1,\dots ,c_N\}$, α, and $\mathbf{Z}={({\mathbf{z}}_1,\dots ,{\mathbf{z}}_N)}^{\prime }$. Then, at each iteration, we execute the following steps:   Step 1: Sample the stick-breaking variable $w_k$ for $k=1,\dots ,K$ such that $w_1={\pi }_1$ and $w_k={\pi }_k{\prod }_{\ell =1}^{k-1}(1-{\pi }_{\ell })$, with ${\pi }_k\sim \mathit{Beta}\left(1+N_k,\alpha +{\sum }_{\ell =k+1}^K{N}_{\ell }\right)$ and $N_k={\sum }_{i=1}^N\mathbf{1}(c_i=k)$.       Step 2: Update the latent variable $u_i$ for $i=1,\dots ,N$ by sampling from $(u_i|\dots )\sim U(0,w_{c_i})$.       Step 3: Update the cluster assignment variable $c_i$ for $i=1,\dots ,N$ with the conditional probability $p(c_i=k|\dots )\propto I(k\in {\mathcal{K}}_i){\mathcal{N}}_Q({\mathbf{z}}_i;{\mathsf{\mu }}_k,{\Sigma }_k^{\#})$. Here, ${\mathcal{K}}_i=\{k:w_k>u_i\}$ denotes a finite set of ks generated by sampling as many of the $w_k$s for $k=1,\dots ,K$ with K is the smallest integer that satisfies ${\sum }_{k=1}^K{w}_k>1-u^{*}$ and $u^{*}=min\{u_i,i=1,\dots ,N\}$.       Step 4: Update ${\mathsf{\mu }}_k$ for $k=1,\dots ,K$ by sampling from $({\mathsf{\mu }}_k|\dots )\sim {\mathcal{N}}_Q({\mathit{b}}_k,{\mathit{S}}_k)$, where ${\mathit{b}}_k={\mathit{S}}_k({\mathit{S}}_0^{-1}{\mathit{b}}_0+N_k{\mathit{S}}_k^{-1}{\overline{\mathit{z}}}_k)$, ${\mathit{S}}_k={({\mathit{S}}_0^{-1}+N_k{\mathit{S}}_k^{-1})}^{-1}$, and ${\overline{\mathit{z}}}_k$ is the mean of the ${\mathit{z}}_k=({\mathit{z}}_i:i\mathit{such}\mathit{that}{c}_i=k)$.   Step 5: Update the restricted covariance matrix ${\Sigma }_k^{\#}$ using the parameter-extended Metropolis-Hastings algorithm (PX-MH) [26]. First, we need to sample an unrestricted form of covariance matrix ${\Sigma }_k^{*}$ from Wishart distribution such that ${\Sigma }_k^{*}\sim {\mathrm{W}}_Q(\eta ,{\Sigma }_{\mathbf{k}}/\eta )$ with η is tuned such that the total acceptance rate across all the non-empty components is roughly $23\%$ [31]. Then, the restricted covariance matrix ${\Sigma }_k^{\#}$ is generated from the decomposition ${\Sigma }_k^{*}={\mathbf{D}}_k^{\#*\frac{1}{2}}{\Sigma }_k^{\#*}{\mathbf{D}}_k^{\#*\frac{1}{2}}$. The new ${\Sigma }_k^{\#*}$ is accepted as ${\Sigma }_k^{\#}$ according to probability $$min\left\{1,\frac{p({\mathbf{D}}_k^{\#*},{\Sigma }_k^{\#*}|\dots )}{p({\mathbf{D}}_k^{\#},{\Sigma }_k^{\#}|\dots )}\frac{q({\Sigma }_k|{\Sigma }_k^{*})}{q({\Sigma }_k^{*}|{\Sigma }_k)}\right\}.$$ Here, the joint posterior $p({\mathbf{D}}_k^{\#},{\Sigma }_k^{\#}|\dots )$ is derived as $$\begin{array}{cc}\hfill p({\mathbf{D}}_k^{\#},{\Sigma }_k^{\#}|\dots )& \propto p({\mathbf{D}}_k^{\#},{\Sigma }_k^{\#})\times \prod _{i:c_i=k}p({\mathbf{z}}_i|{\mathsf{\mu }}_k,{\Sigma }_k^{\#})\hfill \\ & \propto |{\Sigma }_k{|}^{({\nu }_0-Q-1)/2}\mathit{etr}\left(-\frac{1}{2}{{\Omega }_0}^{-1}{\Sigma }_k\right)J({\Sigma }_k\to {\mathbf{D}}_k^{\#},{\Sigma }_k^{\#})\times \hfill \\ & \prod _{i:c_i=k}{|{\Sigma }_k^{\#}|}^{-1/2}\mathit{exp}\left(-\frac{1}{2}({\mathbf{z}}_i-{\mathsf{\mu }}_k){\Sigma }_k^{\#-1}{({\mathbf{z}}_i-{\mathsf{\mu }}_k)}^{\prime }\right)\hfill \\ & \propto |{\Sigma }_k^{\#}{|}^{\frac{{\nu }_0-Q-1-N_k}{2}}{|{\mathbf{D}}_k^{\#}|}^{\frac{{\nu }_0}{2}-1}\mathit{etr}\left(-\frac{1}{2}\left({\Omega }_0^{-1}{\mathbf{D}}_k^{\#1/2}{\Sigma }_k^{\#}{\mathbf{D}}_k^{\#1/2}+{\Sigma }_k^{\#-1}{\mathbf{S}}_{{\mathsf{\mu }}_k}\right)\right),\hfill \end{array}$$ where ${\mathbf{S}}_{{\mathsf{\mu }}_k}={\sum }_{i:c_i=k}({\mathbf{z}}_i-{\mathsf{\mu }}_k){({\mathbf{z}}_i-{\mathsf{\mu }}_k)}^{\prime }$ and $J({\Sigma }_k\to {\mathbf{D}}_k^{\#},{\Sigma }_k^{\#})={|{\mathbf{D}}_k^{\#}|}^{(Q-1)/2}$. For the proposal density, it is defined as $$\begin{array}{c}\hfill q({\Sigma }_k^{*}|{\Sigma }_k)={\mathrm{W}}_Q({\Sigma }_k^{*};\eta ,{\Sigma }_k/\eta )J({\Sigma }_k^{*}\to {\mathbf{D}}_k^{\#*},{\Sigma }_k^{\#*})\end{array}$$ with $J({\Sigma }_k^{*}\to {\mathbf{D}}_k^{\#*},{\Sigma }_k^{\#*})={|{\mathbf{D}}_k^{\#*}|}^{(Q-1)/2}$.       Step 6: Update the concentration parameter α using the move proposed in [27] such that $(\alpha |\dots )\sim w_{\zeta }\mathit{Gamma}(2+K,4-\mathit{log}\left(\zeta \right))+(1-w_{\zeta })\mathit{Gamma}(2+K-1,4-\mathit{log}\left(\zeta \right)),$ where $(\zeta |\alpha ,K)\sim \mathit{Beta}(\alpha +1,N)$. Here, the weight $w_{\zeta }$ is set to satisfy the following: $$\begin{array}{c}\hfill \frac{w_{\zeta }}{1-w_{\zeta }}=\frac{2+K-1}{N(4-log(\zeta \left)\right)}.\end{array}$$   Step 7: Update the latent variable $\mathbf{Z}$ for ordinal and nominal data as follows:     For each ordinal instance $x_{ij}^{\mathrm{o}},i=1,\dots ,N$, $j=1,\dots ,M_{\mathrm{o}}$, sample $(z_{i{j}^{\prime }}|\dots )\sim \mathbf{1}({\gamma }_{j,x_{ij}^{\mathrm{o}}-1}<z_{i{j}^{\prime }}\le {\gamma }_{j,x_{ij}^{\mathrm{o}}})\mathcal{N}(z_{i{j}^{\prime }};{\tilde{m}}_{i{j}^{\prime }},{\tilde{v}}_{i{j}^{\prime }}),$ where $j^{\prime }=M_{\mathrm{c}}+j$, ${\tilde{m}}_{i{j}^{\prime }}$ and ${\tilde{v}}_{i{j}^{\prime }}$ are the conditional mean and variance of $z_{i{j}^{\prime }}$ given $z_{i\ell },\ell \ne j^{\prime }$ such that $$\begin{array}{c}{\tilde{m}}_{i{j}^{\prime }}={\mathsf{\mu }}_{k{j}^{\prime }}+{\Sigma }_{k{j}^{\prime }(-j^{\prime })}{\Sigma }_{k(-j^{\prime })(-j^{\prime })}^{-1}(z_{i{j}^{\prime }}-{\mathsf{\mu }}_{k(-j^{\prime })}),\\ {\tilde{v}}_{i{j}^{\prime }}={\Sigma }_{k{j}^{\prime }{j}^{\prime }}-{\Sigma }_{k{j}^{\prime }(-j^{\prime })}{\Sigma }_{k(-j^{\prime })(-j^{\prime })}^{-1}{\Sigma }_{k(-j^{\prime })j^{\prime }}.\end{array}$$ Here, ${\mathsf{\mu }}_{k(-j^{\prime })}$ is the vector of the ${\mathsf{\mu }}_k$ excluding its $j^{\prime }$th element, ${\Sigma }_{k{j}^{\prime }(-j^{\prime })}$ is the transpose of ${\Sigma }_{k(-j^{\prime })j^{\prime }}$, ${\Sigma }_{k{j}^{\prime }(-j^{\prime })}$ is the $j^{\prime }$th row vector of covariance matrix ${\Sigma }_k$ excluding its $j^{\prime }$th element, and ${\Sigma }_{k(-j^{\prime })(-j^{\prime })}$ is the covariance matrix ${\Sigma }_k$ excluding its $j^{\prime }$th row and $j^{\prime }$th column.   For each nominal instance $x_{ij}^{\mathrm{n}}$, $i=1,\cdots ,N,j=1,\cdots ,M_{\mathrm{n}}$, sample $(z_{i{j}^{\prime }}|\cdots )\sim \mathbf{1}(z_{i{j}^{\prime }}\in A_{i{j}^{\prime }})\mathcal{N}(z_{i{j}^{\prime }};{\tilde{m}}_{i{j}^{\prime }},{\tilde{v}}_{i{j}^{\prime }}),$ for $j^{\prime }=M_{\mathrm{c}}+M_{\mathrm{o}}+{\sum }_{h<j}(L_h^{\mathrm{n}}-1)+1,\cdots ,M_{\mathrm{c}}+M_{\mathrm{o}}+{\sum }_{h\le j}(L_h^{\mathrm{n}}-1)$ with $$\begin{array}{c}{A}_{i{j}^{\prime }}=\left\{\begin{array}{ll}{z}_{i{j}^{\prime }}<0& \mathit{if}{x}_{ij}^{\mathrm{n}}=0\\ z_{i{j}^{\prime }}=max\left\{z_{i\ell }\right\},>0& \mathit{if}{x}_{ij}^{\mathrm{n}}=r\\ z_{i{j}^{\prime }}<max\left\{z_{i\ell }\right\},& \mathit{otherwise}\end{array}\right.\end{array}$$ for which $\ell =M_{\mathrm{c}}+M_{\mathrm{o}}+{\displaystyle \sum _{h<j}}(L_h^{\mathrm{n}}-1)+1,\cdots ,M_{\mathrm{c}}+M_{\mathrm{o}}+{\displaystyle \sum _{h\le j}}(L_h^{\mathrm{n}}-1)$, $r=j^{\prime }-(M_{\mathrm{c}}+M_{\mathrm{o}}+{\sum }_{h<j}(L_h^{\mathrm{n}}-1))$, and ${\tilde{m}}_{i{j}^{\prime }}$, ${\tilde{v}}_{i{j}^{\prime }}$ are as before. In case of missing instance, the univariate normal distribution with the conditional mean and variance is used to impute the corresponding $z_{i{j}^{\prime }}$.   Step 8: We implement the two label-switching moves described in [30] to improve the mixing of the algorithm. The moves are given as follows: (i) Choose randomly two non-empty clusters, say, i and j, and propose to exchange their assignment variable and cluster-specific parameters. Then, the exchange is accepted with the following probability: $$\mathit{min}\left\{1,{\left(\frac{w_j}{w_i}\right)}^{N_i-N_j}\right\}.$$ (ii) Choose randomly two adjacent non-empty clusters, say, i and $i+1$, and propose to exchange their assignment variable and parameters. Then, the exchange is accepted with the following probability: $$\mathit{min}\left\{1,\frac{{(1-{\pi }_{i+1})}^{N_i}}{{(1-{\pi }_i)}^{N_{i+1}}}\right\}.$$ These moves are complementary, as the former has a high probability of switching two clusters with similar weights while the latter has a high probability of switching two clusters of unequal weights.

Since this study focuses on the clustering problem, we are only interested in the cluster assignment variable $\mathbf{c}={(c_1,\dots ,c_N)}^{\prime }$ where each unique value of $c_i$ represents a cluster. Then, the number of clusters is derived as the number of unique values in $\mathbf{c}$. Both these values will be kept throughout the iterations.

4. Experimental Results

This section presents some empirical experiments with simulated and real data to illustrate the performance of the proposed model.

4.1. Simulated Data Examples

In this experiment, we simulated two different examples to assess the performance of the proposed model in determining the true cluster structure. Throughout this experiment, we consider a setup consisting of two continuous variables $(X_1^{\mathrm{c}},X_2^{\mathrm{c}})$, two ordinal variables $(X_1^{\mathrm{o}},X_2^{\mathrm{o}})$, each having four levels, and two nominal variables $(X_1^{\mathrm{n}},X_2^{\mathrm{n}})$, each having four categories.

In the first example, we consider a case where a mixture model with a common covariance matrix is used to simulate the underlying latent vectors of sample size 1000. Let ${\mathbf{z}}_i=(z_{i1}\dots ,z_{i10})$ for $i=1,\dots ,1,000$ be the underlying latent vectors simulated from an equal weight mixture model of three multivariate normal, ${\mathcal{N}}_{10}({\mathsf{\mu }}_1,{\mathbf{I}}_{10})+{\mathcal{N}}_{10}({\mathsf{\mu }}_2,{\mathbf{I}}_{10})+{\mathcal{N}}_{10}({\mathsf{\mu }}_3,{\mathbf{I}}_{10})$, with the component means as follows

$$\begin{array}{ccc}\hfill {\mu }_{1j}\hfill & \hfill =& (1/j){(-1)}^j+\Delta ,\hfill \\ \hfill {\mu }_{2j}\hfill & \hfill =& (1/j){(-1)}^j,\hfill \\ \hfill {\mu }_{3j}\hfill & \hfill =& (1/j){(-1)}^j-\Delta ,j=1,\dots ,10.\hfill \end{array}$$

Here, $\Delta$ represents the Euclidean distance between the component means, and the following values $\Delta \in \{1.9,2.1,2.3\}$ are considered. These values of $\Delta$ are designed to cover a range of cases, from highly overlapped to highly separated cluster structures. The greater the value of $\Delta$, there will be less overlap between the generated clusters as illustrated in Figure 3. Then, from the simulated latent vectors, we set the continuous data instance such that $(x_{i1}^{\mathrm{c}},x_{i2}^{\mathrm{c}})=(z_{i1},z_{i6})$. For ordinal data instances, we set $x_{i1}^{\mathrm{o}}$ and $x_{i1}^{\mathrm{o}}$ by discretizing the scaled $z_{i2}$ and $z_{i7}$, respectively, using the threshold described in Section 3.1. Lastly, for the nominal data instance, we set $x_{i1}^{\mathrm{n}}$ by transforming scaled $(z_{i3},z_{i4},z_{i5})$ using the probit specification as in Section 3.1, while for $x_{i2}^{\mathrm{n}}$, we use $(z_{i8},z_{i9},z_{i10})$.

To highlight the impact of different covariance structure assumptions on the clustering performance, we also run our model with the covariance matrix set to be common across components such that

$$\begin{array}{cc}\hfill {\mathbf{z}}_i={({\mathbf{x}}_i^{\mathrm{c}\prime },{\mathbf{z}}_i^{\mathrm{o}\prime },{\mathbf{z}}_i^{\mathrm{n}\prime })}^{\prime }|\mathsf{\mu },{\Sigma }^{\#},\mathbf{c}& \sim {\mathcal{N}}_Q({\mathsf{\mu }}_{c_i},{\Sigma }^{\#}),i=1,\dots ,N,\hfill \\ \hfill c_i|\mathbf{W}& \sim \sum _{k=1}^{\infty }{w}_k{\delta }_k(·),i=1,\dots ,N,\hfill \\ \hfill {\mathsf{\mu }}_k& \sim {\mathcal{N}}_Q({\mathbf{b}}_0,{\mathbf{S}}_0),k=1,\dots ,\infty ,\hfill \\ \hfill {\Sigma }^{\#}& \sim {\mathrm{PXW}}_Q(v_0,{\Omega }_0).\hfill \end{array}$$

(10)

The algorithm for the model in (10) is almost similar to Algorithm 1 except for Step 5, where the restricted covariance matrix ${\Sigma }^{\#}$ is updated only once. The joint posterior $p({\mathbf{D}}^{\#},{\Sigma }^{\#}|\dots )$ is then defined based on the likelihood of all data instances as follows:

$$\begin{array}{cc}\hfill p({\mathbf{D}}^{\#},{\Sigma }^{\#}|\dots )& \propto p({\mathbf{D}}^{\#},{\Sigma }^{\#})\times \prod _{i=1}^{N}p({\mathbf{z}}_i|{\mathsf{\mu }}_k,{\Sigma }^{\#})\hfill \\ \hfill & \propto |{\Sigma }^{\#}{|}^{\frac{{\nu }_0-Q-1-N}{2}}{|{\mathbf{D}}^{\#}|}^{\frac{{\nu }_0}{2}-1}\mathrm{etr}\left(-\frac{1}{2}\left({\Omega }_0^{-1}{\mathbf{D}}^{\#1/2}{\Sigma }^{\#}{\mathbf{D}}^{\#1/2}+{\Sigma }^{\#-1}{\mathbf{S}}_{\mathsf{\mu }}\right)\right),\hfill \end{array}$$

where ${\mathbf{S}}_{\mathsf{\mu }}={\sum }_{k=1}^K{\sum }_{i:c_i=k}({\mathbf{z}}_i-{\mathsf{\mu }}_k){({\mathbf{z}}_i-{\mathsf{\mu }}_k)}^{\prime }$.

Throughout this paper, the algorithm is initialized using k-means on random $\mathbf{Z}$ with ten clusters for both cluster-specific and common covariance models. Then, the algorithm is run for 100,000 iterations with 50,000 as burn-in, after which we take samples of cluster assignment variable $\mathit{c}$ with a thinning of 30, resulting in 1667 samples of $\mathit{c}$. As for the parameter settings, we use the default values described in Section 3.1. Our current implementation is performed in the R environment [32]. The total run time varied depending on the data sample size and the number of variables involved. For instance, the simulation study takes around 3 h per one run on a laptop with a 2.7 GHz quad-core processor.

Due to the weak identifiability of the mixture components, the posterior distribution exhibits a multimodal behavior [30]. This may cause the algorithm to stick in one of the modes. In such situations, even if the algorithm is run sufficiently many iterations, it will keep producing outputs around that particular mode. This issue has been noted in many Bayesian nonparametric models (see, for example, [30,33,34]), where most of the time the full exploration of the MCMC samples space is not guaranteed. Figure 4 provides the trace plots of the sampled number of clusters K by both models. These trace plots show that the K stabilizes to a common value in most cases. However, for the trace plot for the model with cluster-specific covariance when $\Delta =1.9$, we can see that the algorithm is stuck in a mode that exhibits a lower number of clusters. To handle this issue, it is common practice to restart the algorithm multiple times, ensuring the consistency of the results. Hence, the clustering results in this manuscript are reported based on ten runs of the algorithms with different initializations of $\mathbf{Z}$. It should be noted that, if the data sample size is too small relative to the perceived number of clusters, the algorithm may fail to converge since the data are not sufficient to dominate the prior.

In order to gauge the clustering performance, we adopt the following criteria as a measure of agreement between the estimated partition and the true partition:

• Normalized mutual information (NMI) [35]

  $$\begin{array}{c}\hfill \mathrm{NMI}=\frac{{\sum }_k{\sum }_{\widehat{k}}\frac{|S_k\cap {\widehat{S}}_{\widehat{k}}|}{N}log\frac{N|S_k\cap {\widehat{S}}_{\widehat{k}}|}{|S_k|·|{\widehat{S}}_{\widehat{k}}|}}{\sqrt{\left({\sum }_k\frac{|S_k|}{N}log\frac{|S_k|}{N}\right)\left({\sum }_{\widehat{k}}\frac{|{\widehat{S}}_{\widehat{k}}|}{N}log\frac{|{\widehat{S}}_{\widehat{k}}|}{N}\right)}}\end{array}$$
• Adjusted Rand index (ARI) [36]

  $$\begin{array}{c}\hfill \mathrm{ARI}=\frac{{\sum }_{ij}\left(\genfrac{}{}{0pt}{}{|S_i\cap {\widehat{S}}_j|}{2}\right)-\frac{{\sum }_i\left(\genfrac{}{}{0pt}{}{|S_i|}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{|{\widehat{S}}_j|}{2}\right)}{\left(\genfrac{}{}{0pt}{}{N}{2}\right)}}{\frac{1}{2}\left[{\sum }_i\left(\genfrac{}{}{0pt}{}{|S_i|}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{|{\widehat{S}}_j|}{2}\right)\right]-\frac{{\sum }_i\left(\genfrac{}{}{0pt}{}{|S_i|}{2}\right){\sum }_j\left(\genfrac{}{}{0pt}{}{|{\widehat{S}}_j|}{2}\right)}{\left(\genfrac{}{}{0pt}{}{N}{2}\right)}}\end{array}$$

  where $\widehat{\mathcal{S}}=({\widehat{S}}_1,\dots ,{\widehat{S}}_K)$ is the estimated partition such that ${\widehat{S}}_k=\{j:c_j=k\mathrm{for}j=1,\dots ,N$}, $\mathcal{S}=(S_1,S_2,\dots ,S_K)$ is the true partition, and $|·|$ denotes the number of instances in a partition.

Both NMI and ARI take a value between zero and one; a higher value indicates a better clustering performance.

Table 1. Results for Simulation 1.

		Common Covariance	Cluster-Specific Covariance
$\Delta =1.9$	NMI	0.803	0.683
		(0.015)	(0.088)
	ARI	0.850	0.625
		(0.015)	(0.177)
	$\widehat{K}$	3	3
$\Delta =2.1$	NMI	0.853	0.836
		(0.014)	(0.018)
	ARI	0.894	0.873
		(0.012)	(0.020)
	$\widehat{K}$	3	3
$\Delta =2.3$	NMI	0.897	0.888
		(0.012)	(0.015)
	ARI	0.932	0.920
		(0.010)	(0.013)
	$\widehat{K}$	3	3

shows the average values of NMI and ARI (together with their standard deviation in parentheses) of both models. Since the algorithm is run multiple times, the average values here are taken over all 1667 × 10 sampled $\mathbf{c}$s. The estimated number of clusters $\widehat{K}$ is also reported in Table 1, which is the most frequent number of clusters visited in all the sampled $\mathbf{c}$s. In general, the model with cluster-specific covariance performs worse than the model with common covariance. This result has been anticipated since the simulated data are consistent with the assumption of the common covariance model. The difference in performance is more evident in the case of clusters with high overlap among them. Nevertheless, both models exhibit an almost similar clustering performance when the clusters are well separated. In terms of estimating the true number of clusters, both models succeed in capturing the true cluster structure with $\widehat{K}=3$. We also made an attempt to apply the model proposed in [17] on the simulated data as a comparison. Unfortunately, since their algorithm involves many prior hyperparameters that need to be tuned specifically for each dataset, we failed to obtain their algorithm to mix properly, resulting in many singleton clusters. In addition, compared to their algorithm, our proposed algorithm requires only one tuning hyperparameter to control the acceptance rate of the Metropolis–Hasting update. Note that we also use the same model parameters and algorithm settings throughout the paper.

As for the second example, we use a similar experimental setup as in the previous one but with a different covariance structure for each cluster. We set ${\Sigma }_1=\mathrm{CS}\left(0.9\right)$, ${\Sigma }_2=\mathrm{CS}\left(0.5\right)$, and ${\Sigma }_3=\mathrm{CS}\left(0.1\right)$ where CS refers to a compound symmetry matrix. The CS matrix is a matrix with equal variances on the main diagonal and equal correlations on all off-diagonal elements. The different parameters of the CS matrices introduce different variances and correlations between the underlying latent variables and thus change the geometric orientation of the covariance matrix of each cluster. From the scatterplot of the simulated data in Figure 5, especially on plot $X_1^{\mathrm{c}}$ versus $X_2^{\mathrm{c}}$, we can see that each cluster exhibits different shapes. In this example, we consider $\Delta \in \{3.3,3.5,3.7\}$.

We report the result for the second scenario in

Table 2. Results for Simulation 2.

		Common Covariance	Cluster-Specific Covariance
$\Delta =3.3$	NMI	0.762	0.834
		(0.044)	(0.014)
	ARI	0.757	0.873
		(0.082)	(0.013)
	$\widehat{K}$	5	3
$\Delta =3.5$	NMI	0.779	0.862
		(0.041)	(0.012)
	ARI	0.778	0.897
		(0.071)	(0.011)
	$\widehat{K}$	4	3
$\Delta =3.7$	NMI	0.839	0.890
		(0.032)	(0.011)
	ARI	0.861	0.923
		(0.045)	(0.009)
	$\widehat{K}$	4	3

. It is clear that the model with cluster-specific covariance matrices performs better with a higher NMI and ARI scores compared to the model with common covariance. In addition, the model with cluster-specific covariance assumption is able to recover the true three-cluster structure in all cases. On the contrary, the model with a common covariance assumption overestimates the number of clusters. Even in a highly separated cluster structure, the model with common covariance fails to infer the true K, and the overestimation of K is still observed. This overestimation of the number of clusters is the result of the nonparametric feature of the DPMM where it will adapt the number of components to optimize the model fit. For illustration, we plot the simulated data according to one of the sampled assignment variables from the inferential algorithm in Figure 6. Due to the strict assumption of a common covariance matrix, the model in Figure 6a is forced to introduce superfluous components to fit the data better while maintaining the same shape of the covariance matrix across all clusters in the data.

4.2. Real Data Applications

As real data applications, we consider two datasets: prostate cancer and chronic kidney disease data.

4.2.1. Prostate Cancer Data

We examine a dataset from a randomized clinical trial [37], which has been used in several studies (see [10,38]). The dataset involves 502 patients classified into clinical stage 3 and stage 4 prostate cancer. The stage 3 cancer patients only show a local extension of the tumor and exclude the presence of distant metastasis, while stage 4 cancer patients report distant metastasis. As listed in

Table 3. The variables in the prostate cancer data.

Variable	Type of Data	Number of Levels/Categories
Age	Continuous
Weight index	Continuous
Systolic blood pressure	Continuous
Diastolic blood pressure	Continuous
Serum hemoglobin	Continuous
Size of primary lesion	Continuous
Index of tumor stage/grade	Continuous
Prostatic acid phosphatase	Continuous
Performance rating	Ordinal	4
Cardiovascular disease	Nominal	2
Electrocardiogram code	Nominal	7
Bone metastasis	Nominal	2

, there are twelve clinical variables collected from the patients, with eight of them being continuous, one being ordinal, and three being nominal. Following the pre-processing steps outlined in [38], the variable size of primary lesion and prostatic acid phosphatase are transformed using square root and log function, respectively. In this study, we will evaluate the ability of the proposed model to differentiate between the two stages of cancer.

The clustering results for models with common covariance and cluster-specific covariance are reported in

Table 4. Results for the prostate cancer data.

	Common Covariance	Cluster-Specific Covariance
NMI	0.291	0.477
	(0.079)	(0.041)
ARI	0.287	0.584
	(0.113)	(0.043)
$\widehat{K}$	5	2

. The NMI and ARI scores are measured based on the agreement with the clinical stages. The model with cluster-specific covariance accurately select $\widehat{K}=2$, which is in line with the clinical stages. On the other hand, as expected, the model with common covariance overestimates the number of clusters to better accommodate the data. It is thus evident that the common covariance assumption is too rigid in real application, thus impairing the clustering performance.

Given samples of the posterior draw of cluster assignment variable $\mathbf{c}$, it is often desirable to determine an optimal partition that best represents all the output of the algorithm. To obtain that single estimate, we use the method proposed in [39] to post-process all the sampled $\mathbf{c}$s based on the variation of information (VI). This method involves calculating the loss function between each output of the inferential algorithm $\widehat{\mathbf{c}}$ and the unknown true assignment variable $\mathbf{c}$ such that

$$\begin{array}{cc}\hfill {\mathbf{c}}^{\mathrm{opt}}& ={\mathrm{argmin}}_{\widehat{\mathbf{c}}}\mathbb{E}[\mathrm{VI}(\mathbf{c},\widehat{\mathbf{c}})|\mathcal{D}]\hfill \\ \hfill & ={\mathrm{argmin}}_{\widehat{\mathbf{c}}}\sum _{n=1}^{N}log\left(\sum _{n^{\prime }=1}^N\mathbf{1}({\widehat{c}}_n={\widehat{c}}_{n^{\prime }})\right)-\hfill \\ \hfill & 2\sum _{n=1}^{N}log\left(\sum _{n^{\prime }=1}^{N}p({\widehat{c}}_n={\widehat{c}}_{n^{\prime }}|\mathcal{D})\mathbf{1}({\widehat{c}}_n={\widehat{c}}_{n^{\prime }})\right)\hfill \end{array}$$

(11)

where $\mathcal{D}$ represents the data. Hence, the optimal partition is derived from ${\mathbf{c}}^{\mathrm{opt}}$ that minimizes (11). The search for ${\mathbf{c}}^{\mathrm{opt}}$ is restricted to all the output $\mathbf{c}$s sampled from the inferential algorithm.

Table 5. The confusion matrix for the prostate cancer data under the model with common covariance matrix.

		Clinical Classification
		Stage 3	Stage 4
Estimated	Cluster #1	287	123
	Cluster #2	1	80
	Cluster #3	0	10
	Cluster #4	1	0

reports the confusion matrix of the optimal partition under the model with common covariance. As the optimal partition is the one that best represents all the $\mathbf{c}$s, the number of clusters reported in Table 5 may not match the most frequent number of clusters $\widehat{K}$ as inferred by the algorithm (see Table 4). Upon initial observation, the model appears to perform satisfactorily in grouping stage 3 patients into Cluster #1, with only two stage 3 patients grouped into different clusters. Despite this, over half of the stage 4 patients are also grouped into Cluster #1. This suggests that the model with common covariance has failed to differentiate between stage 3 and stage 4 patients.

Table 6. The confusion matrix for the prostate cancer data under the model with cluster-specific covariance matrices.

		Clinical Classification
		Stage 3	Stage 4
Estimated	Cluster #1	267	25
	Cluster #2	22	188

reports the confusion matrix of the optimal partition under the model with cluster-specific covariance. It appears that there is a disparity between the optimal partition and the clinical classification for 47 patients. However, in general, the model is able to differentiate between the two stages of cancer, with the majority of stage 3 patients being grouped into Cluster #1 and stage 4 patients into Cluster #2. It is also worth mentioning that our model seems to outperform the model in [10], which reports 67 patients being grouped into the wrong cluster. Their study treats variable performance rating as a nominal variable instead of an ordinal since their location mixture model can only handle continuous and nominal data. Apart from that, the performance of our model also surpasses the model in [38], which groups the patients into three clusters. We believe that this is due to the parsimonious covariance structure assumed by their model, which imposes some restrictions on the shape of the clusters and thus requires more clusters to better fit the data.

4.2.2. Chronic Kidney Disease

To further showcase the utility of the proposed model, we apply the model to the chronic kidney disease dataset from [40]. The dataset consists of 400 instances of clinical data from patients with an age range from 2 to 90 years. Out of 400 samples, 250 of them are diagnosed with chronic kidney disease (CKD). Therefore, we will use the proposed model to discriminate CKD patients from non-CKD patients. All 24 clinical variables collected are listed in

Table 7. The variables in the chronic kidney disease data.

Variable	Type of Data	Number of Levels/Categories
Age	Continuous
Blood pressure	Continuous
Blood glucose random	Continuous
Blood urea	Continuous
Serum creatinine	Continuous
Sodium	Continuous
Potassium	Continuous
Hemoglobin	Continuous
Packed cell volume	Continuous
White blood cell count	Continuous
Red blood cell count	Continuous
Specific gravity	Ordinal	5
Albumin	Ordinal	6
Sugar	Ordinal	6
Red blood cells	Nominal	2
Pus cell	Nominal	2
Pus cell clumps	Nominal	2
Bacteria	Nominal	2
Hypertension	Nominal	2
Diabetes mellitus	Nominal	2
Coronary artery disease	Nominal	2
Appetite	Nominal	2
Pedal edema	Nominal	2
Anemia	Nominal	2

. As far as we know, these data have not yet been analyzed in previous related studies involving model-based clustering.

Upon first inspection of the data, we observed that some data instances appear abnormal. One data point reports an unnatural level of sodium at $4.5$ mEq/L, lower than that ever recorded in the medical literature, which is at 99 mEq/L [41]. In addition, we have also come across two other data points exhibiting significantly high potassium levels, at 39 and 47 mEq/L, respectively. These readings exceed those reported in a documented case study of extreme hyperkalemia with a potassium level of 14 mEq/L [42]. To account for the possibility of data entry errors, these three data points are treated as missing values. Furthermore, we also observe a significant skewness in variable blood urea and serum creatinine. Hence, a logarithmic transformation is applied to these variables.

From

Table 8. Results for the chronic kidney disease data.

	Common Covariance	Cluster-Specific Covariance
NMI	0.577	0.704
	(0.026)	(0.023)
ARI	0.449	0.767
	(0.033)	(0.025)
$\widehat{K}$	9	2

, a similar trend in performance can be seen where the model with common covariance overclusters the data with the estimated number of clusters of 9 and thus shows a poor clustering performance as measured by the NMI and ARI. On the other hand, the model with a cluster-specific covariance matrix correctly identifies the true number of clusters at 2.

The confusion matrix of the optimal estimated partition under the model with common covariance model is reported in

Table 9. The confusion matrix for the chronic kidney disease data under the model with common covariance matrix.

		Clinical Classification
		CKD	Non-CKD
Estimated	Cluster #1	9	150
	Cluster #2	138	0
	Cluster #3	42	0
	Cluster #4	28	0
	Cluster #5	15	0
	Cluster #6	13	0
	Cluster #7	2	0
	Cluster #8	2	0
	Cluster #9	1	0

. We can see that all 150 non-CKD patients are grouped into one cluster, Cluster #1. However, Cluster #1 does not contain only non-CKD individuals but also nine misclassified CKD patients. The way that the remaining CKD patients are grouped into eight distinct clusters, most of which are smaller in size, suggests that a single covariance shape was insufficient to fully capture the cluster structure of the CKD group. The difference only in the cluster locations is not enough to adequately represent the difference in shapes of the underlying data due to the different proportions of the instances across levels/categories along with the correlation that exists between variables. The assumption of common covariance results in overfitting one particular cluster, which in turn hinders the ability to fit the rest of the data.

Table 10. The confusion matrix for the chronic kidney disease data under the model with cluster-specific covariance matrices.

		Clinical Classification
		CKD	Non-CKD
Estimated	Cluster #1	250	23
	Cluster #2	0	127

shows the optimal partition under the model with cluster-specific covariance matrices. It is evident that all of the 250 CKD patients have been accurately classified into Cluster #1. Out of 150 individuals who are not diagnosed with CKD, only 23 are misclassified as having CKD and grouped into Cluster #1. While the model with cluster-specific covariance provides a better fit for the CKD groups, it comes at the cost of a reduced fit for the non-CKD group. The outcome contradicts the one obtained from the model with a common covariance structure, which demonstrates a superior fit for the non-CKD group in comparison to the CKD group. This is due to the difference in covariance structure that molds the underlying latent vector and thus causes changes in the cluster configuration.

5. Conclusions

This article presents a model for clustering mixed continuous, ordinal, and nominal data that allow for cluster-specific covariance matrices. Combining the continuous variables with the latent variables framework to underlie the ordinal and nominal variables allows us to use a DPM model with multivariate Gaussian kernels. The flexibility of our model over the model with common covariance matrix assumption is illustrated in a simulated example of clusters with different geometric orientations. This is often the case in real practice, as supported by the application in prostate cancer staging and chronic kidney disease classification.

To compensate for the fixed cutoffs of the underlying variables of the non-continuous data, the different shapes of the covariance matrices are needed to fully capture the proportion variation across different levels/categories that contribute towards cluster discrimination. Applying the model with a common covariance assumption may occasionally improve the clustering performance in a dataset where every cluster is believed to have an exact identical shape. However, such situations are rare in real clustering problems since a mere correlation between the variables is enough to give the different orientations to the covariance matrix.

One key issue that arises in sampling is the restricted covariance matrices. By taking advantage of the development in Bayesian prior construction, we have run our model with the use of the PX-MH [26] due to its relatively simple implementation. Nevertheless, in general, other updating steps could be used, such as in [17], although computational complexity is a factor that needs to be considered.

Our future work in progress is to consider how to incorporate other hierarchical prior structures in an attempt to select relevant variables in clustering mixed-type data.