Multi-CC: A New Baseline for Faster and Better Deep Clustering

Abstract

The aim of our paper is to introduce a new deep clustering model called Multi-head Cross-Attention Contrastive Clustering (Multi-CC), which seeks to enhance the performance of the existing deep clustering model CC. Our approach involves first augmenting the data to form image pairs and then using the same backbone to extract the feature representation of these image pairs. We then undertake contrastive learning, separately in the row space and column space of the feature matrix, to jointly learn the instance and cluster representations. Our approach offers several key improvements over the existing model. Firstly, we use a mixed strategy of strong and weak augmentation to construct image pairs. Secondly, we get rid of the pooling layer of the backbone to prevent loss of information. Finally, we introduce a multi-head cross-attention module to improve the model’s performance. These improvements have allowed us to reduce the model training time by 80%. As a baseline, Multi-CC achieves the best results on CIFAR-10, ImageNet-10, and ImageNet-dogs. It is easily replaceable with CC, making models based on CC achieve better performance.

1. Introduction

Clustering is a fundamental problem in unsupervised learning, aiming to divide samples in a dataset into several categories so that samples within the same category have high similarity and those in different categories have low similarity. The goal of traditional clustering methods is to minimize the clustering objective function under a given distance metric, but the computational cost is often enormous for high-dimensional data [1]. With the development of deep learning, deep clustering has become one of the mainstream methods for unsupervised classification. Early deep clustering methods used deep neural networks for feature representation and then applied traditional clustering methods (such as K-Means [2]) for clustering. However, this approach cannot jointly optimize feature representation and clustering, leading to suboptimal solutions [3]. In order to address the above issues, [4] proposed a method to jointly train and optimize feature representation and clustering, thus learning a feature space that clusters data into different groups. In recent years, some new methods, such as in [5], can also jointly optimize feature representation and clustering, achieving good performance on high-dimensional datasets.

Contrastive learning is one of the research hotspots in the field of deep learning [6]. By comparing the similarity or difference between different samples, it learns the representation of samples, which has advanced advantages compared to traditional supervised and unsupervised learning methods. Existing representative contrastive learning methods include SimCLR [6], MoCo [7], and SimSiam [8], which typically adopt a Siamese network structure, obtain positive and negative instance pairs of images through data augmentation, and perform a large number of positive and negative sample pair comparisons to better extract data feature representations.

In the field of clustering, Contrastive Clustering (CC) proposed by [9] is a widely used method. It combines contrastive learning and deep clustering to improve clustering performance. CC constructs positive and negative instance pairs using data augmentation and maps them to the feature space using two independent Multi-Layer Perceptrons (MLPs). Instance-level and cluster-level contrastive learning is performed in row space and column space. This aims to maximize the similarity between positive pairs and minimize the similarity between negative pairs. However, mapping features to row space and column space may not fully capture the complex structure and semantic information in the data, which can lead to suboptimal solutions. To address this issue, we propose a multi-head cross-attention module to help the model capture information from multiple perspectives of the input data. This results in richer feature representations. Our contributions are as follows:

• Our proposed baseline model for deep clustering has outperformed almost all existing deep clustering models in terms of performance metrics such as accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). Moreover, our model can be used in place of CC and can be easily applied to improved methods such as C3 and SACC, resulting in even better performance.
• Our model also shows qualitative improvement in terms of time performance, allowing it to converge quickly. In fact, as shown in Section 4.4, our model has reduced training time by 80% relative to CC.
• Our model is designed to work in a single-stage, end-to-end fashion, which allows for batch optimization and application to large-scale scenarios.

2. Related Work

In this section, we briefly review work related to unsupervised representation learning and deep clustering.

2.1. Unsupervised Representation Learning

Unsupervised representation learning is a crucial aspect of deep learning that does not require labeled data. This technique learns more discriminative feature representations by learning the intrinsic structure of data or using certain means such as data augmentation and contrastive learning. This is applied in various fields such as computer vision and natural language processing. MoCo [7] introduced a momentum-based contrastive learning method for unsupervised representation learning, which enhances the stability of contrastive learning by introducing a momentum network. SimCLR [6] is another contrastive learning-based unsupervised representation learning method that learns feature representations of data by learning the similarity between image pairs. BYOL [10] proposed a bootstrapped-based unsupervised representation learning method that learns data representations by learning the target representation of the network. SimSiam [8] is unique because it does not require the use of negative samples but rather learns feature representations through interdependencies between positive samples.

2.2. Deep Clustering

Traditional clustering methods rely on similarity or distance measures to group data, but they struggle with high-dimensional or non-linear data structures. Deep learning-based clustering methods, such as DSEC [11,12], have been developed to address these limitations. These methods transform the clustering problem into a binary discrimination problem, which enables automatic feature extraction and clustering. Contrastive learning methods, such as MoCo, SimCLR, and BYOL, have been introduced into deep clustering, leading to the development of CC [9], C3 [13], TCL [14], SPICE [5], and SACC [15]. CC combines contrastive learning and deep clustering using the instance contrast head and cluster contrast head. C3 and TCL propose new loss functions to augment positive samples and mitigate the impact of false negative pairs, respectively. SPICE designs a semantic pseudo-label-based image clustering using the idea of CC, while SACC expands the backbone network of CC.

3. Method

When we apply deep clustering techniques to analyze data, we utilize a deep clustering model denoted as $f(x;\theta )$, where $x\in {\mathbb{R}}^n$ represents the raw data, $f(x;\theta )\in {\mathbb{R}}^d$ represents the latent space representation of the data, and $\theta$ represents the model parameters. The deep clustering model comprises two main components: a feature extraction unit and a clustering unit. These can be expressed as $f(x;\theta )=g(h(x;{\theta }_1);{\theta }_2)$. The feature extraction unit is represented as $h(x;{\theta }_1)$ and ${\theta }_1$ represents parameters of the feature extraction model, while $g(h(x;{\theta }_1);{\theta }_2)$ represents the clustering unit and ${\theta }_2$ represents parameters of the cluster model. Various deep learning techniques such as Convolutional Neural Networks, Autoencoders, and Variational Autoencoders can be used to implement feature extraction and clustering units. By extracting features from the data and mapping them onto the latent space, we can obtain the latent representation of the data $f(x;\theta )$. Finally, this latent representation can be used as input to the clustering unit to obtain the clustering results of the data.

We propose a novel method that builds on the baseline CC model by improving three modules: Pair Construction Backbone (PCB), Instance-level Contrastive Head (ICH), and Cluster-level Contrastive Head (CCH), as shown in Figure 1. Data augmentation is performed in PCB to create data pairs, and the same backbone is used to obtain the feature representations of the data pairs. Contrastive learning is then performed on the matrix row space and column space in ICH and CCH, respectively. Our proposed method has three primary enhancements compared to the CC model:

• Based on the findings in Section 4.2.4, we decided not to use the data augmentation technique proposed by CC in PCB. Instead, we opted for a combination of strong and weak augmentation. Previous experiments have demonstrated that the direct application of strong augmentation can lead to suboptimal results [16]. Therefore, we used a mixed approach of strong and weak augmentation.
• After obtaining the image feature representation in PCB, we skipped the pooling operations and returned the calculated feature representation directly. According to [17], applying pooling operations directly may result in some information loss. Hence, we aimed to make the most of this information, as explained in the following point.
• In ICH, since we eliminated the pooling layer, the MLP was no longer suitable. Instead, we used a multi-head cross-attention module. Each attention head contained a spatial attention unit and a channel attention unit. We then combined the features extracted from these heads.

In the following, we will introduce the three component modules of the model.

3.1. Pair Construction Backbone

We utilized a mixed strategy of strong and weak augmentation, inspired by recent contrastive learning works [14,15], which had a positive impact on experimental results (Section 4.2.4). Our weak augmentation method is similar to SimCLR [6], while our strong augmentation method is based on RandAugment [18]. We applied random flip and random mask operations first and then randomly selected n from a set of image transformations, including autocontrast, brightness, color, contrast, equalize, identity, posterize, rotate, sharpness, shearX/Y, solarize, and translateX/Y, to achieve efficient data expansion.

To explain the process, for any data instance $x_i$ where i is a member of the set $\{1,2,\dots ,N\}$ and N represents the batch size, we perform two types of augmentation: strong ($T_s$) and weak ($T_w$). Using these, we create data pairs $x_i^a=T_s\left(x_i\right)$ and $x_i^b=T_w\left(x_i\right)$.

We use a residual network ${\mathcal{R}}_r$ [19] as the shared backbone network, similar to CC. This network will generate feature representations of the sample pairs, namely, $h_i^a={\mathcal{R}}_r(x_i^a;w_r)$ and $h_i^b={\mathcal{R}}_r(x_i^b;w_r)$, where $w_r$ represents the parameters of network ${\mathcal{R}}_r$.

3.2. Instance-Level Contrastive Head

Contrastive learning aims to create an embedding space that clusters similar samples together and separates dissimilar ones. This results in effective clustering. To achieve this, CC uses an MLP (consisting of a linear layer, a ReLU activation function, and another linear layer) in the ICH module to map data to a low-dimensional space, similar to SimCLR. According to [20], high-dimensional data contains a lot of noise and redundancy, hence MLP is used in SimCLR. However, we believe that better methods can be used to map these features to another space. Therefore, instead of the original MLP, we have adopted a multi-head cross-attention module based on [20] that explains the limitations of the pooling layer.

3.2.1. Multi-Head Cross Attention

The proposed multi-head cross-attention module comprises of multiple cross-attention heads that work in parallel. As depicted in Figure 2, each cross-attention head includes a spatial attention unit and a channel attention unit. The spatial attention unit extracts spatial features from the input features provided by the PCB; whereas, the channel attention unit extracts channel features from the output features of the spatial attention unit. These two sets of features are then merged into a single feature representation. The left part of Figure 2 shows the spatial attention unit, which is made up of four convolutions and an activation function. The convolutions are constructed in dimensions of $3\times 3$, $1\times 3$, and $3\times 1$ to capture multi-scale local features. On the other hand, the right part of the channel attention unit consists of a pooling layer, two linear layers, and an activation function. The two linear layers are used to implement an autoencoder.

Formally, let $\mathcal{H}=\{H_1,H_2,\dots H_k\}$ be the set of spatial attention heads, and $\mathcal{S}=\{s_1,s_2,$$\dots s_k\}$ be the output spatial attention feature representations, where k is the number of cross-attention heads. The j-th output feature can be represented as:

$$s_i^t=h_i^t\times H_j(h_i^t;w_s),t\in \{a,b\}$$

(1)

where $i\in \{1,2\dots N\}$ and $w_s$ denotes the parameters of the network $H_j$.

Similarly, let $\tilde{\mathcal{H}}=\{\widetilde{H_1},\widetilde{H_1},\dots \widetilde{H_k}\}$ be the set of channel attention heads, and $\mathcal{C}=\{c_1,$$c_2,\dots c_k\}$ be the final output attention feature representations. The j-th output feature can be represented as:

$$c_i^t=s_i^t\times \widetilde{H_j}(s_i^t;w_c),t\in \{a,b\}$$

(2)

where $j\in \{1,2\dots N\}$, $w_c$ denotes the parameters of the network $\widetilde{H_j}$.

3.2.2. Attention Fusion

Currently, the cross-attention heads we have obtained are not organized, so we need to direct them to focus on different areas. Firstly, we amplify the attention maps by using a log-softmax function to highlight the most important regions. Then, we introduce a partition loss to guide the cross-attention heads to concentrate on different relevant regions and avoid overlapping attention. Finally, we merge these heads into one. If $c_k$ represents the output of the k-th cross-attention head, then the log-softmax output can be expressed as $c_k^{\prime }=log\left(\frac{exp\left(c_k\right)}{{\sum }_{i=1}^{k}exp\left(c_i\right)}\right)$. A partition loss is used to maximize the variance between the feature representations produced by these parallel cross-attention heads. This loss is represented as:

$${\mathcal{L}}_{multi}=\frac{1}{NC}\sum _{i=1}^N\sum _{j=1}^{C}log(1+\frac{k}{{\sigma }_{ij}^2})$$

(3)

where C is the number of cross-attention head channels, and ${\sigma }_{ij}^2$ represents the variance of the j-th channel in the i-th sample.

For convenience, we use $g_I(·)$ to summarize the representation of the multi-head cross-attention head operation process. Then, the above process can be simplified as $z^t=g_I\left(h^t\right),t\in \{a,b\}$.

3.2.3. Instance Contrast Learning

After obtaining the multi-head cross-attention feature representations $z_i^a$ and $z_i^b$, we use cosine distance as a similarity measure:

$$s(z_i^{k1},z_j^{k2})=\frac{z_i^{k1}·z_j^{k2}}{{∥z_i^{k1}∥}_2{∥z_j^{k2}∥}_2}$$

(4)

where $i,j\in \{1,2,\dots ,N\}$, $k_1,k_2\in \{a,b\}$. To represent pairwise similarity, given the sample $z_i^a$, we treat $\{z_i^a,z_i^b\}$ as a positive pair and others as negative pairs. The loss for the positive pair of samples $\{z_i^a,z_i^b\}$ is represented as:

$${\ell }_i^a=-log\frac{exp(s(z_i^a,z_i^b)/{\mathcal{T}}_{\mathcal{I}})}{{\sum }_{j=1}^N[exp(s(z_i^a,z_j^b)/{\mathcal{T}}_{\mathcal{I}})+exp(s(z_i^a,z_j^a)/{\mathcal{T}}_{\mathcal{I}})]}$$

(5)

where ${\mathcal{T}}_{\mathcal{I}}$ is the instance-level temperature parameter. Therefore, the loss for all samples can be represented as:

$${\mathcal{L}}_{ins}=\frac{1}{2N}\sum _{i=1}^N({\ell }_i^a+{\ell }_i^b)$$

(6)

3.3. Cluster-Level Contrastive Head

In line with the concept of contrastive learning, we observe that during clustering, data samples need to be mapped to a matrix whose dimensions are equal to the number of clusters. In this matrix, each column of a data sample represents the probability of it belonging to the respective cluster. Initially, we use $z_i^a$ and $z_i^b$ as input data and then pass them through a linear layer, Relu activation function, and Softmax layer. This transforms them into a matrix whose dimensions are determined by the number of clusters. The cluster contrastive learning is performed in this embedding space.

To formalize the above-stacked network, we use $g_C(·)$ and assume that the number of clusters is M. We then map the feature matrix to the cluster contrastive subspace via $c^a=g_C\left(z^a\right)$ and $c^b=g_C\left(z^b\right)$. We use cosine distance as a similarity measure, as described in Equation (4). To represent pairwise similarity, we consider $\{c_i^a,c_i^b\}$ as a positive pair and the others as negative pairs, given the sample $c_i^a$. We can represent the loss for the positive pair samples $\{c_i^a,c_i^b\}$ as:

$$\widehat{{\ell }_i^a}=-log\frac{exp(s(c_i^a,c_i^b)/{\mathcal{T}}_{\mathcal{C}})}{{\sum }_{j=1}^N[exp(s(c_i^a,c_j^b)/{\mathcal{T}}_{\mathcal{I}})+exp(s(c_i^a,c_j^a)/{\mathcal{T}}_{\mathcal{C}})]}$$

(7)

where ${\mathcal{T}}_{\mathcal{C}}$ is the cluster-level temperature parameter. Therefore, the loss for all samples can be represented as:

$${\mathcal{L}}_{clu}=\frac{1}{2N}\sum _{i=1}^N(\widehat{{\ell }_i^a}+\widehat{{\ell }_i^b})$$

(8)

To address the issue of uneven clustering results, we introduce a loss term to the training process that incorporates the negative entropy of the cluster assignment probability. This term helps to regulate the distribution of the cluster assignment probabilities, ensuring that each cluster has a uniform representation of data samples. By doing so, the resulting cluster assignments are more evenly distributed and the effectiveness of contrastive learning is improved. This is illustrated below:

$$H\left(C\right)=-\sum _{i=1}^M[p\left(c_i^a\right)logp\left(c_i^a\right)+p\left(c_i^b\right)logp\left(c_i^b\right)]$$

(9)

where $p(·)$ represents the assignment probability of the i-th cluster, $p\left(c_i^k\right)={\sum }_{j=1}^M\frac{c_{ji}^k}{\left|\right|c^k{\left|\right|}_1},k\in \{a,b\}$. The network’s final loss is the sum of the three loss functions.

$$\mathcal{L}={\mathcal{L}}_{multi}+{\mathcal{L}}_{ins}+{\mathcal{L}}_{clu}+H\left(C\right)$$

(10)

4. Experiments

In this section, we present a series of experiments to showcase the effectiveness of the proposed method.

4.1. Experimental Configuration

In order to evaluate the proposed method, we conducted experiments on five different image datasets. A brief description of these datasets can be found in

Table 1. Summary of the dataset used for evaluation.

Dataset	Split	Samples	Classes
CIFAR-10	✔	60,000	10
CIFAR-100	✔	60,000	20
STL-10	✔	13,000	10
ImageNet-10	×	13,000	10
ImageNet-dogs	×	19,500	15

. Previous unsupervised classification methods have used both the training and test sets during training, but this approach has been criticized as not optimal. Following the recommendations of [21], we trained only on the training set of CIFAR-10, CIFAR-100, and STL-10, instead of using both sets simultaneously as in previous works, which resulted in a 0.02 or even higher increase in metric values. For ImageNet-10 and ImageNet-dogs, we used the entire dataset because there is no widely accepted data splitting scheme. Finally, we used 20 superclasses instead of 100 subclasses for CIFAR-100.

To ensure a fair comparison with CC and other works, we adopted ResNet34 as the backbone and resized all images to $224\times 224$. Our instance-level temperature parameter ${\mathcal{T}}_{\mathcal{I}}$ was set to 0.5, while the cluster-level temperature parameter ${\mathcal{T}}_{\mathcal{C}}$ was set to 1.0. The number of cross-attention heads was set to 4, and we conducted an ablation study in Section 4.2. To optimize the entire network, we used an Adam optimizer with an initial learning rate of 0.003. We trained the network for 100 epochs, after which we adjusted the learning rate to 0.0003 and continued training for another 100 epochs. The experiments were conducted on an NVIDIA GeForce RTX 2080 Ti 11G GPU, and we set the batch size to 128.

To evaluate performance, we use three common metrics in clustering: accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI) [22].

4.2. Ablation Studies

4.2.1. Number of the Cross-Attention Heads

We conducted an experiment on the ImageNet-dogs dataset to study the impact of cross-attention heads on the model’s performance. The experimental settings are mentioned in Section 4.1. According to the results shown in Figure 3, our proposed multi-head attention module outperforms the single attention module. Moreover, we found that the multi-head attention module equipped with 4 cross-attention heads showed the most significant performance gain.

4.2.2. Effect of Contrastive Learning Head

To validate the accuracy of both comparison headers, we conducted an experiment where we removed the ICH and CCH individually and optimized only one of the comparison headers. The cluster representation of the data is not directly available for the ICH when the CCH is removed. Therefore, we applied the K-Means method on the ICH results. The experimental results are presented in

Table 2. Ablation study of two contrastive learning heads.

Dataset	Contrastive Head	NMI	ACC	ARI
CIFAR-10	ICH + CCH	0.811	0.889	0.785
	ICH	0.431	0.514	0.323
	CCH	0.788	0.845	0.735
ImageNet-dogs	ICH + CCH	0.64	0.676	0.535
	ICH	0.582	0.475	0.409
	CCH	0.561	0.531	0.406

. These results indicate that the joint optimization of both comparison headers is more effective than using only ICH or CCH.

4.2.3. Effect of Multi-Crossed Attention

In order to demonstrate the effectiveness of multi-head cross-attention in ICH when compared to the original MLP, we replaced the MLP in the ICH structure of CC with multi-head cross-attention (CC(Multi)). We conducted tests on the CIFAR-10 and ImageNet-dogs datasets, using experimental settings described in Section 4.1.

Table 3. Multi-head cross-attention module validity.

Dataset	Model	NMI	ACC	ARI
CIFAR-10	CC (Multi)	0.728	0.813	0.669
	CC	0.705	0.79	0.637
ImageNet-dogs	CC (Multi)	0.559	0.592	0.425
	CC	0.445	0.429	0.274

displays the experimental results. The performance of the model improved when the MLP in the ICH structure of CC was replaced with our proposed multi-head cross-attention. For the CIFAR-10 dataset, the NMI, ACC, and ARI increased by 3.3%, 2.9%, and 5.0% respectively. Likewise, for the ImageNet-dogs dataset, the NMI, ACC, and ARI increased by 25.6%, 38.0%, and 55.1%, respectively.

In order to demonstrate the effectiveness of multi-head cross-attention in clustering tasks in a more intuitive way, we used the T-SNE visualization method to display the clustering effect of CC and CC (Multi) on the CIFAR-10 dataset. The results are shown in Figure 4. We used the multi-head cross-attention features in ICH as input data for T-SNE and used the cluster assignment predictions of CCH as labels, represented by different colors. The results indicate that, compared to CC which only uses MLP, the inter-class distance is smaller, the intra-class distance is larger, and the cluster assignment is more balanced after using multi-head cross-attention. This proves that our proposed method is more effective in improving the quality of clustering.

Multi-head attention is more effective than traditional MLP in sharing information among different attention heads, which enhances the feature representation ability. By using multi-head cross-attention, the model can learn more precise and comprehensive feature representations, leading to improved model performance.

4.2.4. Effect of Data Augmentation

As described in Section 3, we presented the data augmentation methods used for the model. To evaluate the effectiveness of these methods, we performed data augmentation ablation studies on CIFAR-10 and ImageNet-dogs datasets. We compared the model’s performance using mixed strategies of strong and weak augmentation, weak augmentation only, and no data augmentation.

Table 4. Ablation studies with strong and weak augmentation methods.

Dataset	Augumentation	NMI	ACC	ARI
CIFAR-10	$T_s\left(x\right)+T_w\left(x\right)$	0.811	0.889	0.785
	$T_w\left(x\right)+T_w\left(x\right)$	0.728	0.813	0.669
	$x+x$	0.064	0.197	0.04
	$T_w\left(x\right)+T_w\left(x\right)$ (CC)	0.705	0.79	0.637
ImageNet-dogs	$T_s\left(x\right)+T_w\left(x\right)$	0.64	0.676	0.535
	$T_w\left(x\right)+T_w\left(x\right)$	0.559	0.592	0.425
	$x+x$	0.117	0.193	0.046
	$T_w\left(x\right)+T_w\left(x\right)$ (CC)	0.445	0.429	0.274

displays the performance improvements due to data augmentation. When we applied the same weak augmentation strategy as CC, the model’s performance increased by 3.9% on CIFAR-10, and by 39.3% on ImageNet-dogs. Using a mixed strategy of strong and weak augmentation further improved the performance.

4.3. Comparisons with State of the Arts

We evaluated our method on five public datasets and compared it with 15 representative clustering methods, including K-means [23], SC [24], DAE [25], DCGAN [26], VAE [27], DEC [28], DAC [29], DCCM [30], PICA [31], CC [9], IDFD [32], C3 [13], SPICE [5], TCL [14], SACC [15].

Table 5. The clustering performance on the five baseline datasets. The first and second best results are indicated using bold and underlining, respectively.

Dataset	CIFAR-10			CIFAR-100			STL-10			ImageNet-10			ImageNet-Dogs
Metrics	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI	NMI	ACC	ARI
K-means	0.087	0.229	0.049	0.084	0.13	0.028	0.125	0.192	0.061	0.119	0.241	0.057	0.055	0.105	0.02
SC	0.103	0.247	0.085	0.09	0.136	0.022	0.098	0.159	0.048	0.151	0.274	0.076	0.038	0.111	0.013
DAE	0.251	0.297	0.163	0.111	0.151	0.046	0.224	0.302	0.152	0.206	0.304	0.138	0.104	0.19	0.078
DCGAN	0.265	0.315	0.176	0.12	0.151	0.045	0.21	0.298	0.139	0.225	0.346	0.157	0.121	0.174	0.078
VAE	0.245	0.291	0.167	0.108	0.152	0.04	0.2	0.282	0.146	0.193	0.334	0.168	0.107	0.179	0.079
DEC	0.257	0.301	0.161	0.136	0.185	0.05	0.276	0.359	0.186	0.282	0.381	0.203	0.122	0.195	0.079
DAC	0.396	0.522	0.306	0.185	0.238	0.088	0.366	0.47	0.257	0.394	0.527	0.302	0.219	0.275	0.111
DCCM	0.496	0.623	0.408	0.285	0.327	0.173	0.376	0.482	0.262	0.608	0.71	0.555	0.321	0.383	0.182
PICA	0.591	0.696	0.512	0.31	0.337	0.171	0.611	0.713	0.531	0.802	0.87	0.761	0.352	0.352	0.201
CC	0.705	0.79	0.637	0.431	0.429	0.266	0.764	0.85	0.726	0.859	0.893	0.822	0.445	0.429	0.274
IDFD	0.711	0.815	0.663	0.426	0.425	0.264	0.643	0.756	0.575	-	-	-	-	-	-
C3	0.748	0.838	0.707	0.434	0.451	0.275	-	-	-	0.905	0.942	0.861	0.448	0.434	0.28
SPICE	0.734	0.838	0.705	0.448	0.468	0.294	0.817	0.908	0.812	0.828	0.921	0.836	0.572	0.646	0.479
TCL	0.819	0.887	0.78	0.529	0.531	0.357	0.799	0.868	0.757	0.875	0.895	0.837	0.623	0.644	0.516
SACC	0.765	0.851	0.724	0.448	0.443	0.282	0.691	0.759	0.626	0.877	0.905	0.843	0.455	0.437	0.285
Multi-CC	0.811	0.889	0.785	0.527	0.498	0.335	0.731	0.818	0.682	0.924	0.964	0.93	0.64	0.676	0.535
Multi-CC(SACC)	0.818	0.898	0.799	0.542	0.518	0.354	0.739	0.822	0.686	0.931	0.972	0.939	0.652	0.684	0.547
Multi-CC(C3)	0.81	0.884	0.776	0.538	0.506	0.35	0.735	0.837	0.695	0.94	0.976	0.948	0.713	0.732	0.625

shows the results of our proposed method on the baseline datasets.

4.3.1. Metrics Performance Analysis

After CC introduced its learning methods to the deep clustering work, most of the optimal models in the field of deep clustering have used CC as a baseline. These models, which include CC, such as C3, SPICE, TCL, and SACC, utilize both the training and testing sets during the training process. As explained in Section 4.1, this practice leads to better results. Even though this could be disadvantageous for us, we still separated the CIFAR-10, CIFAR-100, and STL-10 datasets, which already have recognized splitting schemes.

Although Multi-CC is considered to be a baseline, it has produced the best outcomes on CIFAR-10, ImageNet-10, and ImageNet-dogs datasets. It ranked second on CIFAR-100 after TCL and after CC, SPICE, and TCL on STL-10 datasets. The improvement of Multi-CC over CC is significant, particularly on the ImageNet-dogs dataset, where there was an increase in NMI, ACC, and ARI by 43.8%, 57.6%, and 95.3%, respectively. We also saw an increase in performance by 17.9%, 21.4%, and 10.0% for CIFAR-10, CIFAR-100, and ImageNet-10 datasets, respectively.

During our study, we noticed that the STL-10 dataset results of Multi-CC were lower than those of CC. We determined that this is because of the joint training and testing set training method used, as well as the use of unlabeled data. We confirmed our hypothesis and decided to retest the CC model by performing data segmentation on STL-10. We used the same data as Multi-CC, without any additional unlabeled data. The training set had 5000 images, and the test set had 8000 images. The results are presented in

Table 6. Retesting CC with STL-10.

Dataset	Model	NMI	ACC	ARI
STL-10	CC	0.401	0.499	0.255
	Multi-CC	0.731	0.818	0.682

. Our conjecture was validated when the CC model was trained without the joint training and test set strategy and without introducing additional unlabeled data, resulting in a significant decrease in model performance. This also confirms that the joint training strategy mentioned in SCAN yields higher performance.

4.3.2. Metrics Performance Analysis with Other Options

To demonstrate the effectiveness and ease of use of our scheme, we applied the C3 and SACC improvements on CC to Multi-CC. The summary of C3 and SACC can be found in the Appendix A and Appendix B. The results are shown in Table 4, where the improved models are marked as Multi-CC(C3) and Multi-CC(SACC), respectively. These methods performed better than Multi-CC, indicating the superiority of our approach.

Multi-CC(C3) showed a significant improvement in NMI, ACC, and ARI by 11.4%, 8.3%, and 16.8%, respectively, on the ImageNet-dogs dataset compared to Multi-CC. Furthermore, compared to C3 implemented based on CC, Multi-CC(C3) increased NMI, ACC, and ARI by 59.2%, 68.7%, and 123.2%, respectively, on the ImageNet-dogs dataset. It also showed an average of 10% improvement on other datasets.

Multi-CC(SACC) also showed improvements compared to SACC implemented based on CC. For instance, on the ImageNet-dogs dataset, NMI, ACC, and ARI increased by 41.1%, 56.5%, and 91.9%, respectively. Similarly, there were also improvements on other datasets. These results not only confirm the effectiveness of the Multi-CC method but also show its flexibility and usability under different improvement schemes.

4.4. Time Performance Analysis

It is worth noting that Multi-CC outperforms CC in terms of time efficiency. According to CC, it takes 1000 epochs to train all datasets, which uses up a lot of GPU resources. However, Multi-CC is 80% faster and only requires 200 epochs to achieve a nearly optimal result, which is better than CC. We conducted a comparison between Multi-CC and CC on the ImageNet-dogs dataset and evaluated their performance in terms of NMI, ACC, and ARI, using only the first 200 epochs of results for CC. The results demonstrate that Multi-CC is capable of achieving a better result quickly and fine-tuning the model will lead to even better results. The comparison results are presented in Figure 5a–c.

4.5. Contributions to the Field of Unsupervised Learning

In the case of datasets, such as CIFAR-10, ImageNet-10, etc., SimCLR requires training in stages if classification is needed. However, our focus for this paper is not on classification; thus, we perform the K-Means algorithm directly on the features extracted by SimCLR to simplify the process. We also modified our proposed scheme for SimCLR, which we call Multi-SimCLR. To maintain fairness, both schemes make use of ResNet34 as the backbone and perform K-Means to compute the clustering accuracy after 200 epochs of training on the CIFAR-10 and STL-10 datasets.

Table 7. Effectiveness of the Multi-Cross Attention Module on SimCLR.

Dataset	Model	ACC
CIFAR-10	SimCLR	0.418
	Multi-SimCLR	0.636
STL-10	SimCLR	0.431
	Multi-SimCLR	0.557

displays the results that demonstrate how our method can efficiently incorporate SimCLR and achieve better performance. Our method has shown a significant improvement in accuracy on the CIFAR-10 and STL-10 datasets, with an increase of 52.2% and 29.2%, respectively, compared to SimCLR. These results demonstrate the potential of our method to be easily generalized to the whole comparison learning framework.

5. Conclusions

We have made significant advancements in the field of deep clustering by improving the baseline CC and introducing a new technique called Multi-Head Cross-Attention Contrastive Clustering (Multi-CC). Despite considering it as a baseline, Multi-CC has achieved the best results on datasets such as CIFAR-10, ImageNet-10, and ImageNet-dogs. Notably, we have not jointly trained the training set and test set, which was conducted in previous works for these tests. Our model can effectively replace CC-based models such as C3, SACC, TCL, SPICE, etc. We have also applied the improvement methods from these models to Multi-CC, resulting in even better performance. Multi-CC has significantly reduced the time required for deep clustering by 80% compared to CC. Hence, we aspire to build upon our model as a foundation for making improvements to deep clustering models. However, for datasets such as Tiny-ImageNet and CIFAR-100, which contain 200 classes, there still isn ot a suitable method in the field of deep clustering, leaving it as an open problem.

Furthermore, we made a tentative endeavor to incorporate our approach into SimCLR, which still exerted a certain influence on SimCLR. This attempt has opened up a new research direction for applying similar multi-head cross-attention mechanisms in unsupervised learning in the future, thereby improving model performance and efficiency. In the next phase, we will shift our focus towards exploring the feasibility of our approach in a broader scope of unsupervised learning. Additionally, we will also delve into the exploration of super clustering.