Three-Way Ensemble Clustering Based on Sample’s Perturbation Theory

Abstract

The complexity of the data type and distribution leads to the increase in uncertainty in the relationship between samples, which brings challenges to effectively mining the potential cluster structure of data. Ensemble clustering aims to obtain a unified cluster division by fusing multiple different base clustering results. This paper proposes a three-way ensemble clustering algorithm based on sample’s perturbation theory to solve the problem of inaccurate decision making caused by inaccurate information or insufficient data. The algorithm first combines the natural nearest neighbor algorithm to generate two sets of perturbed data sets, randomly extracts the feature subsets of the samples, and uses the traditional clustering algorithm to obtain different base clusters. The sample’s stability is obtained by using the co-association matrix and determinacy function, and then the samples can be divided into a stable region and unstable region according to a threshold for the sample’s stability. The stable region consists of high-stability samples and is divided into the core region of each cluster using the K-means algorithm. The unstable region consists of low-stability samples and is assigned to the fringe regions of each cluster. Therefore, a three-way clustering result is formed. The experimental results show that the proposed algorithm in this paper can obtain better clustering results compared with other clustering ensemble algorithms on the UCI Machine Learning Repository data set, and can effectively reveal the clustering structure.

1. Introduction

Clustering is a powerful data analysis technology which is widely used in different fields such as information granulation [1,2,3], information fusion [4,5,6], attribute reduction [7,8,9,10], feature selection [11,12,13], image analysis [14,15,16] and other fields. The purpose of clustering is to divide the samples with high similarity into one cluster and the samples with low similarity into different clusters [17]. There are many clustering methods, such as DPC [18], DBSCAN [19], spectral clustering [20]. In addition to those aforementioned clustering algorithms, the non-iterative ANN can also be used to solve the clustering problem [21,22,23].

The aforementioned clustering algorithms are all hard clustering algorithms, which means that the objects which are in the set definitely belong to this cluster and the objects which are not in the set do not definitely belong to this cluster. The hard clustering algorithm cannot solve the problem of inaccurate decision-making caused by inaccurate information or insufficient data. In recent years, a new clustering algorithm based on three-way decision [24,25] has been proposed, named three-way clustering [26]. Three-way decision is an extension of two-way decision, in which a definite decision is given to the objects with definite information and a deferment decision is adopted when the object’s information is insufficient to avoid the decision risk. Three-way clustering divides the data set into the core region, fringe region and trivial region. By further processing the data objects in the fringe region, we can clearly understand their impact on the cluster.

Since the idea of three-way clustering was put forward, it has been widely concerned. Wang et al. [27] proposed a CE3 framework for three-way clustering and a three-way clustering method based on the dynamic domain. Zhang [28] proposed a three-way c-means algorithm, which proposed a three-way weight according to the inherent characteristics of each data point, making the weight more accurate. Then, a three-way allocation is proposed to allocate data points to the cluster. Liu [29] used gray relational clustering and three-way decision making, and constructed a three-way gray relational clustering method based on the principle of complementary advantages.

As the amount of big data increases, it is difficult to identify all different data structures from a large number of complex data sets by a single clustering algorithm. In order to solve this problem and improve the stability and robustness of the algorithm, ensemble clustering [30] is proposed. The concept of clustering ensemble was proposed by Strehl and Ghosh [31] in 2003. The ensemble method can be integrated by setting different parameters, different clustering algorithms, different representations of features and weak clustering. By constructing the matrix of the ensemble results, analyzing the differences, looking for appropriate algorithms to analyze the ensemble results, and finally obtain better clustering results. The clustering ensemble can combine multiple partitions of a data object set into a unified clustering result. Cluster ensemble is mainly divided into two stages: the cluster member generation stage and consistency function design stage. The design of the consistency function has a great impact on the quality of clustering results. At present, the consistency function is mainly divided into the voting method [30], hypergraph method [31], evidence accumulation method [32] and method based on mutual information theory [33]. Liu et al. [34] constructed a new algorithm model by using the clustering integration algorithm of the neural network algorithm to gather the majority votes. Yu et al. [35] developed a framework of three-way ensemble clustering based on Spark and proposed a consensus clustering algorithm based on cluster units.

In this paper, by setting different parameters for the same algorithm, a group of base clustering results with differences are obtained. In a given data set, some samples belong to the same category in a set of base clusters, while they may belong to other categories in other base clusters. This phenomenon is characterized by the tendency of samples to change classification. In order to quantify the trend of sample change, this trend is called the measure of a sample’s stability [36]. The algorithm first combines the natural nearest neighbor algorithm to generate two sets of perturbed data sets, randomly extracts the feature subsets of the samples, and uses the traditional clustering algorithm to obtain different base clusters. The sample’s stability is obtained by using the co-association matrix and determinacy function, and then the samples can be divided into a stable region and unstable region according to a threshold for the sample’s stability. The stable region consists of high-stability samples and is divided into the core region of each cluster using the K-means algorithm [37]. The unstable region consists of low-stability samples and is assigned to the fringe regions of each cluster. Therefore, a three-way clustering result is formed.

The paper has two contributions as follows:

(1)

Compared with other ensemble clustering algorithms, this paper proposes a sample’s perturbation theory based on the natural nearest neighbor algorithm, which makes full use of the characteristics of samples in the process of generating base clustering.

(2)

This paper proposes a three-way ensemble clustering algorithm. A hard clustering algorithm cannot solve the problem of inaccurate decision-making caused by inaccurate information or insufficient data, while the proposed algorithm can avoid decision-making.

The main contents of this paper are organized as follows. We briefly introduce related concepts such as three-way clustering, natural nearest neighbor, the sample’s perturbation theory, and ensemble clustering. In Section 3, we mainly propose a three-way ensemble clustering algorithm based on sample’s perturbation theory. In Section 4, we analyze the performance of the proposed three-way ensemble clustering algorithm. Conclusions and future work are given in Section 5.

2. Preliminaries

This section mainly introduces the background of the algorithm in this paper, including three-way clustering, natural nearest neighbor, sample’s perturbation theory, and ensemble clustering.

2.1. Three-Way Clustering

The three-way decision [38,39,40] was proposed by adding one more uncommitted option than the traditional two-way decision theory. In traditional binary-decision model, one only has two decisions to select: acceptance or rejection decision. When current information is sufficient, we can easily make an acceptance decision or a rejection decision. However, when information is limited or insufficient, making an acceptance or a rejection decision, regardless of which is made, it may cause inaccuracy in the decision. A three-way decision adds a deferment decision (or non-commitment decision) to address the problem of inaccurate decision making. The main idea of three-way decision is to divide a universe into three disjoint regions and make three strategies for different regions that represent three levels of uncertainty.

Recently, by integrating three-way decision with clustering, a new type of clustering algorithm [41,42] was proposed, named three-way clustering. Three-way clustering uses the core region and fringe region to represent a cluster. These two sets divide the universe into three parts $Co\left(C_i\right),Fr\left(C_i\right)$ and $Tr\left(C_i\right)$, which classify the three types of relationships between the objects and cluster, namely objects belonging to the cluster; objects not belonging to the cluster; and objects partially belonging to the cluster. For the samples in $Co\left(C_i\right)$, they definitely belong to the cluster $\left(C_i\right)$ and have higher within-class similarities. For the samples in $Fr\left(C_i\right)$, they may belong to the cluster $\left(C_i\right)$ and have lower similarity to the core samples. For the samples in $Tr\left(C_i\right)$, they do not definitely belong to the cluster $C_i$. In contrast to the single set for representing the cluster, three-way clustering uses $Co\left(C_i\right),Fr\left(C_i\right)$ to represent one cluster. Such representation can clearly distinguish the core and fringe parts of a cluster, such as a concept having an intension and extension.

Given a set of data objects $U=\left\{x_1,x_2,\cdots ,x_n\right\}$, $C=\left\{c_1,c_2,\cdots ,c_k\right\}$ is a finite set of clusters, and U is divided into k classes. The idea of three-way clustering is to use a pair of sets to represent each cluster. This pair of sets consists of core region (Co), fringe region (Fr), and trivial fringe (Tr) [43]. Then, the three-way cluster is expressed as,

$$\begin{array}{c}\hfill C_i=\left(Co\left(C_i\right),Fr\left(C_i\right)\right).\end{array}$$

(1)

Three-way clustering results of data object set U are expressed as,

$$\begin{array}{c}\hfill C=\left\{\left(Co\left(C_1\right),Fr\left(C_1\right)\right),\left(Co\left(C_2\right),Fr\left(C_2\right)\right),\cdots ,\left(Co\left(C_k\right),Fr\left(C_k\right)\right)\right\}.\end{array}$$

(2)

According to the definition of clustering results, $Co\left(C_i\right)$ and $Fr\left(C_i\right)$ must meet the three following conditions,

$$\begin{array}{c}\hfill C\mathrm{o}\left(C_i\right)\ne \varphi ,i=1,2,\cdots ,k;\end{array}$$

(3)

$$\begin{array}{c}\hfill \bigcup _{i=1}^k(Co\left(C_i\right)\cup Fr\left(C_i\right))=U;\end{array}$$

(4)

$$\begin{array}{c}\hfill Co\left(C_i\right)\cap Co\left(C_j\right)=\varphi ,i\ne j.\end{array}$$

(5)

Among them, condition (1) indicates that any cluster is non empty, condition (2) indicates that the sample $x_i\in U$ belongs to at least one cluster, and condition (3) indicates that the core region of the cluster is pairwise disjoint.

Recently, lots of three-way clustering approaches have been developed. For example, Afridi et al. [44] proposed novel three-way clustering using a game-theoretic set to handle missing data. Wang et al. [45] improved the K-means algorithm by integrating the three-way decision and then developed a three-way k-means method; Yu et al. [46] presented three-way density peak clustering which used evidence theory. Jia et al. [47] developed an automatic three-way clustering approach by combining the proposed threshold selection based on the roughness degree using the sample’s similarity and the cluster number selection method. In addition to the papers listed above, there are other contributions that enrich the theories and models of three-way clustering [48,49,50,51,52].

2.2. Natural Nearest Neighbor

In the study of the properties of data structure, the concept of the nearest neighbor is proposed, and the commonly used nearest neighbor methods are K-neighborhood and $\epsilon$-neighborhood [53] proposed by Stevens. The natural nearest neighbor is a new neighbor definition proposed by Zou [54] in 2011. Compared with the K-nearest neighbor and $\epsilon$-nearest neighbor, its biggest difference is that it is generated by the data structure itself without any parameters. Its main idea is that if the data point a appears in the r-neighborhood of b, then point b is the natural nearest neighbor of point a.

Definition 1.

Natural nearest neighbor [55]. For data object X, if there is a data object Y which has X as its neighbor, and the most outlier data object Z in the data set has a data object that thinks Z as its neighbor, the data object Y is called the natural nearest neighbor of data object X. Furthermore, the outlier data point Z is the last point in the data set that first appeared in the neighborhood of other points.

Definition 2.

Natural characteristic value [55]. The natural characteristic value of the data set is the minimum r value that makes any data point x be included by the r-neighborhood of another data point $y(y\ne x)$. Its mathematical definition is as follows:

$$\begin{array}{c}\hfill su{p}_k=min\left\{\left.r\right|\forall x\in X,\exists y\in X,y\ne x,s.t.x\in N{N}_r\left(y\right)\right\}\end{array}$$

(6)

where $N{N}_r\left(y\right)$ represents the r-th nearest neighbor of point y, and $su{p}_k$ is also known as the average number of natural neighbors.

2.3. Sample’s Perturbation Theory

We first generate the perturbed data sets by using the natural nearest neighbor. Given a data set $U=\{v_1,v_2,\cdots ,v_n\}$, the natural nearest neighbor of each element can be obtained by Algorithm 1. For an element v, we can achieve two new elements by the following formulas:

$$\begin{array}{c}\hfill v_1=v+\alpha \times {std}_v\end{array}$$

(7)

$$\begin{array}{c}\hfill v_2=v-\alpha \times {std}_v\end{array}$$

(8)

where $\alpha$ is a given parameter, and ${std}_v$ is the standard deviation of the natural nearest neighbor of the element v. Thus, the processes of generating perturbed data sets are as shown in Algorithm 2.

Algorithm 1: Natural nearest neighbor algorithm.

Algorithm 2: Generation of perturbed data sets.

2.4. Ensemble Clustering

In cluster analysis, different clustering algorithms will lead to different clustering results. As we all know, a single clustering algorithm cannot always achieve a good clustering result, especially when the data set has complex structures, indistinguishable boundaries, non-spherical distribution, and high dimensionality.

The ensemble clustering algorithm [31,32] uses the consistency function to effectively combine the base clusters, and integrates on the basis of the base clusters to achieve a clustering result with high performance. Ensemble clustering has two main steps, namely base cluster generation and consistency integration. In the base cluster generation stage, different clustering algorithms or the same clustering algorithm under different parameters are used to generate multiple different base clustering results. In the consistency ensemble stage, multiple different base clustering results are merged into a same cluster division by designing a consistency function. That is, a given data set $X=\left\{x_1,x_2,\cdots ,x_n\right\}$, where $x_i={\left\{x_{i1},x_{i2},\cdots ,x_{id}\right\}}^T\in R^d$, d is the dimension of the attribute of $x_i$, and n is the number of samples. Aa clustering algorithm is used to divide the data set X into M base partitions, $\pi =\left\{{\pi }_1,{\pi }_2,\cdots ,{\pi }_L\right\}$ represents a set of base partitions, where ${\pi }_l=\left\{C_1^l,C_2^l,\cdots ,C_k^l\right\}$ is the m-th partition in the data set X. $C_j^l$ represents the j-th cluster in ${\pi }_l$, and k is the number of clusters in the ${\pi }_l$ partition. The clustering ensemble is to search a new compartmentalization ${\pi }^{\#}=\left\{C_1^{\#},C_2^{\#},\cdots ,C_k^{\#}\right\}$, which is the result of consistent clustering. The process of ensemble clustering is shown in Figure 1.

3. Processes of Algorithms

Assuming the data sample $U=\left\{v_1,v_2,\cdots ,v_n\right\}$, a group of clustering results $\Pi =\left\{C_1,C_2,\cdots ,C_M\right\}$ is obtained by setting different clustering algorithm parameters for a clustering ensemble. Then, the relation matrix [36] is constructed, in which the relationship between any two points is calculated as:

$$\begin{array}{c}\hfill p_{ij}=\frac{1}{m}\sum _{m=1}^L\prod \left(C_m\left(v_i\right),C_m\left(v_j\right)\right)\end{array}$$

(9)

where M represents different clustering results; $v_i$ and $v_j$ represent two samples; and $C_m\left(v_i\right)$ represents the cluster number of point $v_i$ in the l-th clustering result. Of which:

$$\begin{array}{c}\hfill \prod \left(C_m\left(v_i\right),C_m\left(v_j\right)\right)=\left\{{}_{0,C_m\left(v_i\right)\ne C_m\left(v_j\right)}^{1,C_m\left(v_i\right)=C_m\left(v_j\right)}\right.\end{array}$$

(10)

A linear method is used to find the stability. First define the function f with respect to the variables p, t, where $p\in [0,1],t\in [0,1]$.

(1) If $p<t$, $f^{\prime }\left(p\right)<0$; if $p>t$, $f^{\prime }\left(p\right)>0$;

(2) If $p_i<t<p_j$, and $\frac{t-p_i}{p_j-t}=\frac{t}{1-t}$, then $f\left(p_i\right)=f\left(p_j\right)$.

Suppose a data set contains n samples. Based on this function f, for each point $v_i$, the stability [36] is defined as follows:

$$\begin{array}{c}\hfill s\left(v_i\right)=\frac{1}{n}\sum _{j=1}^{n}f\left(p_{ij}\right)\end{array}$$

(11)

A set of the core region and fringe region is usually used to represent each cluster of a three-way clustering result. In this section, we propose a three-way clustering algorithm based on the natural nearest neighbor and samples’ stability. Then, the proposed algorithm randomly extracts the feature subset of the sample and uses the K-means algorithm to generate multiple different base clusters. Algorithm 3 shows the steps of generating the base clustering results.

Algorithm 3: Base cluster generation.

The second step is that we compute the sample’s stability by using multiple clustering results and divide the data set into two regions according to a threshold t for stability, where t is obtained by the Otus algorithm [56].

$$\begin{array}{c}\hfill O=\left\{i\right|v_i^M>t,i=1,2,\cdots ,n\}\end{array}$$

(12)

$$\begin{array}{c}\hfill H=\left\{i\right|v_i^M\le t,i=1,2,\cdots ,n\}\end{array}$$

(13)

where the elements of O have higher stability while others of H have lower stability.

Determining how a three-way clustering result can be obtained the final problem to solve. In this paper, we take different strategies to handle the different regions. For the stable data set O, we may get the clusters $C_i,(i=1,2,\cdots ,k)$ of the elements of stable set O by K-means algorithm, which can be considered as the core regions of a three-way clustering result, namely $Co\left(C_i\right)$. For each element v of the unstable data set U and random k clustering centers $x_1,x_2,\cdots ,x_k$ of O, the minimum distance from element v to k cluster centers is $d(v,x_i)=\underset{1\le j\le k}{min}d(v,x_j)$, where $d(v,x_j)$ is the Euclidean distance between v and $x_j$. A parameter p is given, then we can achieve the set $T=\{j:d(v,x_j)-d(v,x_i)\le p\wedge j\ne i\}$, from which there are two possible situations:

(1)

If $T=\varnothing$, then $v\in Fr\left(C_i\right)$;

(2)

If $T\ne \varnothing$, then $v\in Fr\left(C_i\right)$ and $v\in Fr\left(C_j\right)$.

A three-way ensemble clustering can be naturally formed. Algorithm 4 shows the all steps of the three-way ensemble clustering based on sample’s perturbation theory.

Algorithm 4: Three-way Ensemble Clustering Based on Sample’s Perturbation Theory(3WESP).

4. Experimental Analyses

We mainly verify the performances of our proposed 3WESP algorithm in this section. In the base cluster generation, we use the K-means algorithm to generate diverse base clusters by selecting different percentages of feature subsets such as 50%, 60%, 70%, 80% and 90%. We demonstrate and explore the performances of the proposed algorithm and other clustering algorithms on 12 UCI data sets. We discuss the influence of selecting different percentages of feature subsets on the proposed algorithm.

4.1. Evaluation Indices

In the evaluation of clustering, we compare the proposed algorithm with other clustering algorithms by calculating several cluster evaluation indices such as $AMI$ [57], $NMI$ [31], $ARI$ [57], and $ACC$ [45]. For the cluster evaluation indices used in this paper, a good clustering result should have a higher value.

1. Adjusted Mutual Information ($AMI$)

The AMI index is proposed based on information theory. The larger the AMI value, the more consistent the result of the clustering algorithm with the real situation. AMI is calculated as follows:

$$AMI(U,V)=\frac{MI(U,V)-E\left\{MI\right(U,V\left)\right\}}{max\left\{H\right(U),H(V\left)\right\}-E\left\{MI\right(U,V\left)\right\}}$$

(14)

where $E\left\{MI\right(U,V\left)\right\}$ is the expectation of mutual information $MI(U,V)$, $H\left(U\right)$ is the information entropy, and its calculation formula is:

$$H\left(U\right)=-\sum _{i=1}^R{p}_{i}log{p}_i$$

(15)

2. Normalized Mutual Information ($NMI$)

$$NMI=\frac{I(X,Y)}{\sqrt{H\left(X\right)H\left(Y\right)}}$$

(16)

where X is the test label, and Y is real label. $H\left(X\right)$ and $H\left(Y\right)$ represent the entropy of X and Y, respectively. $I(X,Y)$ is the mutual information between X and Y.

3. Adjusted Rand Index ($ARI$)

$$ARI=\frac{2(TP·TN-FN·FP)}{(TP+FN)(FN+TN)+(TP+FP)(FP+TN)}$$

(17)

where $TP$ is the number of data points in a pair that belong to the same cluster in both real and experimental situations; $FN$ is the number of data points in a pair that belong to the same cluster in real but not in experimental situations; $FP$ is the number of data points in a pair that belong to the same cluster in experimental but not in real situations; $TN$ is the number of data points in a pair that do not belong to the same cluster in both real and experimental situations.

4. Accuracy ($ACC$)

$$ACC=\frac{1}{N}\sum _{i=1}^k{n}_i$$

(18)

where N is the total number of elements, $n_i$ is the number of elements which are correctly divided into the corresponding cluster i, and k is cluster number. $ACC$ represents the ratio between the number of correctly partitioned elements and the total number.

4.2. Performances of 3WESP

We employ 12 UCI [58] data sets to show the performance of the proposed algorithm.

Table 1. A description of UCI data sets.

ID	Data Sets	Samples	Attributes	Classes
1	Cardiocotography	2126	21	10
2	Congressional voting	435	16	2
3	Dermatology	366	34	6
4	Forrest	523	27	4
5	Iris	150	4	3
6	Musk	476	166	2
7	Parkinson	1208	26	2
8	Seeds	210	7	3
9	Segmentation	2310	19	7
10	Waveform	5000	21	3
11	Wdbc	569	30	2
12	Wine	178	13	3

presents the basic information of these used data sets. For a data set, we can obtain a three-way clustering result through Algorithm 4, where in the base cluster generation process, we use the K-means algorithm to generate diverse base clusters by selecting different percentages of feature subsets such as 50%, 60%, 70%, 80% and 90%.

Cluster evaluation indices are used to show the performances of a hard clustering result. While the 3WESP($\alpha =3,p=0.8$) algorithm is a soft clustering algorithm which cannot directly use these cluster evaluation indices, in this paper, we therefore only use all core regions to represent the clustering result when we calculate the $AMI$, $ARI$, $NMI$ and $ACC$ value. By running 50 times on all data sets, we can obtain the average $AMI$, $ARI$, $NMI$ and $ACC$ value where the ensemble size is 60. The performances of K-means, Voting [30] and CSPA [31] are also presented in

Table 2. Experimental results of $AMI$ on UCI data sets.

ID	Data Sets	3WESP	K-Means	Voting	CSPA
1	Cardiotocography	0.3710	0.3404	0.2227	0.2367
2	Congressional voting	0.6577	0.6531	0.6531	0.6321
3	Dermatology	0.8639	0.7811	0.4815	0.4965
4	Forrest	0.6565	0.5418	0.4778	0.5003
5	Iris	0.8231	0.7387	0.7387	0.6895
6	Musk	0.0519	0.0098	0.0154	0.0125
7	Parkinson	0.0183	0.0133	0.0133	0.0154
8	Seeds	0.6655	0.3412	0.5263	0.6605
9	Segmentation	0.5327	0.5245	0.3275	0.5194
10	Waveform	0.3650	0.3640	0.1535	0.3102
11	Wdbc	0.6242	0.6126	0.6024	0.6143
12	Wine	0.8851	0.7893	0.5683	0.8131

,

Table 3. Experimental results of $ARI$ on UCI data sets.

ID	Data Sets	3WESP	K-Means	Voting	CSPA
1	Cardiotocography	0.1760	0.1750	0.1245	0.1236
2	Congressional voting	0.7064	0.6612	0.6602	0.5247
3	Dermatology	0.7056	0.6426	0.2377	0.6448
4	Forrest	0.6878	0.5029	0.4165	0.4372
5	Iris	0.8017	0.7163	0.7234	0.6652
6	Musk	0.0427	0.0023	0.0057	0.0034
7	Parkinson	0.0243	0.0204	0.0302	0.0236
8	Seeds	0.7008	0.3180	0.4256	0.7004
9	Segmentation	0.4393	0.4158	0.2135	0.3928
10	Waveform	0.2555	0.2536	0.0943	0.2530
11	Wdbc	0.7589	0.7302	0.7203	0.7002
12	Wine	0.9068	0.7842	0.4625	0.7954

,

Table 4. Experimental results of $NMI$ on UCI data sets.

ID	Data Sets	3WESP	K-Means	Voting	CSPA
1	Cardiotocography	0.3601	0.3364	0.2302	0.3250
2	Congressional voting	0.7429	0.7194	0.7034	0.4558
3	Dermatology	0.8729	0.6837	0.4934	0.8005
4	Forrest	0.6995	0.5448	0.4817	0.4958
5	Iris	0.8319	0.7419	0.7235	0.6934
6	Musk	0.0593	0.0114	0.0170	0.0133
7	Parkinson	0.0292	0.0139	0.0135	0.0158
8	Seeds	0.6900	0.3488	0.5286	0.6679
9	Segmentation	0.7193	0.5845	0.3597	0.4958
10	Waveform	0.3653	0.3642	0.1537	0.3649
11	Wdbc	0.6456	0.6231	0.6140	0.6237
12	Wine	0.8873	0.7915	0.5712	0.8243

and

Table 5. Experimental results of $ACC$ on UCI data sets.

ID	Data Sets	3WESP	K-Means	Voting	CSPA
1	Cardiotocography	0.4831	0.4026	0.4210	0.3833
2	Congressional voting	0.9100	0.4706	0.4567	0.7625
3	Dermatology	0.8769	0.8126	0.6913	0.7366
4	Forrest	0.8448	0.7839	0.8375	0.7086
5	Iris	0.9268	0.8867	0.8767	0.8200
6	Musk	0.6820	0.6534	0.6639	0.5412
7	Parkinson	0.5820	0.5728	0.5678	0.5781
8	Seeds	0.8950	0.8857	0.8905	0.8895
9	Segmentation	0.7778	0.5905	0.4810	0.6285
10	Waveform	0.5401	0.5530	0.5178	0.5014
11	Wdbc	0.9351	0.9279	0.9256	0.9079
12	Wine	0.9697	0.9270	0.9071	0.9352

. The best performance for each data set is highlighted in bold.

It can be seen from the data in the tables that the algorithm in this paper has advantages in these four evaluation indices on most data sets. For example, on the Wine data set, the four index results of the algorithm in this paper are 0.8851, 0.9068, 0.8873 and 0.9697, respectively. The effects of other algorithms are obviously slightly worse than that of this algorithm.

4.3. Experimental Results for Selected Feature Percentage

In order to analyze the influence of different percentages of features on the results of the clustering ensemble, the following experiments were carried out. On 12 different UCI data sets, this paper analyzes the comprehensive effect of the integration algorithm under different feature percentages. Given that the number of base clusters is 60, then select 50%, 60%, 70%, 80% and 90% feature ratios to generate base clusters. Fifty repeated experiments were conducted under different feature subsets, and the results of the evaluation indicators were averaged to analyze their changes. The specific results are shown in the following figures. Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 show the specific experimental results.

From the experimental results recorded in Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13, we can find that different data sets obtain the best performances at different percentages. It can also be seen from the results shown in the figure that with the increasing proportion of feature subsets, the four evaluation indices used in this algorithm fluctuate a little on each data set. Although individual data sets are slightly affected when the feature proportion changes, overall, the change is small.

5. Conclusions and Future Work

It has been recognized that a single clustering algorithm cannot identify all types of data structures. Ensemble clustering is an effective way to solve the problem that a single clustering algorithm cannot identify all types of data structures. A traditional clustering algorithm may lead to high decision-making which is caused by inaccurate information or insufficient data. Therefore, in this paper, we propose a three-way ensemble clustering algorithm based on the sample’s perturbation theory. In this algorithm, we use sample’s perturbation theory to obtain perturbed data sets, and then randomly extract feature subsets of samples, and use traditional clustering algorithms to obtain different base clusters. We use a set of basis-clustering results as input to obtain the stability of each sample through the co-association matrix and determinacy function, and the stability threshold obtained by the Otus algorithm divides the samples into stable region and unstable region. Different regions take different strategies. The stable region is composed of samples with high stability, and we use K-means algorithm to assign them to the core region of each cluster. The unstable region is composed of samples with low stability, and we assign them to the fringe region of each cluster. Therefore, a three-way ensemble clustering result is obtained. The experimental results on UCI data sets show that the new algorithm can effectively reveal the data structure compared with the traditional ensemble clustering algorithm.

The following topics deserve further investigation:

(1)

The parameter selection of the clustering algorithm is a complicated problem. The parameters q is significant to the clustering results in this paper. Thus, the studies on the effect of the parameter dynamic changes on the results of the algorithm will be the subject of our future study.

(2)

The base clusters generated by different feature subsets may have a low quality which may affect the final ensemble clustering result. Determining how bad base clusters will be a good research avenue of future research.