An Empirical Study of Cluster-Based MOEA/D Bare Bones PSO for Data Clustering ^†

Abstract

Previously, cluster-based multi or many objective function techniques were proposed to reduce the Pareto set. Recently, researchers proposed such techniques to find better solutions in the objective space to solve engineering problems. In this work, we applied a cluster-based approach for solution selection in a multiobjective evolutionary algorithm based on decomposition with bare bones particle swarm optimization for data clustering and investigated its clustering performance. In our previous work, we found that MOEA/D with BBPSO performed the best on 10 datasets. Here, we extend this work applying a cluster-based approach tested on 13 UCI datasets. We compared with six multiobjective evolutionary clustering algorithms from the existing literature and ten from our previous work. The proposed technique was found to perform well on datasets highly overlapping clusters, such as CMC and Sonar. So far, we found only one work that used cluster-based MOEA for clustering data, the hierarchical topology multiobjective clustering algorithm. All other cluster-based MOEA found were used to solve other problems that are not data clustering problems. By clustering Pareto solutions and evaluating new candidates against the found cluster representatives, local search is introduced in the solution selection process within the objective space, which can be effective on datasets with highly overlapping clusters. This is an added layer of search control in the objective space. The results are found to be promising, prompting different areas of future research which are discussed, including the study of its effects with an increasing number of clusters as well as with other objective functions.

1. Introduction

Clustering is widely used to find hidden structures in data. In clustering, a set of C cluster centers $\mathbf{v}=\{v_1,\dots ,v_C\}$ represents prototypes of clusters. Each cluster contain similar objects in a dataset $Z=\{z_1,\dots ,z_N\}$ with N objects and D dimensions, where each object is a vector $\mathbf{z}=\{z_1,\dots ,z_D\}$. The goal of clustering is to learn the partition matrix $\mathbf{U}$ of a dataset. The partition matrix shows that an object $z_j$ belongs to the cluster $C_i$ and is represented by a $C\times N$ matrix as $\mathbf{U}=\left[u_{ij}\right]$ where $i=1,\dots ,C$ and $j=1,\dots ,N$ such that ${\sum }_{i=1}^C{u}_{ij}=1$. In hard clustering, such as k-means, $u_{ij}$ = 1 if $z_j\in C_i$ and 0, otherwise. In fuzzy clustering, such as fuzzy C-means (FCM), $u_{ij}\in \{0,1\}$ and $\arg {max}_i{u}_{ij}$ if $z_j\in C_i$. These conventional techniques aims to minimize a single objective function as follows:

$$J=\sum _{i=1}^C\sum _{j=1}^N{u}_{ij}^m\left|\right|z_j-v_i{\left|\right|}^2$$

(1)

where m, the fuzzy parameter, is 2. With a single objective function, different characteristics of hidden structures found in the dataset may not be well captured [1]. Combining multiple objectives into one may not fully reflect conflicting characteristics, such as cluster compactness and separation [2]. Thus, multiple objective functions that are optimized simultaneously in an evolving population of solutions were developed [3]. In this way, cluster characteristics can be more accurately defined, evolving to better solutions. Applications of multiobjective evolutionary algorithms (MOEA) in clustering have shown to be effective in identifying meaningful clusters [1,3,4].

In MOEA clustering or multiobjective clustering (MOC), many works focused on improving the algorithms through the treatment of chromosome encoding, use of objective functions [5], choice of MOEA, incorporation of other techniques, such as swarm intelligence, choice of evolutionary operations and selection of the final solution [6,7]. While applications of MOEAs are covered in many works and surveys [1,6], there is a research gap where few studies have applied newer MOEA strategies in data clustering problems, tested on datasets with different characteristics [8,9,10]. With this motivation, we previously investigated the application of a variety of MOEA algorithms to QPSO, BBPSO and fuzzy clustering algorithms tested on 10 UCI datasets [4]. Decomposition-based MOEA with bare bones particle swarm optimization (MOEA/D BBPSO) performed the best on most datasets. In this work, as continuation, we applied a cluster-based MOEA/D optimization technique to BBPSO clustering (CM-BBPSO) to introduce good Pareto solutions into the Pareto set to be considered for selection and investigated its performance for data clustering problems. To the best of our knowledge, there is very little work done in cluster-based MOEA for data clustering. Most cluster-based MOEA are for solving other engineering problems but not data clustering.

The cluster-based approach in solution selection have become increasingly popular in multiobjective optimization (MOO). The cluster-based selection approach applied to more than one objective function optimization was first proposed by [11]. The approach generated a reduced Pareto set in many-objective optimization (MaOO) problems to solve its scaling problem. Furthermore, cluster-based approaches are also applied to improve good Pareto set in multi- and many-objective optimization problems, such as applying clustering with a flexible similarity metric [12] and applying cluster-based solution selection [13]. Chen et al. [14] applied density-based fuzzy C-means clustering in a dynamic nondominated sorting multiobjective genetic algorithm to improve traffic congestion. While these cluster-based approaches improve the MOO algorithms, to the best of our knowledge, such approaches are not commonly applied to data clustering problems.

In this paper, we aim to incorporate the cluster-based MOEA selection strategy into a BBPSO clustering framework and investigate its clustering performance. By applying clustering on the Pareto set, we hope to benefit from the combined strength of solutions based on found cluster representatives to guide the solution selection. As part of our ongoing studies in PSO clustering, we also investigate and compare with the MOEA strategies in a PSO and fuzzy clustering framework. Our objectives are two-fold: first, to compare and study the clustering performance of CM-BBPSO and MOPSO clustering when compared to single objective BBPSO clustering algorithms to determine whether cluster-based MOEA/D strategy is effective for clustering; secondly, we would like to study the clustering behavior of these algorithms by applying them on datasets of different characteristics, such as size, cluster number, number of dimensions, density and overlap. We test on 13 UCI datasets [15] and compare with BBPSO and MOEA clustering algorithms from our previous work.

The outline of the paper is as follows. We discuss a few latest works that applied cluster-based selection in MOEA. Next, we present the materials and methods we used in our proposed technique, starting with a brief introduction of MOEA/D, followed by an explanation of our methodology of cluster-based MOEA/D BBPSO (CM-BBPSO). Next, we present the details of our experiments and data used. The results and discussion are presented next, and the paper ends with conclusion.

2. Related Work

In this section, we discuss two aspects relating to this study. First, the development of cluster-based MOO, where clustering approaches are used to solve solution selection problems in MOO for engineering problems, not data clustering. Here, clustering is applied on the fitness values of MOO solutions, and not on the data samples. Such techniques have not been applied to data clustering problems such as in pattern recognition or knowledge discovery. Second, we review the latest development of multiobjective clustering techniques applied to the data clustering problem.

2.1. Cluster-Based Multiobjective Optimisation

An early work on cluster-based solution selection in the Pareto set was proposed by [11], applied to generate a reduced Pareto set to solve the scaling and high-dimensional problems in many-objective optimization (MaOO). There is an increasing number of works that apply such an approach to improve the MOO solution selection. Here, the fitness values of solutions are clustered to find representatives (cluster centers). This means that solutions with similar fitness values are grouped together and a representative for each group is generated for guiding the solution selection or for the evolution of the solutions. Agrawal et al. [16] applied fuzzy clustering in multiobjective particle swarm optimization to perform electrical power dispatching, particularly to minimize fuel cost and emission. Their technique outperformed traditional MOO techniques, including NSGA, NPGA and SPEA. In [17], a density-based clustering approach DBSCAN was applied in MOO to find diverse optimal and near-optimal Pareto fronts to solve water resource problems. During dominance check, only local comparisons with solutions sharing the same clusters were checked.The authors stated that the introduction of a dissimilar index in the clustering for deciding whether to select or remove solutions for further evolution gave users more control as decision support. Hua et al. [18] used an agglomerative hierarchical cluster-based approach with Ward’s linkage in MOEA to guide selection in a way that maintains diversity and reduces convergence time. Their technique was applied to solve the problem of poor solutions selection due to an irregular Pareto front. The authors shared that using K-means generated spherical clusters and required a prior number of clusters. Liu et al. [12] proposed a flexible similarity metric for cluster-based MOEA to handle both convex- and concave-shaped Pareto fronts.

Chen et al. [14] applied density-based Fuzzy C-means in clustering in a dynamic nondominated sorting multiobjective genetic algorithm to determine Pareto fronts distribution, applied to improve traffic congestion. In their previous work [19], they applied a clustering method that was based on the VAR and PRE method. While these cluster-based approaches improve MOO algorithms, to the best of our knowledge, such approaches are not commonly applied to data clustering problems. So far, we found only [20], who applied cluster-based MOO for data clustering, proposing the hierarchical topology-based cluster representation, which is based on a tree-based graph for multiobjective clustering (HT-MOC), scalable to large datasets. The approach involves the use of a seeding algorithm followed by the use of the formulation of minimum spanning trees before MOC clustering processes, which include an ensemble operator to produce consensus clusterings before the final selection operator is employed. As such cluster-based MOO techniques have shown to improve optimized solutions, we opted to apply K-means, as it is simple to implement and works fast, compared to density-based clustering. We understand that the shape of the Pareto front may not be convex, but we proceeded with k equaling two. As there is little work on cluster-based MOO for clustering, we chose simple techniques to evaluate the effective of the approach. Our approach is different from [20], as we did not apply ensemble, nor is our cluster-representation tree-based. We used the generic centroid-based encoding.

2.2. Multiobjective Clustering

MOEA is popularly applied in data clustering. Mukhopadhyay et al. discussed extensively MOEA clustering as well as the application of metaheuristics, such as differential evolution, particle swarm optimization and genetic programming, for data clustering. In [21], Armano and Farmani developed a multiobjective particle swarm optimization algorithm for automatic data clustering, using connectivity and cohesion objective functions and a locus-based adjacency genetic scheme. They reported their results using ARI, but ARI was not used in this study, making it not comparable. Wang et al. [22] proposed two ensemble MOEA clustering algorithms, NSGA-II-ECFE and MOEA/D-ECFE, using four objective functions, DB, Dunn, cohesion, and stability, for patient stratification. Their ensemble construction involved using K-means to generate base clustering. They tested on 55 synthetic and 35 real patient stratification datasets. Unlike Wang et al. [22] and Zhu et al. [20], we did not use ensemble techniques but chose to use a simple swarm intelligence technique BBPSO to maintain our population of solutions. Instead of PSO, as in [21], we used BBPSO, which is parameter-free. It is interesting to note that despite its simplicity, there are not as many works applying BBPSO or MOO BBPSO to data clustering. To name a few, Omran et al. applied BBPSO clustering for unsupervised image segmentation [23], using a quantization-error-based objective function. Jiang and Wang [24] proposed cooperative bare-bone particle swarm optimization (CBPSO) for data clustering, using a new centroid-based encoding scheme for each particle and the Chernoff bounds on a suitable population size.

Other MOC techniques apply other nature-inspired approaches. Peng et al. [25] proposed a multiobjective fuzzy clustering framework, MOFC-TMS, with a tissue-like membrane system, which has a special cell structure for integrating a non-dominated sorting technique with a modified differential evolution mechanism. The proposed technique was developed to optimize three objective functions and was demonstrated to produce good results on many benchmark datasets. Kushwaha et al. [26] proposed a magnetic optimization algorithm (MOA) inspired by the magnetic force for data clustering. The algorithm is not sensitive to initialization, and the particle update is based on the magnetic resultant force. Guo et al. [27] proposed the evolutionary state-based novel multiobjective periodic bacterial foraging optimization algorithm (ES-NMPBFO). This is a novel multiobjective periodic bacterial foraging optimization (BFO) algorithm for data clustering, incorporating PSO mechanisms into the chemotaxis operation.

2.3. Motivation

The HT-MOC [20] is a highly effective cluster-based MOO algorithm for solving challenging and large clustering problems, with overlapping and mixed-density clusters, applying many sophisticated techniques, such as tree-based graph cluster representation for a hierarchical topology for clustering and ensemble clustering. As there are few works in cluster-based MOO for data clustering, we hope to develop a simple cluster-based MOC technique, comprising familiar good-performing algorithms, with little to no parameter setting, to show its effectiveness to interested practitioners. For MOEA clustering, there is no local search in the evolutionary algorithm. The search for $pbest$ in BBPSO is a local search within the sample search space. By applying cluster-based MOO selection, we introduced local search when selection solutions evolve in the next generation in the objective space. For the cluster-based part of the MOO, we applied K-means with two clusters. Like [17], only local comparisons with solutions sharing the same clusters are checked for dominance. For the MOO, we used MOEA/D to find $gbest$ using the $pbest$ particles found by BBPSO. MOEA/D can simultaneously evolve subproblem multiobjectives and provide a scalar solution. They showed good performance in data clustering tasks. BBPSO does not require any parameter setting and showed good results for clustering. We hope that by testing on common datasets and using common evaluation metrics, the work can be a reference for the future direction of applying and developing cluster-based MOC to solve sophisticated clustering problems.

2.4. Contribution

• We developed a novel framework that requires little to no parameter setting consisting of a simple swarm intelligence technique BBPSO and MOO technique MOEA/D with a solution update based on K-means for data clustering. To the best of our knowledge, this approach has not been explored.
• We ran extensive investigations to evaluate its performance. We tested on 13 datasets with different characteristics and evaluated using 5 different popular metrics. We compared our results with other work to show how the proposed technique performs relatively. We conducted other analysis such as analysis of the convergence plot to assess the stability of the proposed technique and visual examination of biplots of the clustering found by the proposed technique.

3. Materials and Methods

In this section, we describe the techniques, experiments and data used to implement and evaluate the proposed technique, CM-BBPSO.

3.1. Bare Bones Particle Swarm Optimisation

The BBPSO was introduced to simplify PSO, removing the need for parameters or velocity [28]. Each particle $p\in \{1,\dots ,P\}$ in the swarm holds two properties: (a) its current position ${\mathbf{x}}_p$ and (b) its personal best position $pbes{t}_p$. At each iteration t, the position of the particle is updated based on $N(\mu ,\sigma )$, a multivariate Gaussian distribution with mean $\mu$, and standard deviation $\sigma$ as follows:

$$\begin{array}{c}\hfill x_p(t+1)=N\left(\mu ,\sigma \right)\\ \hfill \mu =\frac{pbes{t}_p\left(t\right)+gbest\left(t\right)}{2}\\ \hfill \sigma =|pbes{t}_p\left(t\right)-gbest\left(t\right)|\end{array}$$

(2)

In addition to parameter reduction, the multivariate Gaussian distribution operator helps maintain diverse candidates in the population for $pbest$ particles to improve from. The BBPSO clustering algorithm to be used in this study is shown in Algorithm 1. The $gbest$ particles are evolved from pbest particles and refined using cluster-based MOEA/D.

Algorithm 1 Pseudocode for BBPSO Clustering

1: Set max number of iterations I, max number of iterations T, Population size S, $t=0$, $i=0$, generate population P and initialize $gBestList$ using MOEA/D and all pbest $pbestList$ 2: while $i<I$do 3:    while $t<T$ do 4:      Calculate any parameters 5:      for Each particle do 6:         Calculate new p using (2) 7:         Check new p is within search range R 8:         Calculate fitness of p and pBest using quantization error function from [23] 9:         If new p dominates pBest, update p and pBest with new p 10:      end for 11:      gBestList = getGBestList(pbestList) using cluster-based MOEA/D (Algorithm 2) 12:    end while 13: end while 14: Evaluate crowding distance of all solutions in gBestList 15: Choose gBest with largest crowding distance

3.2. Decomposition-Based Multiobjective Evolutionary Algorithm

The decomposition-based MOEA (MOEA/D) uses three approaches—the weighted sum, Tchebycheff and penalty-based boundary intersection (PBI)—to convert MOP into scalar optimization subproblems, simultaneously solved by evolving a population of solutions [29]. Unlike previous MOO techniques, such as dominance-based MOEA, there is no clear ranking of Pareto solutions, while with MOEA/D, such a ranking is possible, as the $g^{pbi}$ function is expressed as a scalar value as defined in the objective below. MOEA/D with the PBI approach for an m-objective problem [29] is defined as follows:

$$\begin{array}{cc}\hfill \mathrm{Minimize}:& g^{pbi}\left(x\right|\lambda ,z^{*})=d_1+\theta d_2\hfill \\ \hfill \mathrm{subject}:& x\in \mathsf{\Omega }\hfill \end{array}$$

(3)

where

$$\begin{array}{cc}\hfill & d_1=\frac{\left|\right|{(z^{*}-F\left(x\right))}^T\lambda \left|\right|}{\left|\right|\lambda \left|\right|}\hfill \\ \hfill & d_2=\left|\right|F\left(x\right)-(z^{*}-d_1\lambda )\left|\right|,\hfill \end{array}$$

(4)

where x is a candidate solution, $\theta >0$ is a predefined penalty parameter, $z^{*}={(z_1^{*},\dots ,z_m^{*})}^T$ is the reference point such that $z_i^{*}=min\left\{f_i\left(x\right)\right|x\in \mathsf{\Omega }\}$, $\lambda ={({\lambda }_1,\dots ,{\lambda }_m)}^T$ is a weight vector such that ${\lambda }_i\ge 0$ for $\forall i=1,\dots ,m$ and ${\sum }_{i=1}^m{\lambda }_i=1$. In this study, x is a candidate solution that undergoes the evaluation process of cluster-based MOEA/D to be selected for next generation, as detailed in Algorithm 2, line 15.

Algorithm 2 Pseudocode for Cluster-based MOEA/D

1: Initialization phase: 2: Set external population EP = ∅ 3: Initialize N, number of subproblems 4: Generate a uniform spread of N weight vectors ${\lambda }^1,\dots ,{\lambda }^N$ 5: For each $i=1,\dots ,N$, generate $B\left(i\right)=\{i_1,\dots ,i_T\}$ where ${\lambda }^{i_1},\dots ,{\lambda }^{i_T}$ are the T closest weight vectors to ${\lambda }^i$ using Euclidean distance 6: Using solutions in pbestList to generate initial population $x^1,\dots ,x^N\in \mathsf{\Omega }$ where $x^i$ is current solution to ith subproblem and set fitness value of all individuals $F{V}^i=F\left(x^i\right)$ for $i=1,\dots ,N$ 7: Initialize reference point $z={(z_1,\dots ,z_m)}^T$ where $z_i$ is the best value so far for objective $f_i$ 8: Set number of clusters k as 2 9: Cluster individuals $x^1,\dots ,x^N$ based on their similarity in the $FV$ matrix and find which $x^{ic}$ is most representative for each cluster c. 10: Update phase: 11: for $c=1,\dots ,k$ do 12:    Reproduction: Generate new solution y using genetic operators on two solutions $x^k$ and $x^l$ where indexes k and l are randomly chosen from $B\left(i\right)$ 13:    Improvement: Apply any improvement heuristic on y to produce $y^{\prime }$ 14:    Update z: For each $j=1,\dots ,m$, if $z_j<f_j\left(y^{\prime }\right)$, set $z_j=f_j\left(y^{\prime }\right)$ 15:    Update neighbors: For each $j\in B\left(i\right)$ belonging to cluster c, if $g^{pbi}\left(y^{\prime }\right|{\lambda }^j,z)\le g^{pbi}\left(x^j\right|{\lambda }^j,z)$, $g^{pbi}$ values calculated using (3), set $x^j=y^{\prime }$ and $F{V}^j=F\left(y^{\prime }\right)$ 16:    Update EP: Remove solutions in EP dominated by $y^{\prime }$ and add $y^{\prime }$ to EP if no vectors in EP dominate $y^{\prime }$. 17: end for 18: Termination Condition: Stop and output EP if termination condition is met.

3.3. Cluster-Based MOEA/D BBPSO

Each solution in a multiobjective clustering (MOC) problem is regarded as a set of cluster centers ${\mathbf{X}}_s$ or $\mathbf{v}$ in (1). Cluster connectedness (Conn) and within-sum-of-squares (WSS) [3] are used as objective functions in this work to evaluate the best solution candidates. The final solution is then selected using crowding distance on the external population. In a MOC framework, the population P contains S solutions with $\mathbf{X}=\{{\mathbf{X}}_1,\dots ,{\mathbf{X}}_s\}$ where the matrix of each solution ${\mathbf{X}}_s$ represents C cluster centers of a dataset containing N objects and D dimensions as follows:

$$\begin{array}{c}\hfill {\mathbf{X}}_s=\left[\begin{array}{cccc}{x}_{11}& x_{12}& \cdots & x_{1D}\\ x_{21}& x_{22}& \cdots & x_{2D}\\ ⋮& ⋮& \ddots & ⋮\\ x_{C1}& x_{C2}& \cdots & x_{CD}\end{array}\right]\end{array}$$

(5)

In the CM-BBPSO clustering framework, there are two parts. The BBPSO clustering algorithm is in charge of updating the particles in the swarm. The $pBest\mathrm{s}$ from BBPSO are selected based on the quantization error-based fitness function from [23]. BBPSO is used because it requires lesser parameters than PSO and performs well in clustering, based on previous work [30]. Next, we subject the $pbest$ solutions found by BBPSO to the cluster-based MOEA/D algorithm for solution evolving and selection to find the $gbest$ for the BBPSO framework. The population size for the cluster-based MOEA/D is the same as the swarm size for the BBPSO. The objective functions Conn and WSS are used to update $FV$ matrix and then evaluate solutions in lines 6, 15, 16 in Algorithm 2. Newly evolved solutions are compared with existing ones and the better one is selected to be updated. The fitness matrix $FV$ containing WSS and Connectedness fitness values of the better solution are stored.

The cluster-based approach used in this study is adapted from [13] and applied at line 9 in Algorithm 2 in the MOEA/D PBI framework from [29] to find the $gBest$ candidates, given all $pBest$. Here, we apply k-means clustering with $k=2$ to guide the solution selection. Based on the fitness values in the $FV$ matrix of N subproblems, we cluster them. The improved potential individual $y^{\prime }$ is evaluated against neighbors belonging to the group and added to $EP$ if it has a more favorable evaluation value. In doing so, the application of the cluster-based approach is to improve the local search within the objective space for new candidates, while the genetic operators maintain a random global search. We will test on higher k values to explore deeper in future work. In this work, we apply a small k value and simple conventional algorithms and distance measures, such as k-means and Euclidean distance, to study the effects of clustering-based MOEA/D as a preliminary study. Indeed, more investigation is required to understand deeply the Pareto solutions found in the objective space and the more sophisticated methods to deal with challenging Pareto solutions as proposed in [18].

To investigate its performance, we tested on 13 public datasets and compared cluster-based MOEA/D BBPSO (CM-BBPSO) with the following MOPSO clustering frameworks: IBEA QPSO (IB-QPSO), SPEA2 QPSO (SP-QPSO), MOEA/D QPSO (M-QPSO)and BBPSO (M-BBPSO), and MOEA/GLU QPSO (MG-QPSO) and BBPSO (MG-BBPSO), where the respective MOEAs are individually incorporated into a fuzzy, QPSO or BBPSO clustering framework as described in Algorithm 1 in [4]. An abbreviation list is given before the References section.

3.4. Experiments

We employed similar experiment settings in [4] with a swarm size of 20 and maximum iteration of 100, together with the following experiment settings. We previously obtained good results using similar datasets. Ten generations are run in the cluster-based MOEA/D framework, and each experiment is run thirty times. The data are scaled before applying clustering. To produce different solutions, we set the MOC algorithms to generate more than one solution. The final solution is selected by crowding distance. The number of clusters is set as the number of classes in the dataset.

The evaluation metrics used are accuracy, F1 and Kappa for external evaluation, and QE, WSS and BSS for internal evaluation. Different works in data clustering use different measures, which make comparison of other work difficult. There appears to be a mixed opinion to use external metrics to evaluate clustering, while another school of thought is that the point of clustering is to generate labels to perform classification and aims to generate such labels. This depends on the objective of the users and it is not within our scope to discuss this matter. We present both types of criteria for relevant researchers.

To evaluate the clustering performance and investigate in clustering behavior of CM-BBPSO, we conduct several analyses. We compare the CM-BBPSO with non-cluster based MOEA/D BPPSO and with BBPSO to study how the cluster-based MOEA/D approach has made a difference. Previously in [4], we observed that the application of MOEA/D in BBPSO algorithm improves the performance. By comparing with BBPSO clustering techniques, we can determine if the cluster-based MOEA/D approach further improves BBPSO and M-BBPSO.

CM-BBPSO is then compared with other MOPSOs and MOO Fuzzy algorithms to evaluate its performance. In particular, M-Fuzzy is found to perform competitively to M-BBPSO in [4]. Previously, we were not able to obtain results on CMC, segment and vehicle datasets. Through this analysis here, we are re-evaluating by comparison with the said algorithms and the additional datasets, which usually give poor results, as was reported in [30]. We also compare CM-BBPSO results with the existing cluster-based MOO technique HT-MOC [20], MOC techniques MOFC-TMS [25], VAMOSA [31], GenClustMOO [32], MOAC [26], ES-NMPBFO [27] and BBPSO clustering technique CBPSO [24] based on commonly reported datasets and evaluation metrics.

The following swarm sizes and maximum iterations (presented as ss-mi) are tested: 20–100, 30–100, 30–200 and 40–100. This is done to study whether am increase in the two parameters will improve clustering. The biplot visual examination allows us to evaluate CM-BBPSO performance in comparison with the ground truth, examining how well CM-BBPSO identifies overlapping clusters. With the analysis of the convergence plots of $pbest$, we can evaluate the stability of CM-BBPSO.

3.5. Data

The algorithms are tested on 13 datasets obtained from the UCI Machine Learning Repository: Appendicitis, CMC, Dermatology, Ecoli, Glass, Iris, Seeds, Sonar, Vehicle, WDBC, Wine, WOBC and Segment [15]. Full details and t-SNE biplots of the datasets are found in [30]. These datasets have different types of characteristics, such as sizes N, dimensions n and cluster numbers c, as well as having different intensity of densities and overlapping. We are particularly interested to see the effects of CM-BBPSO on challenging datasets with highly overlapping clusters and a combination of the following characteristics: high number of classes, mixed densities, imbalanced class ratios. We choose a mixture of datasets for which most algorithms tend to produce good results as well as challenging datasets, such as Glass, Sonar, Ecoli, Segment, Vehicle and CMC. Many works tested algorithms with datasets with 2 or 3 classes, such as Iris, Wine, Appendicitis, WOBC and WDBC, which give good results.

4. Results

4.1. BBPSO Comparisons

In

Table 1. Performance of cluster-based MOEA/D (CM), MOEA/D (M) and ordinary BBPSOs with the best result produced by the algorithm for each dataset presented in bold.

Dataset	BBPSO	Accuracy	F1	Kappa	BSS	WSS	QE
Appendicitis	CM-BBPSO	$0.771\pm 0.08$	$0.629\pm 0.15$	$0.280\pm 0.28$	$325\pm 144$	$249\pm 18$	$2.56\pm 0.43$
	M-BBPSO	$0.817\pm 0.05$	$0.676\pm 0.14$	$0.367\pm 0.27$	$229\pm 108$	$\mathbf{230}\pm \mathbf{189}$	$\mathbf{2}.\mathbf{49}\pm \mathbf{0}.\mathbf{54}$
	BBPSO	$\mathbf{0}.\mathbf{818}\pm \mathbf{0}.\mathbf{07}$	$\mathbf{0}.\mathbf{717}\pm \mathbf{0}.\mathbf{11}$	$\mathbf{0}.\mathbf{449}\pm \mathbf{0}.\mathbf{20}$	$\mathbf{979}\pm \mathbf{572}$	$271\pm 58$	$2.54\pm 0.56$
CMC	CM-BBPSO	$\mathbf{0}.\mathbf{402}\pm \mathbf{0}.\mathbf{02}$	$\mathbf{0}.\mathbf{371}\pm \mathbf{0}.\mathbf{04}$	$\mathbf{0}.\mathbf{089}\pm \mathbf{0}.\mathbf{05}$	$6590\pm 1540$	$4150\pm 186$	$2.88\pm 0.25$
	M-BBPSO	$0.400\pm 0.02$	$0.352\pm 0.03$	$0.063\pm 0.04$	$5230\pm 973$	$\mathbf{4020}\pm \mathbf{142}$	$\mathbf{2}.\mathbf{77}\pm \mathbf{0}.\mathbf{19}$
	BBPSO	$0.390\pm 0.02$	$0.355\pm 0.05$	$0.064\pm 0.06$	16,600±3680	$5080\pm 504$	$3.49\pm 0.330$
Dermatology	CM-BBPSO	$0.651\pm 0.07$	$0.556\pm 0.08$	$0.570\pm 0.08$	$7620\pm 1060$	$1870\pm 58$	$5.31\pm 0.31$
	M-BBPSO	$0.666\pm 0.08$	$0.572\pm 0.10$	$0.588\pm 0.10$	$7440\pm 936$	$\mathbf{1760}\pm \mathbf{56}$	$\mathbf{5}.\mathbf{11}\pm \mathbf{0}.\mathbf{37}$
	BBPSO	$\mathbf{0}.\mathbf{680}\pm \mathbf{0}.\mathbf{08}$	$\mathbf{0}.\mathbf{607}\pm \mathbf{0}.\mathbf{11}$	$\mathbf{0}.\mathbf{605}\pm \mathbf{0}.\mathbf{10}$	10,300$\pm \mathbf{1340}$	$1930\pm 94$	$5.28\pm 0.37$
Ecoli	CM-BBPSO	$\mathbf{0}.\mathbf{499}\pm \mathbf{0}.\mathbf{07}$	$\mathbf{0}.\mathbf{333}\pm \mathbf{0}.\mathbf{05}$	$\mathbf{0}.\mathbf{372}\pm \mathbf{0}.\mathbf{06}$	$1060\pm 183$	$576\pm 19$	$1.97\pm 0.33$
	M-BBPSO	$0.493\pm 0.08$	$0.326\pm 0.06$	$0.364\pm 0.08$	$1110\pm 181$	$\mathbf{558}\pm \mathbf{28}$	$\mathbf{1}.\mathbf{89}\pm \mathbf{0}.\mathbf{20}$
	BBPSO	$0.437\pm 0.05$	$0.266\pm 0.04$	$0.295\pm 0.05$	$\mathbf{1910}\pm \mathbf{293}$	$694\pm 43$	$2.01\pm 0.13$
Glass	CM-BBPSO	$0.429\pm 0.04$	$0.351\pm 0.05$	$0.235\pm 0.04$	$818\pm 186$	$438\pm 19$	$2.75\pm 0.45$
	M-BBPSO	$\mathbf{0}.\mathbf{462}\pm \mathbf{0}.\mathbf{05}$	$\mathbf{0}.\mathbf{382}\pm \mathbf{0}.\mathbf{05}$	$\mathbf{0}.\mathbf{280}\pm \mathbf{0}.\mathbf{06}$	$864\pm 168$	$\mathbf{422}\pm \mathbf{28}$	$\mathbf{2}.\mathbf{69}\pm \mathbf{0}.\mathbf{56}$
	BBPSO	$0.424\pm 0.04$	$0.354\pm 0.05$	$0.229\pm 0.05$	$\mathbf{2430}\pm \mathbf{861}$	$640\pm 119$	$2.95\pm 0.66$
Iris	CM-BBPSO	$0.721\pm 0.10$	$0.672\pm 0.14$	$0.582\pm 0.15$	$421\pm 123$	$205\pm 34$	$1.41\pm 0.29$
	M-BBPSO	$0.731\pm 0.09$	$0.677\pm 0.13$	$0.597\pm 0.13$	$439\pm 85.8$	$172\pm 23$	$1.27\pm 0.32$
	BBPSO	$\mathbf{0}.\mathbf{786}\pm \mathbf{0}.\mathbf{08}$	$\mathbf{0}.\mathbf{769}\pm \mathbf{0}.\mathbf{11}$	$\mathbf{0}.\mathbf{679}\pm \mathbf{0}.\mathbf{12}$	$\mathbf{674}\pm \mathbf{93}.\mathbf{6}$	$\mathbf{166}\pm \mathbf{9}$	$\mathbf{1}.\mathbf{10}\pm \mathbf{0}.\mathbf{07}$
Seeds	CM-BBPSO	$0.727\pm 0.12$	$0.684\pm 0.16$	$0.590\pm 0.18$	$1050\pm 456$	$431\pm 53$	$2.10\pm 0.37$
	M-BBPSO	$0.798\pm 0.13$	$0.758\pm 0.18$	$0.696\pm 0.19$	$1100\pm 295$	$375\pm 37$	$1.83\pm 0.24$
	BBPSO	$\mathbf{0}.\mathbf{876}\pm \mathbf{0}.\mathbf{08}$	$\mathbf{0}.\mathbf{870}\pm \mathbf{0}.\mathbf{10}$	$\mathbf{0}.\mathbf{814}\pm \mathbf{0}.\mathbf{12}$	$\mathbf{1890}\pm \mathbf{191}$	$\mathbf{358}\pm \mathbf{25}$	$\mathbf{1}.\mathbf{71}\pm \mathbf{0}.\mathbf{13}$
Segment	CM-BBPSO	$0.547\pm 0.06$	$0.497\pm 0.06$	$0.471\pm 0.07$	23,700 ± 3610	$6520\pm 324$	$3.15\pm 0.41$
	M-BBPSO	$\mathbf{0}.\mathbf{581}\pm \mathbf{0}.\mathbf{06}$	$\mathbf{0}.\mathbf{528}\pm \mathbf{0}.\mathbf{08}$	$\mathbf{0}.\mathbf{511}\pm \mathbf{0}.\mathbf{07}$	23,500 ± 2980	$\mathbf{6140}\pm \mathbf{267}$	$\mathbf{3}.\mathbf{1}\pm \mathbf{0}.\mathbf{57}$
	BBPSO	$0.560\pm 0.07$	$0.523\pm 0.08$	$0.487\pm 0.08$	33,700$\pm \mathbf{4470}$	$7730\pm 704$	$3.3\pm 0.29$
Sonar	CM-BBPSO	$\mathbf{0}.\mathbf{552}\pm \mathbf{0}.\mathbf{05}$	$\mathbf{0}.\mathbf{516}\pm \mathbf{0}.\mathbf{06}$	$\mathbf{0}.\mathbf{112}\pm \mathbf{0}.\mathbf{09}$	$7670\pm 1430$	$1810\pm 76$	$9.13\pm 0.70$
	M-BBPSO	$0.541\pm 0.03$	$0.474\pm 0.07$	$0.076\pm 0.07$	$6300\pm 1360$	$\mathbf{1750}\pm \mathbf{55}$	$\mathbf{9}.\mathbf{05}\pm \mathbf{0}.\mathbf{82}$
	BBPSO	$0.536\pm 0.04$	$0.455\pm 0.10$	$0.067\pm 0.09$	21,300±11,200	$2360\pm 396$	$11.8\pm 2.08$
Vehicle	CM-BBPSO	$0.374\pm 0.02$	$0.340\pm 0.032$	$0.168\pm 0.03$	11,100 ± 2310	$2950\pm 194$	$\mathbf{3}.\mathbf{66}\pm \mathbf{0}.\mathbf{59}$
	M-BBPSO	$0.378\pm 0.02$	$0.328\pm 0.03$	$0.175\pm 0.03$	10,600 ± 2200	$\mathbf{2780}\pm \mathbf{125}$	$3.79\pm 0.83$
	BBPSO	$\mathbf{0}.\mathbf{379}\pm \mathbf{0}.\mathbf{02}$	$\mathbf{0}.\mathbf{360}\pm \mathbf{0}.\mathbf{03}$	$\mathbf{0}.\mathbf{173}\pm \mathbf{0}.\mathbf{03}$	17,500±3130	$3110\pm 308$	$3.69\pm 0.36$
WDBC	CM-BBPSO	$0.782\pm 0.13$	$0.712\pm 0.19$	$0.478\pm 0.31$	$9290\pm 3260$	$3000\pm 226$	$6.08\pm 1.07$
	M-BBPSO	$\mathbf{0}.\mathbf{852}\pm \mathbf{0}.\mathbf{09}$	$\mathbf{0}.\mathbf{814}\pm \mathbf{0}.\mathbf{14}$	$0.651\pm 0.24$	$8910\pm 2490$	$\mathbf{2840}\pm \mathbf{190}$	$\mathbf{5}.\mathbf{67}\pm \mathbf{0}.\mathbf{94}$
	BBPSO	$0.785\pm 0.11$	$0.698\pm 0.19$	$0.463\pm 0.30$	37,300±10,200	$4320\pm 539$	$8.26\pm 1.23$
Wine	CM-BBPSO	$0.719\pm 0.15$	$0.691\pm 0.18$	$0.579\pm 0.23$	$1710\pm 366$	$622\pm 36$	$3.5\pm 0.25$
	M-BBPSO	$0.800\pm 0.15$	$0.778\pm 0.20$	$0.695\pm 0.24$	$1640\pm 395$	$\mathbf{586}\pm \mathbf{25}$	$\mathbf{3}.\mathbf{31}\pm \mathbf{0}.\mathbf{18}$
	BBPSO	$\mathbf{0}.\mathbf{840}\pm \mathbf{0}.\mathbf{13}$	$\mathbf{0}.\mathbf{827}\pm \mathbf{0}.\mathbf{16}$	$\mathbf{0}.\mathbf{764}\pm \mathbf{0}.\mathbf{18}$	$\mathbf{3170}\pm \mathbf{489}$	$640\pm 44$	$3.55\pm 0.33$
WOBC	CM-BBPSO	$0.896\pm 0.10$	$0.865\pm 0.16$	$0.749\pm 0.27$	$3800\pm 1700$	$1630\pm 392$	$2.87\pm 0.62$
	M-BBPSO	$\mathbf{0}.\mathbf{949}\pm \mathbf{0}.\mathbf{07}$	$\mathbf{0}.\mathbf{945}\pm \mathbf{0}.\mathbf{07}$	$\mathbf{0}.\mathbf{894}\pm \mathbf{0}.\mathbf{11}$	$3560\pm 988$	$\mathbf{1300}\pm \mathbf{109}$	$\mathbf{2}.\mathbf{27}\pm \mathbf{0}.\mathbf{17}$
	BBPSO	$0.878\pm 0.03$	$0.85\pm 0.048$	$0.707\pm 0.09$	$\mathbf{9460}\pm \mathbf{545}$	$1560\pm 90$	$2.91\pm 0.27$

, we compare the performance of CM-BBPSO with MOEA/D BBPSO and BBPSO evaluated on 13 UCI datasets and using 5 metrics: accuracy, F1-score (F1), Kappa (Cohen’s $\kappa$ index), between sum of squares (BSS), within sum of square (WSS) and quantization error (QE). The best result for each measure is highlighted in bold.

We observe that CB-MOEAD-BBPSO performed best in the CMC, Ecoli and Sonar datasets, three of which are challenging datasets containing either many imbalanced, highly overlapping and/or mixed density clusters. These datasets appear to be more challenging as shown in our previous work ([4,30]). This demonstrates that using the cluster-based selection of optimal solutions helps to choose good solutions for challenging datasets. We observed that M-BBPSO produced the best WSS values in 11 datasets, indicating that M-BBPSO tends to optimize cluster compactness. BBPSO produced the best BSS values in all the datasets, indicating that BBPSO optimizes cluster well-separatedness. CM-BBPSO had the worst (smallest) BSS values in four datasets and the worst (largest) WSS values in two datasets. M-BBPSO produced the worst BSS values in 9 datasets, while BBPSO produced the worst WSS values in 11 datasets. This shows that BBPSO is good for finding well-separated clusters, such as in Iris and Seeds, with a small number of clusters, while M-BBPSO is good for finding clusters that are compact, such as WDBC and WOBC. Based on the BSS and WSS measure, we observed that CM-BBPSO balances between these two properties of compactness (WSS) and well-separatedness (BSS), not doing better on one, such as M-BBPSO and BBPSO as discussed earlier. Such characteristics are useful when dealing with datasets with highly overlapping clusters and high dimensions. QE is another type of measure to indicate compactness and shares the same trend as WSS.

4.2. MOO Comparisons

We compare CM-BBPSO with other MOC algorithms from our previous work. We present the results on three datasets, CMC, Segment and Vehicle, in

Table 2. Performance of CM-BBPSO and other clustering algorithms on CMC dataset with the best result produced by the algorithm presented in bold.

Clus Algo	Accuracy	F1	Kappa	BSS	WSS	QE
CM-BBPSO	$0.402\pm 0.02$	$0.371\pm 0.04$	$0.089\pm 0.05$	$6590\pm 1540$	$4150\pm 186$	$2.88\pm 0.25$
IB-Fuzzy	$0.393\pm 0.02$	$0.368\pm 0.04$	$0.081\pm 0.05$	$5650\pm 1380$	$4260\pm 194$	$2.9\pm 0.14$
IB-QPSO	$0.401\pm 0.02$	$0.321\pm 0.05$	$0.056\pm 0.05$	$3980\pm 1130$	$4120\pm 157$	$2.97\pm 0.29$
M-Fuzzy	$\mathbf{0}.\mathbf{404}\pm \mathbf{0}.\mathbf{02}$	$\mathbf{0}.\mathbf{383}\pm \mathbf{0}.\mathbf{03}$	$\mathbf{0}.\mathbf{100}\pm \mathbf{0}.\mathbf{04}$	$5310\pm 1280$	$\mathbf{3950}\pm \mathbf{174}$	$\mathbf{2}.\mathbf{72}\pm \mathbf{0}.\mathbf{15}$
M-QPSO	$0.402\pm 0.01$	$0.356\pm 0.04$	$0.069\pm 0.04$	$5820\pm 1370$	$4050\pm 129$	$2.86\pm 0.23$
M-BBPSO	$0.400\pm 0.02$	$0.352\pm 0.03$	$0.063\pm 0.04$	$5230\pm 973$	$4020\pm 142$	$2.77\pm 0.19$
MG-Fuzzy	$0.389\pm 0.02$	$0.365\pm 0.04$	$0.084\pm 0.05$	$6020\pm 1660$	$4250\pm 225$	$2.93\pm 0.18$
MG-QPSO	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
MG-BBPSO	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
SP-QPSO	$0.403\pm 0.02$	$0.331\pm 0.04$	$0.056\pm 0.04$	$3740\pm 988$	$3970\pm 142$	$2.75\pm 0.21$
BBPSO	$0.390\pm 0.02$	$0.355\pm 0.05$	$0.064\pm 0.06$	16,600±3680	$5080\pm 504$	$3.49\pm 0.33$

,

Table 3. Performance of CM-BBPSO and other clustering algorithms on Segment dataset with the best result produced by the algorithm presented in bold.

Clus Algo	Accuracy	F1	Kappa	BSS	WSS	QE
CM-BBPSO	$0.547\pm 0.06$	$0.497\pm 0.06$	$0.471\pm 0.07$	23,700±3610	$6520\pm 324$	$3.15\pm 0.42$
IB-Fuzzy	$0.548\pm 0.08$	$0.513\pm 0.08$	$0.472\pm 0.09$	18,900 ± 3580	$6650\pm 516$	$2.92\pm 0.34$
IB-QPSO	$0.536\pm 0.07$	$0.485\pm 0.08$	$0.459\pm 0.08$	20,000 ±5920	$6780\pm 356$	$3.34\pm 0.64$
M-Fuzzy	$\mathbf{0}.\mathbf{586}\pm \mathbf{0}.\mathbf{07}$	$\mathbf{0}.\mathbf{549}\pm \mathbf{0}.\mathbf{08}$	$\mathbf{0}.\mathbf{517}\pm \mathbf{0}.\mathbf{08}$	22,600 ± 3780	$\mathbf{5980}\pm \mathbf{448}$	$\mathbf{2}.\mathbf{73}\pm \mathbf{0}.\mathbf{31}$
M-QPSO	$0.569\pm 0.07$	$0.524\pm 0.09$	$0.497\pm 0.08$	24,800 ± 2610	$6250\pm 371$	$3.18\pm 0.68$
M-BBPSO	$0.581\pm 0.06$	$0.528\pm 0.08$	$0.511\pm 0.07$	23,500 ± 2980	$6140\pm 267$	$3.10\pm 0.57$
MG-Fuzzy	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
MG-QPSO	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
MG-BBPSO	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
SP-QPSO	$0.554\pm 0.05$	$0.499\pm 0.05$	$0.479\pm 0.05$	23,900 ± 3980	$6060\pm 274$	$2.96\pm 0.58$
BBPSO	$0.560\pm 0.07$	$0.523\pm 0.08$	$0.487\pm 0.08$	33,700 ±4470	$7730\pm 704$	$3.30\pm 0.29$

and

Table 4. Performance of CM-BBPSO and other clustering algorithms on Vehicle dataset with the best result produced by the algorithm presented in bold.

Clus Algo	Accuracy	F1	Kappa	BSS	WSS	QE
CM-BBPSO	$0.374\pm 0.02$	$0.34\pm 0.032$	$0.168\pm 0.0263$	11,100 ± 2310	$2950\pm 194$	$3.66\pm 0.59$
IB-Fuzzy	$0.369\pm 0.03$	$0.351\pm 0.03$	$0.160\pm 0.04$	$8450\pm 2430$	$2810\pm 283$	$3.33\pm 0.60$
IB-QPSO	$0.375\pm 0.03$	$0.321\pm 0.04$	$0.171\pm 0.03$	$8440\pm 2730$	$2940\pm 144$	$4.00\pm 0.83$
M-Fuzzy	$\mathbf{0}.\mathbf{387}\pm \mathbf{0}.\mathbf{03}$	$0.355\pm 0.04$	$\mathbf{0}.\mathbf{185}\pm \mathbf{0}.\mathbf{03}$	10,200 ± 1360	$\mathbf{2610}\pm \mathbf{86}$	$\mathbf{3}.\mathbf{16}\pm \mathbf{0}.\mathbf{29}$
M-QPSO	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
M-BBPSO	$0.378\pm 0.02$	$0.328\pm 0.03$	$0.175\pm 0.03$	10,600 ± 2200	$2780\pm 125$	$3.79\pm 0.83$
MG-Fuzzy	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
MG-QPSO	$NA$	$NA$	$NA$	$NA$	$NA$	$NA$
MG-BBPSO	$0.369\pm 0.02$	$0.323\pm 0.04$	$0.162\pm 0.03$	$9870\pm 1820$	$2770\pm 134$	$3.68\pm 0.85$
SP-QPSO	$0.378\pm 0.02$	$0.296\pm 0.03$	$0.179\pm 0.02$	$8210\pm 2010$	$2740\pm 114$	$4.42\pm 1.04$
BBPSO	$0.379\pm 0.02$	$\mathbf{0}.\mathbf{360}\pm \mathbf{0}.\mathbf{03}$	$0.173\pm 0.03$	17,500±3130	$3110\pm 308$	$3.69\pm 0.36$

which were not published in [4] due to run issues for clustering algorithms using multiobjective evolutionary algorithms with the global loop update (MG) [33], indicated as $NA$. The results for the other 10 datasets were published. In these three challenging datasets, CM-BBPSO performed competitively, particularly for CMC where it had the second best accuracy, F1, Kappa and BSS. In this comparison, CM-BBPSO did not outperform any other MOC, nor BBPSO techniques. Instead, M-Fuzzy emerged as the superior algorithm in these datasets for most evaluation metrics.

Using the averages of the six evaluation metrics, we performed sum rank of each algorithm across the 13 datasets. For algorithms with no results for certain datasets, we gave a rank of 6 (middle rank). The sum rank table is presented in

Table 5. Sum rank of algorithms based on performances on 6 metrics on all datasets with best two sum rank values presented in bold.

Clus Algo	Accuracy	F1	Kappa	QE	WSS	BSS	Total
M-Fuzzy	51	45	47	29	34	70	276
M-BBPSO	54	54	54	61	57	66	346
M-QPSO	57	56	58	73	63	56	363
SP-QPSO	62	76	69	67	30	105	409
BBPSO	67	62	67	102	111	13	422
CM-BBPSO	83	77	82	95	102	52	491
IB-Fuzzy	99	88	96	52	101	86	522
IB-QPSO	96	100	101	104	101	104	606
MG-BBPSO *	76	84	80	57	49	85	431
MG-QPSO *	73	85	77	94	67	113	509
MG-Fuzzy *	99	88	91	69	105	73	525

* No results for 2 or more datasets.

. We placed MG algorithms at the end of the table, as they had issues in two or more datasets and the sum rank may not fully reflect their performance. In these three challenging datasets, CM-BBPSO was found to be very competitive. Overall, M-Fuzzy outperformed all other techniques, acquiring the smallest sum rank across most of the evaluation metrics, followed by M-BBPSO and M-QPSO. CM-BBPSO was ranked sixth.

In

Table 6. Sum rank of algorithms based on performances on 6 metrics on CMC and Sonar datasets with best two sum rank values presented in bold.

Clus Algo	Accuracy	F1	Kappa	QE	WSS	BSS	Total
M-Fuzzy	5	5	5	4	3	13	35
CM-BBPSO	6	5	5	11	15	4	46
MG-Fuzzy	10	5	4	9	13	9	50
IB-Fuzzy	9	5	6	7	11	13	51
M-QPSO	11	10	10	11	12	7	61
M-BBPSO	13	13	13	8	10	12	69
IB-QPSO	10	16	15	17	15	12	85
SP-QPSO	13	19	19	12	6	19	88
BBPSO	17	14	15	20	20	2	88
MG-BBPSO *	12	15	14	10	7	17	75
MG-QPSO *	16	16	16	14	10	15	87

* No results for CMC.

, we calculate the sum rank for clustering performance on two datasets only, CMC and Sonar. Based on the external metrics of accuracy, F1 and Kappa, we can observe that CM-BBPSO performed best in CMC and Sonar. We placed MG-QPSO and MG-BBPSO, which had issues, at the end of the table as reference. The reason for isolating the two datasets is to identify the clustering techniques that performed well on the two datasets. Here, we observe that CM-BBPSO was the second best.

In

Table 7. Sum rank values of algorithms for all 6 metrics for each dataset with best two algorithms presented in bold.

Dataset	CM-BBPSO	M-Fuzzy	M-BBPSO	SP-QPSO	BBPSO	IB-Fuzzy	IB-QPSO	M-QPSO	MG-BBPSO	MG-QPSO	MG-Fuzzy
Appendicitis	54	12	46	30	35	29	61	32	37	47	12
CMC	20	11	33	31	39	33	44	25	36 *	36 *	33
Dermatology	36	34	23	15	25	51	56	27	42	29	57
Ecoli	36	34	32	24	50	52	37	23	26	27	52
Glass	48	27	22	20	46	52	43	28	28	27	50
Iris	44	37	33	40	10	51	33	38	19	22	57
Seeds	46	31	22	32	6	56	49	15	29	43	65
Sonar	26	24	36	57	49	18	41	36	39	51	17
WDBC	31	10	16	58	41	38	51	27	46	53	24
Wine	34	19	18	34	23	55	47	22	34	54	52
WOBC	53	16	25	15	51	25	63	33	31	48	34
Segment	35	11	18	24	28	33	46	21	36 *	36 *	36 *
Vehicle	28	10	22	29	19	29	35	36 *	28	36 *	36 *
Total	491	276	346	409	422	522	606	363 *	431 *	509 *	525 *

* Imputed rank values used.

, we study which are the best two performing algorithms for the respective datasets based on the six evaluation metrics, highlighted in bold. M-Fuzzy was the top 2 best for 7 out of 13 datasets, while M-BBPSO was top 2 for 5 datasets. Previously, without the results from CMC, Segment and Vehicle, M-BBPSO was found to be best. CM-BBPSO was top 2 for one dataset only, CMC.

In

Table 8. Comparison of CM-BBPSO with other reported cluster-based MOC, MOC and BBPSO clustering algorithms.

	CMC		WOBC		Iris		Glass		Wine
	Acc	F1	Acc	F1	Acc	F1	Acc	F1	Acc	F1
CM-BBPSO	0.402	0.371	0.896	0.865	0.721	0.672	0.429	0.351	0.719	0.691
HT-MOC (best) [20]			0.961						0.951
MOFC-TMS [25]		0.653		0.97		0.836		0.501		0.696
GenClustMOO [32]		0.647		0.969		0.823		0.503		0.698
VAMOSA [31]		0.593		0.816		0.682		0.415		0.516
MOAC [26]	0.401				0.218		0.893
ES-NMPBFO [27]	0.448	0.478			0.886	0.834
CBPSO [24]	0.398		0.961		0.908		0.509		0.711

, we compare the performance of CM-BBPSO with other reported algorithms in terms of Accuracy and F1 score for five UCI datasets [15]: CMC, WOBC (Cancer), Iris, Glass and Wine. Where results are not available, we left it blank. While CM-BBPSO was not found to be the best in any of the datasets, it was found to outperform certain algorithms in a few datasets. CM-BBPSO outperformed the following algorithms in the respective datasets: MOAC in CMC and Iris, VAMOSA in WOBC and Wine and CBPSO in CMC. This demonstrates that CM-BBPSO performs competitively, and this result is promising. However, HT-MOC, MOFC-TMS, GenClustMOO and ES-NMPBFO outperformed CM-BBPSO in the available common datasets.

4.3. Investigating Swarm Size and Maximum Number of Iterations

In

Table 9. Performances of CM-BBPSO using different swarm size (s) and maximum iteration (i), s-i with the best result produced by the s-i setting for each dataset presented in bold.

	Appendicitis		CMC		Dermatology		Ecoli
s-i	Accuracy	QE	Accuracy	QE	Accuracy	QE	Accuracy	QE
20-100	$0.771\pm 0.08$	$2.56\pm 0.43$	$0.402\pm 0.02$	$\mathbf{2}.\mathbf{88}\pm \mathbf{0}.\mathbf{25}$	$\mathbf{0}.\mathbf{651}\pm \mathbf{0}.\mathbf{07}$	$5.31\pm 0.31$	$0.499\pm 0.07$	$1.97\pm 0.33$
30-100	$0.775\pm 0.08$	$2.61\pm 0.38$	$0.396\pm 0.02$	$3.00\pm 0.24$	$0.619\pm 0.09$	$5.35\pm 0.34$	$0.501\pm 0.06$	$2.10\pm 0.42$
30-200	$\mathbf{0}.\mathbf{781}\pm \mathbf{0}.\mathbf{07}$	$2.57\pm 0.50$	$0.402\pm 0.02$	$2.94\pm 0.18$	$0.632\pm 0.09$	$5.38\pm 0.36$	$0.497\pm 0.08$	$\mathbf{1}.\mathbf{92}\pm \mathbf{0}.\mathbf{19}$
40-100	$0.750\pm 0.11$	$\mathbf{2}.\mathbf{49}\pm \mathbf{0}.\mathbf{30}$	$\mathbf{0}.\mathbf{405}\pm \mathbf{0}.\mathbf{02}$	$2.94\pm 0.24$	$0.623\pm 0.08$	$\mathbf{5}.\mathbf{30}\pm \mathbf{0}.\mathbf{347}$	$\mathbf{0}.\mathbf{514}\pm \mathbf{0}.\mathbf{06}$	$2.00\pm 0.19$
	Glass		Iris		Seeds		Segment
s-i	Accuracy	QE	Accuracy	QE	Accuracy	QE	Accuracy	QE
20-100	$0.429\pm 0.04$	$2.75\pm 0.45$	$0.721\pm 0.10$	$\mathbf{1}.\mathbf{41}\pm \mathbf{0}.\mathbf{29}$	$0.727\pm 0.12$	$2.1\pm 0.38$	$0.547\pm 0.06$	$3.15\pm 0.41$
30-100	$\mathbf{0}.\mathbf{431}\pm \mathbf{0}.\mathbf{04}$	$2.72\pm 0.48$	$0.704\pm 0.12$	$1.49\pm 0.30$	$0.707\pm 0.13$	$2.14\pm 0.41$	$0.542\pm 0.07$	$\mathbf{3}.\mathbf{08}\pm \mathbf{0}.\mathbf{35}$
30-200	$0.419\pm 0.04$	$\mathbf{2}.\mathbf{65}\pm \mathbf{0}.\mathbf{40}$	$0.706\pm 0.09$	$1.51\pm 0.32$	$0.721\pm 0.11$	$2.11\pm 0.39$	$0.546\pm 0.06$	$3.17\pm 0.45$
40-100	$0.412\pm 0.05$	$2.80\pm 0.43$	$\mathbf{0}.\mathbf{746}\pm \mathbf{0}.\mathbf{11}$	$1.45\pm 0.30$	$\mathbf{0}.\mathbf{750}\pm \mathbf{0}.\mathbf{13}$	$\mathbf{2}.\mathbf{05}\pm \mathbf{0}.\mathbf{38}$	$\mathbf{0}.\mathbf{553}\pm \mathbf{0}.\mathbf{05}$	$3.12\pm 0.44$
	Sonar		Vehicle		WDBC		Wine
s-i	Accuracy	QE	Accuracy	QE	Accuracy	QE	Accuracy	QE
20-100	$0.552\pm 0.05$	$9.13\pm 0.70$	$0.374\pm 0.019$	$\mathbf{3}.\mathbf{66}\pm \mathbf{0}.\mathbf{59}$	$0.782\pm 0.13$	$6.08\pm 1.07$	$\mathbf{0}.\mathbf{719}\pm \mathbf{0}.\mathbf{15}$	$\mathbf{3}.\mathbf{5}\pm \mathbf{0}.\mathbf{25}$
30-100	$0.553\pm 0.05$	$\mathbf{9}.\mathbf{05}\pm \mathbf{0}.\mathbf{63}$	$\mathbf{0}.\mathbf{385}\pm \mathbf{0}.\mathbf{02}$	$3.86\pm 0.89$	$\mathbf{0}.\mathbf{811}\pm \mathbf{0}.\mathbf{11}$	$\mathbf{5}.\mathbf{92}\pm \mathbf{0}.\mathbf{92}$	$0.712\pm 0.13$	$3.57\pm 0.31$
30-200	$\mathbf{0}.\mathbf{554}\pm \mathbf{0}.\mathbf{05}$	$9.72\pm 1.02$	$0.377\pm 0.02$	$3.7\pm 0.65$	$0.808\pm 0.10$	$5.95\pm 0.92$	$0.713\pm 0.11$	$3.62\pm 0.27$
40-100	$0.533\pm 0.03$	$9.28\pm 0.75$	$0.378\pm 0.02$	$3.73\pm 0.65$	$0.763\pm 0.12$	$6.2\pm 1.11$	$0.694\pm 0.14$	$3.54\pm 0.25$
	WOBC
s-i	Accuracy	QE
20-100	$0.896\pm 0.10$	$2.87\pm 0.62$
30-100	$0.917\pm 0.08$	$2.64\pm 0.37$
30-200	$0.915\pm 0.09$	$\mathbf{2}.\mathbf{54}\pm \mathbf{0}.\mathbf{21}$
40-100	$\mathbf{0}.\mathbf{927}\pm \mathbf{0}.\mathbf{06}$	$2.64\pm 0.31$

, the performance of CM-BBPSO, using the following swarm-size and maximum iteration pair settings ss-mi, 20–100, 30–100, 30–200 and 40–100, is presented. Experiments with ss-mi of 40–100 achieved the best results in 6 out of 13 datasets. Further experimentation using this setting will be considered. Overall, the results were considered competitive for the different settings.

4.4. Visual Examination of CM-BBPSO Clustering

We visually inspect CM-BBPSO clustering by comparing T-SNE biplots with the ground truth. The biplots are generated using R package $rtsne$, and a perplexity value of 40 is applied to all datasets presented. In Figure 1a,e, we observe that clusters of CMC and Sonar highly overlap, but CM-BBPSO was able to cluster correctly some data points located between clusters. In Figure 2a–f, the ground truth and CM-BBPSO clustering biplots of challenging datasets, Glass, Segment and Vehicle, are presented. For contrast, we present the biplot for Iris in Figure 2h to show that CM-BBPSO can easily find the clusters. Where clusters do not overlap, CM-BBPSO performs with no issues, though its strength is in highly overlapping clusters.

In Figure 3, convergence plots of connectedness values of particles and their pbest against the number of iterations for particles using CM-BBPSO on the Appendicitis dataset are presented to show the convergence and, thus, the stability of CM-BBPSO. We observe that the connectedness values of particles and their pbest converge on all four experiment settings. It is interesting to note that fewer particles dominate the pbest value as the algorithm reaches near the maximum iteration.

5. Discussion

From the various comparative analysis, it is evident that CM-BBPSO does not perform better than the other algorithms on datasets with well-separated or a small number of clusters, such as Iris, Wine, WOBC and WDBC. Instead, it performs well on highly overlapping clusters, such as Sonar and CMC. Although other MOC results, both from our previous work and from other reported work, outperform CM-BBPSO on Sonar and CMC, we consider the CM-BBPSO to be promising. It used a simple clustering with two clusters for MOEA, and the results are not a lot worse. There was also no setting of parameters apart from the number of clusters, swarm size and maximum iteration. Excluding population size, maximum k and total generations, HT-MOC requires the parameters of neighborhood size, upper limit, branching factor and length of minimum spanning tree links. Excluding the number of objects, total generation and crossover, MOFC-TMS requires the degree of the membrane system.

With the 2-cluster K-means cluster-based approach in MOEA/D, the local search is introduced such that the evolved candidate is evaluated with the cluster closest to it during the dominance check. This brought on the effect of balancing the compactness and well-separatedness of the cluster as shown in Table 1. Further experimentation is required to study the effects of clustering on solution selection and whether such effects are consistent across other MOC algorithms.

From Table 2, Table 3 and Table 4, we observed that M-Fuzzy works well with the WSS as one of the objectives to achieve good accuracy. Examining all WSS and accuracy results of MOC algorithms from the previous and current work, those with best accuracy also produce the best WSS for 7 out of 13 datasets. Two thoughts came to mind: first, the objective functions in Fuzzy and BBPSO clustering. All BBPSO objective functions are based on [23]. Using the same objective function as that of Fuzzy clustering on BBPSO clustering may improve it. Second is the objective functions in MOEA. The objective functions WSS and Connectivity in MOEA may not be the best combination for all datasets, and more objective functions can be considered. There is consideration to study the same framework, but with different objective functions [34]. HT-MOC used variance and connectivity, while MOFC-TMS used $J_m$, XB-index and Sym-index. With three objective functions, MOFC-TMS produced good results for challenging datasets, such as CMC and Glass. To test on datasets with highly overlapping clusters, HT-MOC was tested on CoverType and Shuttle datasets. This will be considered in our further investigations.

So far, we applied the cluster-based approach only on MOEA/D BBPSO clustering. From Table 5, despite having issues on the CMC dataset, M-Fuzzy was found to produce good results; further investigation of applications of the cluster-based technique on it can be considered. The latest cluster-based MOEA techniques were investigated in non-convex clusters in objective space, which is of interest to our future investigation in clustering problems. Furthermore, we would like to investigate adding the cluster representatives in our reproduction pool to guide the evolution of new candidates, which was conducted in [13], but we did not apply it in this manner. By including cluster representatives, this can increase diversity while applying local search in the objective space during the dominance check. The results from the current work is promising, and this work has opened up potential research directions in applying cluster-based MOEA techniques for data clustering, using as few parameters as possible.

5.1. Implications

The application of clustering on fitness values in the objective space during the dominance check of solution evaluation in MOC added a layer of search control, which affected the properties of the clustering in the decision space as observed in the clusters’ compactness and well-separatedness characteristics of CM-BBPSO, M-BBPSO and BBPSO in their WSS and BSS values.

5.2. Insights

The local search introduced by cluster-based selection approach can be further manipulated through a more advanced design of clustering techniques in MOC to generate refined clusters that can influence the evolution and selection of new candidates.

5.3. Limitations

The following are limitations of this work. Where improvements were found when comparing with MOC from our previous work, CM-BBPSO did not demonstrate significantly large improvements. This could possibly be due to the conservative cluster number of two used in this study. So far, we applied the cluster-based approach only on MOEA/D BBPSO clustering. From Table 5, despite having issues on the CMC dataset, M-Fuzzy was found to produce good results. Further investigation of applications of the cluster-based technique on it can be considered. Though the latest cluster-based MOEA techniques were investigated in non-convex clusters in the decision space, we did not cover in this work which is of interest to our future investigation. Furthermore, we would like to investigate adding cluster representatives in our reproduction pool to guide the evolution of new candidates, which was conducted in [13], but we did not apply it in this manner. By including cluster representatives, this can increase diversity while applying local search in the decision space.

5.4. Future Work

Based on our discussion and limitations, we consider the following future work and direction. We will investigate the effect of more clusters and a larger swarm size, such as 40. The investigation of non-convex clusters in the decision space through cluster algorithm design and the addition of cluster representatives in our reproduction pool to guide the evolution of new candidates will be further studied, as the inclusion of cluster representatives in the population pool can increase diversity while applying local search in the decision space. As we used a different clustering objective for M-BBPSO as opposed to M-Fuzzy which performed very well, we will be investigating the objective functions in both the clustering of decision space and MOO parts as well as the effect of adding more objective functions to MOC. As differential evolution (DE) is gaining popularity as an effective swarm intelligence technique, we would like to apply DE in MOC and compare with BBPSO and Fuzzy techniques. To increase rigor in our testing, we will consider including two test datasets with highly overlapping clusters, CoverType and Shuttle datasets, used in testing HT-MOC.

6. Conclusions

The cluster-based multiobjective evolutionary algorithm based on decomposition to find gbest in a bare bones particle swarm optimization clustering framework was developed, tested extensively on 13 UCI datasets and compared with 10 MOC algorithms from a previous study. CM-BBPSO produced competitive results and better results in challenging datasets with highly overlapping clusters that are hard to distinguish, such as CMC, Sonar and Ecoli, when compared with other BBPSOs.

The comparison with MOC from our previous work showed that CM-BBPSO produces competitively good results and is ranked highly on datasets with difficult to distinguish clusters, such as CMC and Sonar.

Through the analysis of the WSS and BSS values, we observed that the effect of clustering in MOEA balances between compactness and well-separatedness characteristics, whereas M-BBPSO and BBPSO highly prioritize one over the other. This demonstrates that the clustering adds another layer of search control in the selection of candidates in MOEA, meaning that a local search is introduced during the dominance evaluation of new solutions against neighbors in the objective space. This opens up new research directions to explore novel local search, using clustering in the objective space of MOC.

Based on a comparative study of the results and methodologies in the reported literature, we found the results of our proposed technique, which uses almost no parameters, to be promising and have identified several areas for future investigation, such as in applying a cluster-based approach to M-Fuzzy, increasing k, and investigation of the choice of objective functions. The proposed technique, though it did not produce the best results, outperformed two of the reported MOC algorithms, VAMOSA and MOAC, on several datasets. We demonstrated that, using simple techniques, such as MOEA/D, BBPSO and K-means, the proposed technique is able to perform competitively.

The results from different experimental settings of swarm size and maximum iteration applied on CM-BBPSO showed that those with a smaller swarm size and maximum iteration of 20 and 100, respectively, can achieve good results, which can also be examined visually using the biplots. Furthermore, the convergence plots of CM-BBPSO showed that the algorithm stabilizes as it reaches maximum iterations.

With the application of advanced techniques or additional objective functions, such as those found in HT-MOC and MOFC-TMS, novel cluster-based MOEA can be developed to solve complex clustering problems. We hope that the discussion of this work, together with the extensive comparisons against existing MOCs across 13 datasets and 6 metrics, can provide a useful reference and demonstration of the feasibility and effectiveness of CM-BBPSO clustering to new or interested practitioners of BBPSO clustering, MOC and cluster-based MOC algorithms.