A Selection Process for Genetic Algorithm Using Clustering Analysis

Abstract

This article presents a newly proposed selection process for genetic algorithms on a class of unconstrained optimization problems. The k-means genetic algorithm selection process (KGA) is composed of four essential stages: clustering, membership phase, fitness scaling and selection. Inspired from the hypothesis that clustering the population helps to preserve a selection pressure throughout the evolution of the population, a membership probability index is assigned to each individual following the clustering phase. Fitness scaling converts the membership scores in a range suitable for the selection function which selects the parents of the next generation. Two versions of the KGA process are presented: using a fixed number of clusters K (KGA_f) and via an optimal partitioning K_opt (KGA_o) determined by two different internal validity indices. The performance of each method is tested on seven benchmark problems.

1. Introduction

The fields of computational intelligence and optimization algorithms have grown rapidly in the past few decades. Classical methods are not efficient in solving current problems in engineering such as energy, transportation and management [1]. The development of these optimization algorithms can be mainly divided into deterministic and stochastic approaches [2].

Most conventional algorithms are deterministic, such as gradient-based algorithms that use the function values and their derivatives. These methods work extremely well for smooth unimodal problems, but in the case of some discontinuities, non-gradient algorithms are preferred [3]. Nelder–Mead downhill simplex [4] and Hooke–Jeeves pattern search technique [5] are a few examples of deterministic gradient-free algorithms. For stochastic algorithms, we have two types: heuristic and meta-heuristic. Although there is no agreed definition of each type in the literature, the aim of stochastic methods is to find feasible solutions in a satisfactory timescale. There is no guarantee that the best solutions can be found; however, it is expected that the algorithm will provide nearly optimal solutions most of the time.

In this paper, we propose a genetic algorithm (GA)-based algorithm that uses clustering analysis to organize the population and select the parents for recombination. Cluster analysis is the study of techniques and algorithms to organize data into sensible groupings (clusters) according to measured or apparent similarities [6]. Clustering has been successfully applied in various engineering and scientific disciplines such as biology, medicine, machine learning, pattern recognition, image analysis and data mining [7,8]. The performance of a newly proposed selection process named the k-means genetic algorithm selection process is investigated on a class of unconstrained optimization problems. The KGA technique is composed of four essential stages: clustering, membership phase, fitness scaling and selection. The authors postulate that clustering the evolving population can help preserve a continuous selection pressure throughout the evolution process. A membership probability index is allocated to each individual following the clustering phase. Fitness scaling alters the membership scores into a range suitable for the selection function, which selects the parents of the succeeding generation. Two versions of the KGA technique are examined: using a fixed number of clusters K (KGA_f) and via an optimal number of clusters K_opt (KGA_o). The performance of each method is tested on eight benchmark problems. The numerical simulations reveal that the proposed selection process is superior to or competitive with the standard GA for the given problems.

The remainder of the paper is organized as follows: The next section summarizes some relevant studies that have explored clustering analysis in optimization algorithms. Section 3 presents important definitions on genetic algorithm. Section 4 presents the selection processes (KGA_f and KGA_o) proposed in this paper. Section 5 contains the numerical simulations where the performance of the proposed approaches is demonstrated. Finally, Section 6 concludes with some final remarks and possible future contributions.

2. Literature Review

There are many algorithms that have been proposed in literature to solve the clustering problems. Some relevant studies that have explored the problem of clustering using various approaches include evolutionary algorithms such as evolutionary programing [9], particle swarm optimization [10,11,12], ant colony algorithms [13,14], artificial bee colony [15], simulated annealing [16,17] and tabu search [18]. Conversely, there have been many attempts to use GAs to solve clustering applications [7,19,20,21,22,23,24,25,26,27]. Maulik and Bandyopadhyay [21] proposed a GA approach to clustering. They tested the performance of the algorithm on synthetic and real-life data sets. The GA-k-means algorithm was used to search for the cluster centres which minimize the clustering metric, showing results significantly superior to those of the k-means algorithm. Another genetic algorithm approach, the genetic k-means algorithm, was presented by Krishna and Murty [7]; they defined a mutation operator specific to clustering problems. Recently, a novel optimization algorithm was proposed by Krishnasamy [28] referred to as K-MCI, inspired by the natural and social tendency of cohort individuals to learn from one another.

Since the novelty of the proposed algorithm revolves around the notion of introducing clustering analysis in the selection stage of the genetic algorithm, this section will avoid a survey of clustering techniques. The reader is referred to [29,30,31] for detailed surveys of clustering algorithms. Consequently, in the remainder of this section, we will review the most relevant optimization algorithms that have introduced clustering analysis in one way or another.

In the process of genetic differentiation, the population subdivided was discussed in the literature. For instance, the island model [32] divides the population into discrete finite races, between which some migration occurs. The hypothesis is that multiple subpopulations help preserve a better genetic diversity, since each island can potentially follow a different search trajectory through the search space. Various “islands” conserve some degree of independence and therefore explore different regions of the search space while sharing some information by migration. On the other hand, various niching methods have been introduced into GAs to promote the formation of stable sub-populations in neighborhood of optimal solutions [33]. There are many commonly adopted techniques, such as deterministic crowding [34], sharing [35], clearing [36] and dynamic niche clustering [37]. Standard and deterministic crowding both suffer from genetic drift. In sharing and clearing methods, prior knowledge about the fitness landscape is required to set the niche radius. The set of cluster numbers in dynamic niche clustering will largely affect the quality and quantity of optimal solutions.

Many researchers have investigated evolutionary algorithms for dynamic optimization problems (DOPs) because EAs are fundamentally inspired from biological evolution, which is always subject to an ever-varying environment. From the literature on DOPs, the traditional approaches use the multi-population method to find the optimum solutions for multi-modal functions. The core notion is to divide the search space into different sub-spaces, and then separately search within these sub-spaces. The challenge with these multi-population methods (such as [38,39,40]), is how to choose an appropriate number of sub-populations to cover the entire search space. Three major difficulties arise using multi-population methods: how to guide the particles towards different promising sub-regions, how to define sub-regions and how many sub-populations are required. In order to overcome these questions, a clustering particle swarm optimizer (CPSO) was proposed in [38,41]. In the CPSO algorithm, a proper number of sub-swarms which cover different local regions are created using a clustering method. A hierarchical clustering method is used to locate and track multiple optima and a fast-local search method is employed to find the near optimal solutions in a promising region in the search space. Kennedy [42] originally proposed a PSO algorithm that uses a k-means clustering algorithm to identify the centers of different clusters of particles in the population, and then uses the centers to substitute the personal best or neighborhood best positions. The limitation of this approach lies in that the number of clusters must be predefined. Similarly, a fuzzy clustering-based particle swarm (FCPSO) algorithm was proposed in [43] to solve multiobjective environmental/economic dispatch. The clustering in the FCPSO technique ensures that the obtained Pareto front will have uniform diversity at all stages of the search.

In [44], clustering analysis was applied to adjust the probabilities of crossover p_x and mutation p_m in GAs. By applying the k-means algorithm, the population is clustered in each generation and a fuzzy system is used to adjust the values of the genetic operators. Regulations are based on considering the relative size between the clusters holding the best and worst chromosomes respectively.

Zhang et al. [45] tackled the problem of large-scale many-objective optimization problems based on a decision variable clustering method. The proposed technique divides the decision variables into two clusters based on the features of each variable. The decision variable clustering method adopts the k-means method to divide the decision variables into two types: convergence-related variables and diversity-related variables.

Recently, a self-organizing multiobjective evolutionary algorithm [46] was evaluated on some state-of-the-art multiobjective evolutionary methods. A local PCA partitions the given population into several disjointed clusters, and conducts PCA in each cluster to extract a continuous manifold and build a probabilistic model.

3. Problem Definition

In essence, the basic objective of any clustering algorithm is to find a global or approximate optimal for combinatorial optimization problems which are NP-hard [47]. The k-means algorithm is very likely to converge to a suboptimal partition. The main advantage of stochastic optimization techniques over deterministic-methods is that they are able to avoid convergence to a local optimal solution. Therefore, stochastic approaches have been employed to solve clustering problems; algorithms such as simulated annealing, genetic algorithms, evolution strategies and evolutionary programming. Inspired by the principles of natural selection and biological evolution, evolutionary algorithms seek to optimize a population of individuals by applying a set of evolutionary operators. They are population-based meta-heuristic optimization algorithms that make use of biological evolution operators such as selection, recombination and mutation.

In order to demonstrate the novelty in the use of clustering analysis in the selection process of the genetic algorithm, the performance of the proposed KGA techniques will be compared with existing GA methods. They were originally proposed by Holland [48], inspired by the principle of natural selection of biological systems or ‘Darwinian evolution’. GAs have demonstrated their capability to solve a wide range of optimization problems such as revenue management, optimal engineering system designs, scheduling applications, image processing, quality control etc. John Holland essentially laid the foundation of modern evolutionary computing by fundamentally defining three key genetic operators: crossover, mutation, and selection. These evolutionary operators provide a way to generate offspring from parent solutions.

We summarize the fundamental steps of genetic algorithms in Algorithm 1. In genetic algorithms, each individual (or solution vector) is encoded as either a binary bit string or a real-value vector, both referred to as a chromosome. The standard representation of each individual is a binary array of bits, to facilitate the crossover and mutation operations.

Algorithm 1: Given the function $f(\overrightarrow{\mathbf{x}})$, $\overrightarrow{\mathrm{x}}={\left({\mathrm{x}}_1,\dots ,{\mathrm{x}}_{\mathrm{d}}\right)}^{\mathrm{T}}$ to minimize

(a) Encode the solutions into a set of chromosomes (b) Generate the initial population (c) Initialize the crossover and mutation probabilities (d) Evaluate fitness function of each individual (e) Selection of the current best for the next generation (f) Reproduction by crossover and mutation (g) Update $t=t+1$ (h) Repeat (d) → (g) if t < Max number of generations OR Stopping criteria is met (i) Display the optimal solution ${\overrightarrow{x}}^{*}$

An initial population is generated according to a heuristic rule or randomly. The population size typically depends on the nature of the optimization problem. Often, the initial population is generated in such a way as to allow a larger range of possible solutions inside the given search space. If the population size is too small, there is not enough evolution going on and consequently there is a risk of premature convergence towards a local optimum and ultimately extinction of the population. However, a larger population will require more computational time and fitness evaluations.

At each successive generation, a percentage of the existing population is ‘selected’ to breed a new generation, thus ensuring the continuity of the population. Thus, a selection function chooses ‘parents’ based on a fitness-based selection process, where ‘fitter’ solutions are more likely to be selected. An individual can be selected more than once, in which case it transfers its genes to more than one offspring.

At each generation, the GA uses the current generation to create the new offspring that will define the next generation. The algorithm will apply a set of genetic operators (crossover and mutation) on the parents selected by the selection function to generate the children. Recombination (or crossover) is the combination of a pair of parents, analogous to biological reproduction. Mutated children are created by a random change (or mutation) of the genes of a single parent. Both genetic operators are essential for the success of the optimization search. Crossover enables the algorithm to preserve the best genes from different individuals and recombine them into possibly fitter children. This allows a better ‘exploitation’ of the search space. Whereas mutation increases the diversity of the population and permits a further ‘exploration’ of the search domain. The crossover probability is usually between 0.7 and 1.0, while the mutation probability is lower 0.001~0.05. Mutation probability is dependent upon the representation type and number of genes. For instance, for an n bit representation, the suggested mutation rate is 1/n. In natural systems, if the mutation rate is too high under a high selection pressure, the population might become extinct. A suitable elitist selection function must be employed to avoid loss of good solutions. Selection, crossover and mutation are iteratively applied to the population until a stopping condition is satisfied.

Introducing the concept of clustering analysis in an evolutionary algorithm is inspired from the notion that clustering the evolving population can help avoid excessive exploitation and therefore escape local optimum (local minimum or local maximum). The role of clustering analysis is to improve the probability of discovering the global optimum by sufficiently covering the solution space (exploration) yet ensuring sufficient pressure to obtain even better solutions from current individuals (exploitation).

Furthermore, in practical multimodal domains problems, it is desirable to evaluate several global optima or some local optima that might be suitable alternatives to the global optima. Traditional genetic algorithms perform well on single optimum problems but fail to provide multiple solutions. By combining the strength of clustering analysis and genetic search, the proposed KGA techniques permit the evaluation of multimodal functions. A detailed explanation of the proposed KGA technique is considered in the next section.

4. The Proposed Algorithm

The following section presents a brief description of the proposed k-means genetic selection processes. We are interested in the unconstrained optimization problems in which we attempt to find $\overrightarrow{x^{*}}$ which optimizes $f(\overrightarrow{x})$ using GA-based algorithms. Therefore, only the standard GA will be used to test the performance of the proposed KGA algorithms. Below, two distinct selection techniques KGA_f and KGA_o are presented.

4.1. KGA_f

The proposed KGA is different to the standard GA in several ways. Primary, the chromosomes of the population are partitioned into groups in such a way that all individuals inside the same cluster are similar. This offers a novel approach to solve the two important issues in the evolution process of the genetic search: exploitation and exploration. Exploration is responsible for population diversity by exploring the search space, while exploitation attempts to reduce the diversity by focusing on individuals with higher fitness scores. Strong exploitation encourages premature convergence of the genetic search. Recombining individuals inside the same cluster reduces population diversity, and thus clustering the population can allow an enhanced balance between exploitation and exploration.

The aim of clustering is to find a given structure among the series of data and is therefore exploratory in nature [49]. The task of organizing a set of data using cluster analysis requires some dissimilarity measurement among the set of patterns. The dissimilarity metric is defined according to the nature of the data and the purpose of the analysis. Many types of clustering algorithms have been proposed; the reader is referred to [49,50,51,52] for a taxonomy of clustering techniques, discussions on major challenges and key issues and useful surveys of recent advances. The simplest and most popular clustering algorithm is the k-means algorithms (KMA), and was originally published by Steinhaus [53] in 1956. Even though it was first proposed 60 years ago, it is still the most widely used algorithm for clustering.

A general definition of clustering can be stated as follows: given a set of data composed on n objects, find K groups in such a way that the measure of similarities between objects in the same group is low while the similarities between objects in different groups are high.

The k-means algorithm attempts to find a partition such that the squared error between the empirical mean of a cluster and the objects in the cluster is minimized. The goal is to minimize the sum of the squared error J over all K clusters, as follows:

$$J\left(X,C\right)={\displaystyle \sum }_{k=1}^K{\displaystyle \sum }_{x_i\in c_k}{\Vert x_i-{\mu }_k\Vert }^2$$

(1)

where $\mathrm{X}=\left\{x_i\right\},i=1,\dots ,N$ is the set of N d-dimensional points to be clustered into K clusters, $\mathrm{C}=\left\{c_k\right\},k=1,\dots ,K$ and ${\mu }_k$ the mean of cluster ${\mathrm{c}}_k$.

Minimizing the k-means objective function is an NP-hard problem (even for K = 2) [54], and therefore the algorithm can only converge to local minima. The main steps of the k-means algorithm can be summarized as follows:

1.

Choose an initial partition with K clusters.

2.

Generate a new partition by assigning each pattern to its nearest cluster centroid.

3.

Compute new cluster centroids.

4.

If a convergence criterion is not met, repeat steps 2 and 3.

In the KGA_f algorithm, the number of clusters is kept the same throughout the evolution process. The four main stages of KGA_f are as follows:

5.

Clustering the population by the k-means algorithm

6.

Computing the membership probability (MP) vector (Equations (2)–(4))

7.

Fitness scaling of MP

8.

Selection of the parents for recombination.

In general, we want to maintain an even selection pressure during the evolution of the genetic search. At the beginning, the search may be bias towards high fitness individuals. Near the end of the search, as the population is converging towards an optimal solution, there may not be much separation among individuals in the population. Neither situation is desirable, thus there is a necessity to scale the fitness in such a manner to keep the selection pressure the same throughout in the population.

The membership probability score of an individual is a measurement of its affiliation with respect to both designated and external clusters (Equation (2)). For a given solution i inside a cluster j of size m_j, the membership probability index is calculated as follows:

$$MP\left(i,j\right)=\frac{m_j}{m_j-1}\times \frac{1}{P}\times \frac{S_j-f\left(x_i\right)}{S_j}$$

(2)

and

$$S_j={\displaystyle \sum }_{i=1}^{m_j}f\left(x_i\right)$$

(3)

$$P={\displaystyle \sum }_{j=1}^K{m}_j$$

(4)

where P is the population size and $S_j$ is the sum of the fitness values $f\left(x_i\right)$ inside cluster j.

The key characteristics that are associated with the use of the membership probability function are the following:

• The sum of the membership probability scores of a given cluster j of size m_j is equal to $\frac{m_j}{P}$. Consequently, clusters with more individuals will be attributed a larger probability sum.
• An individual with a lower fitness value $f\left(x_i\right)$ inside a cluster of size m_j is awarded a higher MP score. This is translated in the $\frac{S_j-f\left(x_i\right)}{S_j}$ term, thus allocating fitter solutions a higher probability of selection.
• In order to reduce the probability of recombination between individuals from the same cluster, thus avoiding local optimal traps, fitter individuals in smaller clusters are awarded a higher MP score. This is the direct effect of $\frac{m_j}{m_j-1}$ term.
• The sum of all membership probability scores is equal to one.

Fitness scaling converts the membership scores in a range suitable for the selection function which selects the parents of the next generation. The selection function allocates a higher probability of selection to individuals with higher scaled values.

The range of the scaled values can affect the performance of the genetic algorithm. If the scaled values vary too extensively, higher scaled value individuals will reproduce too rapidly and prevent the GA from searching other regions in the search space. In contrast, for lower scaled value variations, all individuals will have an equal chance of reproduction and therefore will result in very slow search progress.

The general framework of the proposed KGA_f algorithm is shown in Figure 1 below.

4.2. KGA_o

It is obvious that a problem we face in the KGA_f algorithm is to decide the optimal number of clusters. Visual verification of a large multidimensional data set (e.g., more than three) is difficult [55]. In order to find the optimal clustering scheme that best fits the inherent partitions of the data set, the concept of clustering validation has been subject to numerous research efforts. The fundamental concepts, drawbacks and applications of clustering validation techniques were discussed in [55,56,57,58].

In essence, there are three main approaches to examine cluster validity:

• External criteria: evaluation of the clustering algorithm results is based on previous knowledge about data.
• Internal criteria: clustering results are evaluated using a mechanism that takes into account the vectors of the data set themselves and prior information from the data set is not required.
• Relative criteria: aim to evaluate a clustering structure by comparing it to other clustering schemes.

The KGA_o attempts to answer the following questions:

• In how many clusters can the population be partitioned to?
• Is there a better “optimal” partitioning for our evolving population of chromosomes?

Two main approaches to determining the suitable number of clusters for a given data set can be distinguished:

• Compatible Cluster Merging (CCM): starting with a large number of clusters, and successively reducing the number by merging clusters which are similar (compatible) with respect to a similarity criterion.
• Validity Indices (VI’s): clustering the data for different values of K, and using validity measures to assess the obtained partitions.

The CCM approach requires more computational operations than the use of a validity index to determine the optimal number of clusters. Moreover, the size of the evolving population is small (less than or equal to 100 chromosomes), therefore there is no need to apply a CCM approach. On the other hand, the validity index is not a clustering algorithm itself, but rather a measurement of the results and thus suggests a scheme that best fits the data set. At each generation in the proposed KGA_o technique, the optimal number of clusters is calculated using a validity assessment index. Different validity indices suitable for k-means clustering have been proposed in the literature.

In this paper, two different internal validity indices are applied in the KGA_o technique: Silhouette [59] and the Davies–Bouldin index [60] as explained in Figure 2 below.

• Silhouette (S) [59]

The silhouette technique assigns to the ith vector of cluster c_j (j = 1, …, K), a quality measure $s\left(i\right)$ known as the silhouette width defined as S:

$$s_j=\frac{1}{m_j}{\displaystyle \sum }_{i=1}^{m_j}\frac{\left(b\left(i\right)-a\left(i\right)\right)}{\max \left[a\left(i\right),b\left(i\right)\right]}$$

(5)

and

$$S=\frac{1}{K}{\displaystyle \sum }_{j=1}^K{s}_j$$

(6)

where a(i) is the average distance between the ith vector and the remaining elements inside the same cluster j of size $m_j$, b(i) is the minimum average distance between vector i and all elements inside clusters c_k (k = 1, … K; k ≠ j). The optimal partition is expected to minimize the intra-group distance a while maximizing the inter-group distance b, thus maximizing the silhouette width criterion S.

• Davies–Bouldin (DB) [60]

The DB index aims to evaluate intra-cluster similarity and inter-cluster differences by computing the following:

$$BD=\frac{1}{K}{\displaystyle \sum }_{i=1}^K{\max }_{i\ne j}\left[\frac{d\left(x_i\right)+d\left(x_j\right)}{d\left(c_i,c_j\right)}\right]$$

(7)

where $d\left(x_i\right)$ and $d\left(x_i\right)$ are each the sum of all the distances between the centroid of the cluster and the elements of clusters i and j respectively, $d(c_i,c_j)$ is the distance between centroids of cluster $c_i$ and $c_j$. A good partition composed of compact and separated clusters is represented by a small DB value. The Davies–Bouldin index presents decent results for dissimilar groups. However, it is not intended to handle overlapping clusters [27].

Throughout the KGA_o technique, the evaluation of the validity index function is performed within a range of cluster numbers and then an optimal number is chosen. For instance, if the Silhouette index is applied, the number of clusters which maximizes S corresponds to the optimal partition, whereas the minimum DB value determines the optimal number of clusters for the clustering of the population. Since the size of the population is small, the maximum number of partitions is set to ten. Consequently, the search for the optimal partition varies between i = 2 (the minimum number of clusters) and i = 10, as per Figure 2.

5. Numerical Simulations

In this section, the performance of KGA techniques on seven well-known test functions is investigated. In recent years, various kinds of novel computational intelligence methods have been proposed and the field is attracting more and more attention. To promote research on expensive optimization, the CEC 2014 special session competition developed a set of benchmark optimization problems.

All test functions are minimization problems defined as follows:

$$\min f\left(\mathrm{x}\right); \mathrm{x}=\left[x_1,x_2,\dots ,x_D\right]$$

(8)

D dimension of the search space.

Most functions are shifted by $o_i=\left[o_{i1},o_{i2},\dots ,o_{iD}\right]$, randomly distributed in ${\left[-10,10\right]}^D$. Some problems are rotated by a predefined rotation matrix M (

Table 1. Summary of the test functions.

No.	Functions	Search Ranges	${\mathit{f}}_{\mathit{i}}^{*}={\mathit{f}}_{\mathit{i}}\left({\mathit{x}}^{*}\right)$
1	shifted sphere	$\left[-20,20\right]$	0
2	shifted ellipsoid	$\left[-20,20\right]$	0
3	shifted and rotated ellipsoid	$\left[-20,20\right]$	0
4	shifted step	$\left[-20,20\right]$	0
5	shifted Ackley	$\left[-32,32\right]$	0
6	shifted Griewank	$\left[-600,600\right]$	0
7	shifted rotated Rosenbrock	$\left[-20,20\right]$	0

). Each rotation matrix is generated from standard normally distributed entries by Gram–Schmidt ortho-normalization with condition number equal to one or two.

• $f_1\left(x\right)={{\displaystyle \sum }}_{i=1}^D{x}_i^2$ $F_1\left(x\right)=f_1\left(x-o_1\right)$
• $f_2\left(x\right)={{\displaystyle \sum }}_{i=1}^{D}i{x}_i^2$ $F_2\left(x\right)=f_2\left(x-o_2\right)$
• $F_3\left(x\right)=f_2\left(M_3\left[x-o_3\right]\right)$
• $f_3\left(x\right)={{\displaystyle \sum }}_{i=1}^D{\left|x_i+0.5\right|}^2; F_3\left(x\right)=f_3\left(x-o_4\right)$
• $f_4\left(x\right)=-20\exp \left[-0.2\sqrt{\frac{1}{D}{{\displaystyle \sum }}_{i=1}^D{x}_i^2}\right]-\exp \left[\frac{1}{D}{{\displaystyle \sum }}_{i=1}^D\cos \left(2\pi x_i\right)\right]+20+e$; $F_5\left(x\right)=f_4\left(x-o_5\right)$
• $f_5\left(x\right)={{\displaystyle \sum }}_{i=1}^D\frac{x_i^2}{4000}-{{\displaystyle \prod }}_{i=1}^D\cos \left(\frac{x_i}{\sqrt{i}}\right)+1$; $F_6\left(x\right)=f_5\left(x-o_6\right)$
• $f_6\left(x\right)={{\displaystyle \sum }}_{i=1}^{D-1}\left[100{\left(x_i^2-x_{i+1}\right)}^2+{\left(x_i-1\right)}^2\right]; F_6\left(x\right)=f_6\left(M_7\left[\frac{2.048\left(x-o_7\right)}{20}\right]+1\right)$

Results of the KGA techniques (KGA_o-S, KGA_o-DB and KGA_f) were taken for D = 10 and 20 and are compared to those of the standard genetic algorithm GA and the Group Counseling Optimizer (GCO) [61] presented at the IEEE Congress on Evolutionary computation (CEC 2014). In all experiments, common parameters such as population number, maximum generation number and stopping criterion were the same for all algorithms. Population sizes of 50 and 100 were selected for dimensions 10 and 20 respectively. Each experiment is repeated 50 times to obtain the statistical features of the algorithms. A system with an Intel core i7 2.9 GHz processor and 4.096 GB RAM is used for implementing the MATLAB code for the proposed KGA techniques. All algorithms run the same number of fitness evaluations equal to 15,000 for D = 10 and 20,000 for D = 20, to ensure a fair comparison.

The statistical results of the test problems are shown in

Table 2. Comparison of statistical results of four algorithms for test problems 1–7 of dimensions D = 10.

Problem		KGAo (S Index)	KGAo (DB Index)	Genetic Algorithm (GA)	KGAf (K = 10)	GCO
1	Best	8.81 × 10−5	1.95 × 10−4	2.76 × 10−4	5.06 × 10−4	3.23
	Mean	2.45 × 10−3	5.33 × 10−3	8.29 × 10−3	2.84 × 10−2	1.23 × 101
	Worst	1.13 × 10−2	5.43 × 10−2	1.08 × 10−1	3.04 × 10−1	2.96 × 101
	SD	2.63 × 10−3	8.39 × 10−3	1.58 × 10−2	5.02 × 10−2	6.37
2	Best	2.91 × 10−4	3.34 × 10−4	4.60 × 10−4	2.08 × 10−3	8.46
	Mean	7.12 × 10−3	7.12 × 10−3	4.75 × 10−2	2.09 × 10−1	4.14 × 101
	Worst	6.27 × 10−2	4.83 × 10−2	7.21 × 10−1	3.62	2.22 × 102
	SD	1.07 × 10−2	1.06 × 10−2	1.17 × 10−1	6.13 × 10−1	4.61 × 101
3	Best	5.55 × 10−4	2.27 × 10−4	5.32 × 10−4	3.23 × 10−3	1.56 × 101
	Mean	1.01 × 10−2	8.24 × 10−3	5.01 × 10−2	3.12 × 10−1	8.85 × 101
	Worst	5.42 × 10−2	7.49 × 10−2	2.58 × 10−1	2.49	2.09 × 102
	SD	1.25 × 10−2	1.46 × 10−2	6.73 × 10−2	5.63 × 10−1	5.54 × 101
4	Best	4.00	2.00	1.50 × 10−1	3.00	3.00
	Mean	9.14 × 101	6.48 × 101	1.31 × 102	6.50 × 101	1.00 × 101
	Worst	3.86 × 102	4.19 × 102	3.83 × 102	2.05 × 102	2.70 × 101
	SD	9.84 × 101	8.38 × 101	8.88 × 101	5.42 × 101	6.94
5	Best	1.48 × 10−3	8.42 × 10−3	1.21 × 10−2	4.01 × 10−2	3.92
	Mean	1.50	6.55	5.15	5.62	6.36
	Worst	1.26 × 101	1.31 × 101	1.24 × 101	1.30 × 101	9.94
	SD	2.28	4.52	3.62	4.12	1.71
6	Best	4.94 × 10−2	4.97 × 10−2	4.95 × 10−2	5.04 × 10−2	1.24
	Mean	6.41 × 10−2	6.32 × 10−2	6.38 × 10−2	6.96 × 10−2	2.11
	Worst	8.66 × 10−2	8.56 × 10−2	8.14 × 10−2	1.00 × 10−1	4.51
	SD	7.33 × 10−3	6.73 × 10−3	7.56 × 10−3	1.07 × 10−2	6.77 × 10−1
7	Best	2.02 × 10−1	3.84 × 10−3	1.28	1.48 × 10−1	4.42 × 101
	Mean	3.77	3.65	3.22	4.60	9.28 × 101
	Worst	7.77	8.81	5.09	1.55 × 101	1.80 × 102
	SD	1.48	2.12	5.95 × 10−1	2.78	3.22 × 101

and

Table 3. Comparison of statistical results of four algorithms for test problems 1–7 of dimensions D = 20.

Problem		KGAo (S Index)	KGAo (DB Index)	Genetic Algorithm (GA)	KGAf (K = 10)	GCO
1	Best	1.67 × 10−3	2.36 × 10−3	1.05	4.51 × 10−3	3.60 × 101
	Mean	1.22 × 10−2	1.53 × 10−2	1.63	1.16 × 10−1	1.19 × 101
	Worst	6.32 × 10−2	9.89 × 10−2	2.43	6.42 × 10−1	2.17 × 101
	SD	1.45 × 10−2	1.87 × 10−2	3.13 × 10−1	1.40 × 10−1	5.88
2	Best	3.76 × 10−3	5.68 × 10−3	8.99	5.01 × 10−2	7.79 × 101
	Mean	1.17 × 10−1	1.19 × 10−1	1.02 × 101	4.10	9.34 × 101
	Worst	1.16	2.05	1.33 × 101	2.75 × 101	1.79 × 102
	SD	2.17 × 10−1	2.94 × 10−1	1.48	6.05	4.75 × 101
3	Best	1.96 × 10−1	5.50 × 10−3	1.49 × 101	2.27 × 10−2	3.33
	Mean	9.16 × 10−1	3.34 × 10−1	2.45 × 101	2.04	1.44 × 102
	Worst	3.19	4.59	3.53 × 101	1.27 × 101	2.62 × 102
	SD	4.29 × 10−1	6.85 × 10−1	4.19	2.55	4.76 × 101
4	Best	7.00	7.00	3.19 × 102	1.70 × 101	3.00
	Mean	7.91 × 101	7.52 × 101	4.89 × 102	8.83 × 101	8.48
	Worst	5.37 × 102	3.32 × 102	7.23 × 102	3.17 × 102	1.40 × 101
	SD	9.23 × 101	7.44 × 101	9.99 × 101	6.41 × 101	3.01
5	Best	1.46	1.32 × 10−1	9.83	1.55	1.28
	Mean	6.75	5.59	1.16	5.18	4.58
	Worst	1.26 × 101	1.28 × 101	1.25 × 101	1.20 × 101	8.94
	SD	3.69	3.58	7.38 × 10−1	2.47	1.16
6	Best	1.46	1.32 × 10−1	9.83	1.55	1.28
	Mean	6.75	5.59	1.16	5.18	4.58
	Worst	1.26 × 101	1.28 × 101	1.25 × 101	1.20 × 101	8.94
	SD	3.69	3.58	7.38 × 10−1	2.47	1.16
7	Best	1.65 × 10−2	1.02 × 10−2	7.99	5.04 × 10−2	3.10 × 101
	Mean	1.87 × 101	1.59 × 101	2.63 × 101	1.96 × 101	1.13 × 102
	Worst	7.54 × 101	7.21 × 101	6.15 × 101	7.85 × 101	1.72 × 102
	SD	2.82 × 101	2.75 × 101	1.34 × 101	2.97 × 101	2.67 × 101

. The GCO outperformed the proposed techniques in only one data set (test = 4, D = 20). In all other cases, the best solution was obtained with either the KGA_o-S or the KGA_o-DB. This demonstrates the significant feasibility and efficiency of the proposed techniques over the standard GA. Although the average runtime of each experiment was increased by 10–15%, the KGA techniques ensured a broader and more exhaustive search and prevent premature death of potential solutions.

The KGA methods implement an efficient partitioning of the population. They extend the diversity by intensifying the scope of the search process and reducing less favourable solutions. The recombination of two similar solutions will more likely generate a descendant with homogenous chromosomes. The evaluation of the membership probability vector inside the proposed selection process guarantees a more fitting parent selection.

In addition, the elitism strategy that results from partitioning the population into a number of clusters ensures that best solutions are always carried forward to the next generation. In fact, rather than obtaining one elite solution, K-strong optimal solutions are generated in each generation. In the long run, this enhances the exploration of future generations and reduces the possibility of premature convergence at local minima. The latter was recorded with the standard GA in problems 4–7 especially. Unlike the KGA_f, the KGA_o-S and KGA_o-DB are designed in such a way that there are no additional parameters to be fine-tuned.

The most frequently used statistical tests to determine significant differences between two computational intelligence algorithms are the t-test and Wilcoxon signed-ranks test [62]. The later is a non-parametric counterpart of the paired t-test, which ranks the differences in performances of two algorithms over each data set. In brief, the test omits the signs, and compares the ranks for the positive and the negative differences. The differences are ranked based on their absolute values and in case of ties average ranks are calculated.

A Wilcoxon test is used for pairwise comparisons between the following pairs of algorithms: KGA_o (S index)-KGA_o (DB index), KGA_o (S index)-GA, KGA_o (S index)-KGA_f, KGA_o (DB index)-GA, KGA_o (DB index)-KGA_f, GA-KGA_f for test function seven (refer to

Table 4. p-Values for the Wilcoxon test in paired comparisons for test number 7.

Comparison	R+	R−	Alpha	z-Score	p-Value
KGAo (S index)-KGAo (DB index)	307	968	0.05	3.190	1.421 × 10−3
KGAo (S index)-GA	155	1120	0.05	4.658	3.198 × 10−6
KGAo (S index)-KGAf	74	1201	0.05	5.440	5.339 × 10−8
KGAo (DB index)-GA	451	824	0.05	3.800	4.181 × 10−2
KGAo (DB index)-KGAf	134	1141	0.05	4.860	1.170 × 10−6
GA-KGAf	1275	0	0.05	6.154	7.557 × 10−10

).

As we can see, the p-values obtained by the paired Wilcoxon test indicate that the algorithms behave differently, since all p-values are less than the level of significance α = 0.05.

6. Conclusions and Future Work

A k-means Genetic Selection (KGA) is proposed to solve multimodal function optimization problems. Two different versions of the KGA technique were presented: using a fixed number of clusters K (KGA_f) and via an optimal number K_opt (KGA_o). In the latter, the optimal number of clusters is determined using two validity indexes: Silhouette and Davies–Bouldin. The KGA techniques are composed of four stages: clustering, membership phase, fitness scaling and selection. Clustering the population aids the search algorithm to preserve a selection pressure throughout the evolution. A membership probability number is assigned to each individual following the k-mean clustering phase. Fitness scaling converts the membership scores in a range suitable for the selection function which selects the parents of the next generation. The performance of each KGA technique (KGA_o-S, KGA_o-DB and KGA_f) is tested on seven benchmark problems for two separate dimensions of the search spaces D = 10 and 20. Traditional GAs perform well on single optimum problems but fail to provide multiple solutions. By combining the strength of clustering analysis and genetic search, the proposed KGA techniques permit the evaluation of multimodal functions. The computational results reveal that the proposed selection process is superior to or competitive with the standard genetic algorithm for the problems considered.

In the current study, combining the strengths of evolutionary computation and data mining were limited to single-objective optimization problems. Future research could test the performance of KGA techniques in solving constrained optimization problems and/or multiobjective formulations [63]. The stability of the novel selection processes should also be considered in future work. It would be compelling to integrate the KGA processes in further population-based optimization algorithms such as particle swarm optimization (PSO) [64], ant colony optimization (ACO) [65] and firefly algorithm (FA) [66]. Lastly, larger-scale examples should be tested and further research on the impact of GA parameters (such as population size, probabilities of crossover and mutation) on the KGA process will be examined.