B-PSA: A Binary Pendulum Search Algorithm for the Feature Selection Problem

Abstract

The digitization of information and technological advancements have enabled us to gather vast amounts of data from various domains, including but not limited to medicine, commerce, and mining. Machine learning techniques use this information to improve decision-making, but they have a big problem: they are very sensitive to data variation, so it is necessary to clean them to remove irrelevant and redundant information. This removal of information is known as the Feature Selection Problem. This work presents the Pendulum Search Algorithm applied to solve the Feature Selection Problem. As the Pendulum Search Algorithm is a metaheuristic designed for continuous optimization problems, a binarization process is performed using the Two-Step Technique. Preliminary results indicate that our proposal obtains competitive results when compared to other metaheuristics extracted from the literature, solving well-known benchmarks.

1. Introduction

In recent years, the use of different machine learning algorithms has become of great interest in the scientific community. This is due to two main reasons: (1) access to large computational capacities and (2) large volumes of existing data. These techniques have been successfully applied in image analysis, natural language processing, and sentiment analysis, among others.

However, the large volumes of data available for processing may contain redundant or unnecessary variables for classification that decrease the accuracy of the results and the performance of the predictive algorithms. Due to this sensitivity to the input data, it is very difficult to arrive at reliable results without proper data preprocessing.

Feature selection is one of the most widely used preprocessing techniques and aims to remove irrelevant and/or redundant information in order to keep only the most representative information of the dataset [1].

In [2], the authors tell us that there are three methods to solve the Feature Selection Problem:

• Filter Methods: They are independent of learning or classification algorithms since the filtering is based on expert judgment or statistical techniques that allow us to determine which are the most important features.
• Wrapper Methods: The cleaning process is performed online, i.e., it always interacts directly with the classification or learning algorithm. This is because, iteratively, the algorithms need constant performance feedback.
• Embedded Methods: This method is a hybridization between a filter method and a wrapper method. Feature selection is part of the training process of the learning or classification algorithm.

Although wrapper methods are computationally very expensive, their performance is better than that of filter methods. Metaheuristics fall into the category of wrapper methods.

Metaheuristics are general-purpose algorithms that, with a few modifications, can solve different optimization problems. Although metaheuristics do not guarantee global optimization, these provide us with high-quality solutions in a reasonable time. This characteristic makes different authors use metaheuristics to solve the Feature Selection Problem [2,3]. Figure 1 shows how to solve the Feature Selection Problem using metaheuristics.

As input, we have the original dataset where all features are being used. This set enters the metaheuristic module, which will be responsible for eliminating irrelevant and redundant features, resulting in a subset of data with some activated features. This subset of data enters the machine learning module, where the performance of a classifier or predictor with this subset of data will be tested. As the output of this module, we will have different performance metrics. Then, we evaluate whether we have reached our termination criterion. If we have not reached it, we return to the metaheuristic module, but with the feedback of performance obtained previously; if we have reached the termination criterion, we obtain, as a result, the best subset of data with some activated features.

In the literature, we can find different metaheuristics developed by researchers [4]. These researchers rely on the No Free Lunch (NFL) theorem to further deepen their understanding of metaheuristics, as this theorem indicates that there is no optimization algorithm capable of solving all existing optimization problems [5]. This theorem has not only led to the creation of new metaheuristics but also to the modification of existing ones, where we can highlight binarization. Binarization involves adapting metaheuristics designed to solve continuous optimization problems so that they can solve binary combinatorial optimization problems.

According to the work conducted in [6], we can observe that the Feature Selection Problem is the most addressed issue by researchers who binarize metaheuristics. Given this significant interest, there is a motivation to binarize recent metaheuristics that have not yet solved this problem.

In this work, we binarize the Pendulum Search Algorithm (PSA), a recently developed metaheuristic designed to solve continuous optimization problems. Such binarization is performed so that PSA can solve binary combinatorial optimization problems such as the Feature Selection Problem. The motivation for using the Pendulum Search Algorithm is that it has no configurable parameters, and this saves us the process of parameter tuning.

To validate the performance of the proposal, we compared our proposal with the Sine Cosine Algorithm and Moth-Flame Optimization, two widely used metaheuristics in the literature to solve the Feature Selection Problem with good results. Preliminary results indicate that PSA has good competitive results when compared to the other two metaheuristics.

The paper is organized as follows: Section 2 defines the Feature Selection Problem, Section 3 reviews how metaheuristics solve the Feature Selection Problem, Section 4 defines the Pendulum Search Algorithm, Section 5 defines our proposal Binary Pendulum Search Algorithm, the results obtained are shown in Section 6 to end with the conclusion in Section 7.

2. Feature Selection Problem

The Feature Selection Problem (FS) is a multi-objective combinatorial optimization problem that arises from the need to eliminate irrelevant and redundant information since this information is detrimental to the learning tasks of prediction or classification algorithms. On the one hand, we need to improve the prediction of classification algorithms, and on the other hand, we need to decrease the number of features that make up the original dataset.

In its original definition, FS works by assuming a dataset S, which contains a d number of f features, such that $S=\{f_1,f_2,f_3,\dots ,f_d\}$. The objective of the problem is to select the best subset of features $D=\{f_1,f_2,f_3,\dots ,f_n\}$ with $n<d$ so that the features belonging to the selected subset are the most representative of the set of original data.

Internally, Feature Selection has a binary domain, so the representation of vectors S and D has the same character, as exemplified in Figure 2, where a 1 would mean the activation or use of a feature, and a 0 the deactivation or the discarding of it. The vector S represents the utilization of the complete original dataset; that is, all the cells of the vector have a value of 1, whereas the vector D represents the resulting subset of features; that is, the vector is composed of 1 s and 0 s.

In this way, the objective function Z is calculated as the following equation:

$$minZ=\alpha ·Y_R\left(D\right)+\beta ·\frac{\left|R\right|}{\left|N\right|}$$

(1)

where $Y_R\left(D\right)$ represents the classification or prediction error of the ML algorithm, $\left|R\right|$ is the number of selected features, and $\left|N\right|$ the total number of features, with $\alpha \in [0,1]$ and $\beta =(1-\alpha )$ parameters that regulate the importance of the quality of the results and the size of the subset, respectively.

3. Continuous Metaheuristics Solving Feature Selection Problem

Using metaheuristics to solve the Feature Selection Problem is of high interest to the scientific community. In the literature [2,3,7,8,9], different literature reviews can be observed, focusing on documenting which metaheuristics have been used to solve the FSP. For instance, we can highlight Ant Lion Optimization (ALO) [10,11], Artificial Bee Colony Algorithm (ABC) [12,13], Bat Algorithm (BA) [14,15], Cuckoo Search (CS) [16,17,18], Grey Wolf Optimizer (GWO) [19,20,21], Dragonfly Algorithm [22], Harris’ Hawk Optimization (HHO) [23,24], Moth Flame Optimization (MFO) [25,26,27], Sine Cosine Algorithm (SCA) [28,29,30] and, Whale Optimization Algorithm (WOA) [31,32].

All these metaheuristics have one characteristic in common: they are metaheuristics designed to solve continuous optimization problems that were modified to solve binary combinatorial optimization problems, such as the Feature Selection Problem.

3.1. Two Step Techniques

In the literature, there are different ways of binarizing continuous metaheuristics [6], but the most widely used is the Two-Step Technique [33]. As the name suggests, the binarization process is performed in two steps:

• Application of a transfer function, which transfers the continuous value to a value within the range [0, 1].
• Application of a binarization rule which determines the assignment of a 1 or a 0.

In the literature [6], we can find different transfer functions, among which the S-Shaped and V-Shaped transfer functions stand out.

Table 1. S-Shaped and V-Shaped transfer functions.

S-Shaped		V-Shaped
Name	Equation	Name	Equation
S1	$T\left(d_i^j\right)=\frac{1}{1+e^{-2d_i^j}}$	V1	$T\left(d_i^j\right)=\left|erf\left(\frac{\sqrt{\pi }}{2}{d}_i^j\right)\right|$
S2	$T\left(d_i^j\right)=\frac{1}{1+e^{-d_i^j}}$	V2	$T\left(d_i^j\right)=\left|tanh\left(d_i^j\right)\right|$
S3	$T\left(d_i^j\right)=\frac{1}{1+e^{\frac{-d_i^j}{2}}}$	V3	$T\left(d_i^j\right)=\left|\frac{d_i^j}{\sqrt{1+{\left(d_i^j\right)}^2}}\right|$
S4	$T\left(d_i^j\right)=\frac{1}{1+e^{\frac{-d_i^j}{3}}}$	V4	$T\left(d_i^j\right)=\left|\frac{2}{\pi }arctan\left(\frac{\pi }{2}{d}_i^j\right)\right|$

and Figure 3 present the families of transfer functions utilized in this study. The notation $d_i^j$ observed in Table 1 corresponds to the continuous value of the j-th dimension of the i-th individual resulting after the perturbation performed by the continuous metaheuristic.

Additionally, in the literature [33], we can find five different binarization rules, of which we can highlight the following:

• Standard (STD): If the condition is satisfied, the standard binarization rule returns 1; otherwise it returns 0. Mathematically, it is defined as follows:

  $$X_{new}^j=\left\{\begin{array}{cc}1\hfill & ifrand\le T\left(d_i^j\right)\hfill \\ 0\hfill & else.\hfill \end{array}\right.$$

  (2)
• Elitist (ELIT): The best value is assigned if a random value is within the probability; otherwise, a zero value is assigned. Mathematically, it is defined as follows:

  $$X_{new}^j=\left\{\begin{array}{cc}{X}_{Best}^j\hfill & ifrand<T\left(d_i^j\right)\hfill \\ 0\hfill & else.\hfill \end{array}\right.$$

  (3)
• Elitist Roulette (ELIT_ROU): This binarization rule selects randomly among the best individuals of the population, with a probability that is directly proportional to their fitness. Mathematically, it is defined as follows:

  $$X_{new}^j=\left\{\begin{array}{cc}P[X_{new}^j={\zeta }_j]=\frac{f\left(\zeta \right)}{{\sum }_{\delta \in Q_g}f\left(\delta \right)}\hfill & ifrand\le T\left(d_i^j\right)\hfill \\ P[X_{new}^j=0]=1\hfill & else.\hfill \end{array}\right.$$

  (4)

3.2. Evaluation Metrics

As seen in Equation (1), we need the error rate, and this is a classifier performance metric. However, the classifier can not only measure their performance with the error rate but also with the f-score [2]. Both metrics are constructed from the confusion matrix obtained after classifier validation. Figure 4 shows how the confusion matrix is constructed.

With this, the error rate and f-score are constructed as follows:

$$Errorrate=\frac{\mathsf{fP}+\mathsf{fN}}{\mathsf{tP}+\mathsf{fN}+\mathsf{fP}+\mathsf{tN}}$$

(5)

$$f.score=2·\frac{Precision·Recall}{Precision+Recall},where$$

(6)

$$Recall=\frac{\mathsf{tP}}{\mathsf{tP}+\mathsf{fN}},Precision=\frac{\mathsf{tP}}{\mathsf{tP}+\mathsf{fP}}$$

(7)

These metrics are used by the authors to demonstrate that the metaheuristics applied to solve the Feature Selection Problem did so correctly and reliably.

4. Pendulum Search Algorithm

The Pendulum Search Algorithm is a new population-based metaheuristic created in 2022 by Nor Azlina Ab. Aziz and Kamarulzaman Ab. Aziz [34]. This metaheuristic was designed to solve continuous optimization problems, and it is inspired by the harmonic motion of the simple pendulum. The equation of movement is very similar to the one proposed in Sine Cosine Algorithm. The authors incorporate an exponential function, which would improve the balance between exploration and exploitation [35].

The search agents are initialized randomly, and their position is updated using Equation (8).

$$X_{i,j}^t=X_{i,j}^t+pen{d}_{i,j}^t·(Bes{t}_j-X_{i,j}^t)$$

(8)

where $X_{i,j}^t$ is the position of the i-th solution in the j-th dimension in t-th iteration, $Bes{t}_j$ it is the position of the best solution in the j-th dimension in the t-th iteration and $pen{d}_{i,j}^t$ is a parameter which is calculated as follows:

$$pen{d}_{i,j}^t=2·e^{(-t/tmax)}·cos(2·\pi ·rand)$$

(9)

where t is the current iteration, $tmax$ is the maximum number of iterations and $rand$ is a uniform random number between [0, 1]. The pseudo-code of PSA is shown in Algorithm 1.

Algorithm 1 Pendulum Search Algorithm

Input: The population $X=$$\{X_1,X_2,\dots ,X_i\}$   Output: The updated population $X^{\prime }=$$\{X_1^{\prime },X_2^{\prime },\dots ,X_i^{\prime }\}$ and $Best$  1: Initialize random population X  2: Evaluate the objective function of each individual in the population X  3: Identify the best individual in the population ($Best$)  4: for  $iteration\left(t\right)$ do  5:     for $solution\left(i\right)$ do  6:         for $dimension\left(j\right)$ do  7:            Update $pen{d}_{i.j}^t$ by Equation (9)  8:            Update the position of $X_{i,j}^t$ using Equation (8)  9:         end for 10:     end for 11:     Evaluate the objective function of each individual in the population X 12:     Update $Best$ 13: end for 14: Return the updated population $X^{\prime }$ where $Best$ is the best result

5. Binary Pendulum Search Algorithm

As mentioned in Section 4, PSA is a metaheuristic designed to solve continuous optimization problems, so it is necessary to convert the solutions to the binary domain to solve the Feature Selection. Additionally, in Section 3, it was mentioned that the Two-Step Technique is one of the most widely used methods for binarizing continuous metaheuristics.

In [6,33], the authors present eight different transfer functions and five binarization rules. In this work, after preliminary experimenting with different combinations, we decided to use the following transfer function and binarization rule. Equation (10) represents the transfer function, and Equation (11) represents the binarization rule used in this work.

$$T\left(d_w^j\right)=\left|\begin{array}{c}\frac{2}{\pi }arctan\left(\begin{array}{c}\frac{\pi }{2}{d}_w^j\end{array}\right)\end{array}\right|$$

(10)

$$X_{new}^j=\left\{\begin{array}{cc}1& ifrand\le T\left(\begin{array}{c}{d}_w^j\end{array}\right)\\ 0& otherwise\end{array}\right.$$

(11)

with this, we can construct the Binary Pendulum Search Algorithm (B-PSA). First, we initialize the solutions in the binary domain, and in each iteration, the following will be performed: (1) perturbing the binary solutions with Equations (8) and (9), the classical equation of motion of the PSA. (2) after perturbing all the solutions, they will leave the binary domain, and binarization will be applied using Equations (10) and (11). This process is repeated until the end of the iterations. After the solution generation and binarization step, a feasibility test and solution repair are performed (Algorithm  2). In the feasibility test, we verify that each solution has at least one activated feature; that is, there is at least one “1” in the solution vector. If this condition is not met, a new random binary solution is generated, and the feasibility test is applied again. This process is repeated until all feasible solutions are obtained. The pseudo-code of B-PSA is shown in Algorithm 3.

Algorithm 2 Feasibility test and repair solution

Input: The population $X=$$\{X_1,X_2,\dots ,X_i\}$   Output: The feasible population $X=$$\{X_1,X_2,\dots ,X_i\}$  1: repeat  2:     for $solution\left(i\right)$ do  3:         if $solutio{n}_i$ is composed of only 0 then  4:            Generate a new binary random solution  5:         else  6:            feasible solution  7:         end if  8:     end for  9: until all solutions are feasible 10: return the feasible population X

Algorithm 3 Binary-Pendulum Search Algorithm

Input: The population $X=$$\{X_1,X_2,\dots ,X_i\}$   Output: The updated population $X^{\prime }=$$\{X_1^{\prime },X_2^{\prime },\dots ,X_i^{\prime }\}$ and $Best$  1: Initialize binary random population X  2: Apply feasibility test according to Algorithm 2  3: Evaluate the objective function of each individual in the population X  4: Identify the best individual in the population ($Best$)  5: for  $iteration\left(t\right)$ do  6:     for $solution\left(i\right)$ do  7:         for $dimension\left(j\right)$ do  8:            Update $pen{d}_{i.j}^t$ by Equation (9)  9:            Update the position of $X_{i,j}^t$ using Equation (8) 10:         end for 11:     end for 12:     Binarization of population X 13:     Apply feasibility test according to Algorithm 2 14:     Evaluate the objective function of each individual in the population X 15:     Update $Best$ 16: end for 17: return the updated population $X^{\prime }$ where $Best$ is the best result

6. Experimental Results

To validate the performance of our proposal, test datasets provided by https://archive.ics.uci.edu/ (accessed on 29 October 2023) were used and compared to the Binary Sine Cosine Algorithm and Binary Moth-Flame Optimization. Experiments using the Sine Cosine Algorithm and Moth-Flame Optimization were run from zero without using results from previous work. In this repository, there are several datasets where different authors validate their algorithms that solve the Feature Selection Problem.

Table 2. Instances used.

Instance	Data Type	Area	N° of Features	Sample Size	N° of Class	Missed Value?
Breast cancer	Integer	Life	10	699	2	Yes
wisconsin (Original)
Divorce	Integer	Life	54	170	2	No
Immunotherapy	Integer	Life	8	90	2	No
	Real
Ionosphere	Integer	Physical	34	351	2	No
	Real
Connectionist Bench	Real	Physical	60	208	2	No
(Sonar, Mines vs Rocks)
Breast Cancer	Real	Life	32	569	2	No
Wisconsin—Diagnostic (wdbc)

shows the characteristics of the resolved instances. The first column represents the solved instance, the second column represents the data type of the datasets, the third column refers to the field belonging to the instance, the fourth column refers to the number of features that compose the datasets, the fifth column refers to the number of samples in the datasets, the sixth column represents the number of classes to be classified and the seventh column refers to whether the dataset has missing data.

The K-Nearest Neighbors (KNN), one of the most widely used classifiers in the literature [2], was used as a classifier in the Machine Learning Module mentioned in Figure 1. The datasets were separated into two, with 75% of the data being used as training data and 25% of the data being used as validation data. The population size used for the three metaheuristics is 10 individuals, and 100 iterations were performed. The rest of the parameters are shown in

Table 3. Configuration of parameters.

Parameter	Value
Population size	10
Iterations	100
Independent runs	31
Parameter k of KNN	5
Parameter b of MFO	1
Parameter a of SCA	2
PSA has no parameteris	-

.

Fitness and f-score are the metrics used to show the good performance of our proposal.

Table 4. Fitness obtained.

Instance	B-PSA			B-SCA			B-MFO
	best	avg	std-dev	best	avg	std-dev	best	avg	std-dev
ionosphere	0.034	0.045	0.011	0.034	0.045	0.012	0.024	0.043	0.01
sonar	0.02	0.089	0.021	0.039	0.095	0.023	0.059	0.098	0.016
Immunotherapy	0.09	0.09	0.0	0.09	0.091	0.007	0.09	0.09	0.001
Divorce	0.0	0.025	0.01	0.0	0.021	0.009	0.0	0.014	0.011
wdbc	0.015	0.027	0.008	0.009	0.03	0.007	0.016	0.031	0.007
breast-cancer-wisconsin	0.032	0.033	0.003	0.032	0.033	0.001	0.032	0.032	0.0

shows the fitness obtained for each instance using the three metaheuristics under study. The first column of this table indicates the instance under study, and the second, third, and fourth columns are repeated for each metaheuristic. The first column indicates the best fitness obtained; the second indicates the average obtained with the 31 independent runs performed; the third indicates the standard deviation of the 31 runs performed. The best result per instance is highlighted in bold type.

In this table, we can see that B-PSA obtains good results compared to the other two metaheuristics and is even better in the sonar instance. This indicates that our proposal is competitive.

Additionally,

Table 5. F-score obtained.

Instance	B-PSA			B-SCA			B-MFO
	best	avg	std-dev	best	avg	std-dev	best	avg	std-dev
ionosphere	0.971	0.962	0.01	0.971	0.962	0.01	0.98	0.962	0.009
sonar	0.981	0.918	0.019	0.962	0.911	0.022	0.943	0.907	0.015
Immunotherapy	0.95	0.95	0.0	0.95	0.949	0.004	0.95	0.95	0.0
Divorce	1.0	0.974	0.011	1.0	0.978	0.01	1.0	0.985	0.012
wdbc	0.989	0.979	0.006	0.994	0.978	0.006	0.989	0.976	0.006
breast-cancer-wisconsin	0.961	0.96	0.003	0.961	0.961	0.0	0.961	0.961	0.0

shows the f-score obtained for each instance using the three metaheuristics under study. The first column of this table indicates the instance under study and the second. The third and fourth columns are repeated for each metaheuristic. The first indicates the best f-score obtained; the second indicates the average obtained with the 31 independent runs performed; the third indicates the standard deviation of the 31 runs performed.

In this table, we can see that B-PSA obtains f-score above 95% in all instances. This indicates that the B-PSA manages to eliminate unnecessary information correctly so that the KNN can perform a very good classification.

Figure 5 shows the convergence graphs of the best executions performed with the three metaheuristics used for each of the six solved instances.

From these Figures, we can observe that B-PSA has a fast convergence in four of the six solved instances, falling in the best value before 50% of the total iterations to be executed are reached. In particular, it is interesting the behavior obtained in the sonar instance (see Figure 5b) where B-PSA has a fast convergence to very good results while B-MFO has a very slow convergence and even very far from the value obtained by B-PSA.

In addition, Figure 6 shows the evolution of the f-score obtained from the best runs performed with the three metaheuristics used for each of the six solved instances. In the figures, we can observe a behavior similar to those obtained in the convergence graphs, except that for the f-score, we look for a value close to 1 as a good result.

Finally, Figure 7 shows the evolution of the number of features selected by the best execution of each of the three metaheuristics used for each of the six solved instances. Here, we can see that in four of the six instances, the behavior is similar except for two, which are wdbc and sonar. Regarding the wdbc instance (see Figure 7e), we can observe that B-PSA selects fewer features than the other two metaheuristics, but upon reviewing Table 4, we can see that B-PSA’s results are very close to those obtained by B-SCA, which is the metaheuristic that achieves the best results in that instance.

On the other hand, in the sonar instance (see Figure 7b), we can observe that B-PSA is the metaheuristic that selects more features compared to the others. This is interesting as, despite B-PSA selecting more features, it achieves better results, indicating good performance.

7. Conclusions

The Feature Selection Problem has practical applications in the real world in various fields such as health, engineering, telecommunications, or marketing. Given this, it is critical to have highly reliable solutions that deliver high-quality results in reduced time.

In the present work, we demonstrate that the Binary Pendulum Search Algorithm is competitive in solving the Feature Selection Problem, considering both the fitness obtained and the f-score metric. With this, we were able to demonstrate how simple it can be to adapt a continuous metaheuristic to solve binary combinatorial problems. A major advantage of the Binary Pendulum Search Algorithm over the Binary Sine Cosine Algorithm and Binary Moth-Flame Optimization is that it has no configurable parameters. This allows a quick adaptation of the metaheuristic so that it can solve different optimization problems.

However, as indicated by authors in [6], there are different ways to binarize continuous metaheuristics. Such as different transfer functions (U-shaped [36], O-Shaped [37], X-Shaped [38], Z-Shaped [39], time-varying S-Shaped [40] and time-vatying V-Shaped [41]), other binarization rules [33], clustering-based binarization techniques [42] or even binarization supported by machine learning techniques such as Q-Learning [43] or SARSA [44]. The No Free Lunch Theorem [45] tells us that there is no algorithm capable of reaching the global optimum of all existing optimization problems. Therefore, in future work, another binarization technique could be implemented in order to validate whether the good performance obtained in the present work is maintained. In addition, our proposal could be validated with other datasets and even applied to another combinatorial problem, such as the Set Covering Problem or the Knapsack Problem.