A Multi-Task Decomposition-Based Evolutionary Algorithm for Tackling High-Dimensional Bi-Objective Feature Selection

Abstract

Evolutionary algorithms have been widely applied for solving multi-objective optimization problems, while the feature selection in classification can also be treated as a discrete bi-objective optimization problem if attempting to minimize both the classification error and the ratio of selected features. However, traditional multi-objective evolutionary algorithms (MOEAs) may have drawbacks for tackling large-scale feature selection, due to the curse of dimensionality in the decision space. Therefore, in this paper, we concentrated on designing an multi-task decomposition-based evolutionary algorithm (abbreviated as MTDEA), especially for handling high-dimensional bi-objective feature selection in classification. To be more specific, multiple subpopulations related to different evolutionary tasks are separately initialized and then adaptively merged into a single integrated population during the evolution. Moreover, the ideal points for these multi-task subpopulations are dynamically adjusted every generation, in order to achieve different search preferences and evolutionary directions. In the experiments, the proposed MTDEA was compared with seven state-of-the-art MOEAs on 20 high-dimensional classification datasets in terms of three performance indicators, along with using comprehensive Wilcoxon and Friedman tests. It was found that the MTDEA performed the best on most datasets, with a significantly better search ability and promising efficiency.

1. Introduction

Evolutionary algorithms [1] have been widely-used as common tools to solve multi-objective optimization problems (MOPs) [2] during the past decades. In fact, when the number of objectives to be optimized is more than one, they are normally contradictory to each other, and therefore multi-objective evolutionary algorithms (MOEAs) [3] are used for finding a set of nondominated solutions. Compared with other meta-heuristics [4], MOEAs have the advantages of a population-based search mode and no need of domain knowledge. Thus, a huge variety of MOEAs have been proposed all over the world and can be roughly divided into the following categories: dominance-based MOEAs [5,6,7,8], decomposition-based MOEAs [9,10,11,12,13], indicator-based MOEAs [14,15,16,17], surrogate-based MOEAs [18,19,20], cooperative coevolutionary MOEAs [21,22,23], multi-task MOEAs [24,25,26], and so on. There are also many other kinds of excellent MOEAs [27,28,29], including the novel multi-objective particle swarm optimization algorithm proposed by Leung et al. [30], which adopted a hybrid global leader selection strategy with two leaders: one for exploration and the other for exploitation. Moreover, MOEAs have also been used to solve many real-world optimization problems [31,32,33], such as system control [34,35], community detection [36,37], network construction [38,39,40], task allocation [41,42], and feature selection [43,44]. Generally speaking, feature selection is normally used to select useful feature subsets for classification [45], while the bi-objective feature selection problem usually seeks to minimize both the classification error and the number of selected features [46].

However, due to the curse of dimensionality in the decision space as the number of features expands to large scale, traditional MOEAs are likely to encounter setbacks when tackling bi-objective feature selection. In fact, the large-scale multi-objective optimization problem (LSMOP) [47] remains challenging for most MOEAs, although many MOEAs have attempted to solve it. For example, Ma et al. [48] and Zhang et al. [49] focused on analyzing different kinds of decision variables before evolution, while Bai et al. [50] and Yang et al. [51] proposed decision-variable-based MOEAs for solving continuous LSMOPs. However, not all large-scale MOEAs can be used for discrete optimization, especially for feature selection that has to face the complex interrelationships among features and a large number of feasible feature combinations. One possible approach is to bring in the multi-task framework [52,53] for cooperative evolution, as this can achieve an overall effect of learning by transferring genetic knowledge between different evolutionary tasks. Many MOEAs have already adopted multi-tasking [54,55,56] and it seems quite promising for decomposition-based MOEAs to be integrated with the multi-task framework, due to the intrinsic parallelism characteristics inside the weight-vector-related aggregation functions [43].

In fact, the integration of multi-task mechanisms with the decomposition-based evolutionary approaches is theoretically appropriate, mainly because of the following three reasons. First, both multi-task mechanisms and decomposition-based approaches share the same philosophy: to decompose a complex problem (task) into multiple interrelated or independent subproblems (tasks), in order to solve the problem (task) more efficiently and comprehensively. Second, each subproblem in a decomposition-based MOEA can be treated as an independent search task, whose search behaviors can be adjusted by changing the direction of the related weight vector. Third, the global ideal point for a decomposition based MOEA controls the general search areas, while different global ideal points would make the algorithm display quite different search preferences, which can be combined with the multi-task mechanism, allowing different global ideal points to represent different evolutionary search tasks. The abovementioned third point is exactly what is made use of in this paper and also acted as our major motivation.

Therefore, in this paper, we aimed to design an adaptive decomposition-based MOEA framework combined with a dynamic multi-task mechanism, named a multi-task decomposition-based evolutionary algorithm (MTDEA), for tackling the large-scale bi-objective evolutionary feature selection problem in high-dimensional classification datasets. To be more specific, we attempted to organically combine a multi-task mechanism with the decomposition-based approach to self-adaptively adjust the search preferences and evolutionary directions for each task-related subpopulation, in order to improve both the optimization and classification performance, and to achieve cooperative evolution within the population for better diversity and convergence. Overall, our major contributions can be summarized as follows:

• First, a dynamic multi-task mechanism is designed and combined with the decomposition-based MOEA framework, which assigns multiple evolutionary search tasks for different subpopulations within the entire population and then conditionally merges them into a single task or an integrated population as the evolutionary process goes on, for tackling the large-scale bi-objective feature selection in a more effective way.
• Second, an adaptive decomposition-based MOEA framework is set up, which cooperates with the above multi-task mechanism via adaptively adjusting the ideal point for each subpopulation related to different tasks, so that each task has distinct search biases and focuses its computational resources on searching more productive areas in the objective space.
• Third, a series of comprehensive studies were conducted in experiments to analyze the optimization and classification performance of the proposed MTDEA algorithm against other state-of-the-art MOEAs, in terms of multiple indicators and using a variety of high-dimensional classification datasets.

The remainder of this paper is organized as follows: First of all, the related works are introduced in Section 2. Then, the proposed algorithm MTDEA is comprehensively illustrated in Section 3. The experiment setups are given in Section 4, while the empirical results are studied in Section 5. Last, the conclusions are given in Section 6.

2. Related Works

2.1. Bi-Objective Feature Selection Problem

Generally, a multi-objective feature selection problem [57] can be defined as a multi-objective optimization problem, shown as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{cc}\hfill & \hfill minimize\mathit{F}\left(\mathit{x}\right)={(f_1\left(\mathit{x}\right),f_2\left(\mathit{x}\right),\dots ,f_M\left(\mathit{x}\right))}^T\hfill \\ \hfill & \hfill subjectto\mathit{x}=(x_1,x_2,\dots ,x_D),x_i=\{0,1\}\hfill \end{array}\end{array}\end{array}$$

(1)

where M is the total number of objectives to be optimized, and D is the full number of features that can be selected, i.e., also the dimensionality of the decision space. In this paper, M is set to 2, and $\mathit{F}\left(\mathit{x}\right)$ is the objective vector of $\mathit{x}$, while $f_i\left(\mathit{x}\right)$ is the objective value in the $f_i$ direction. $\mathit{x}=(x_1,x_2,\dots ,x_D)$ is the decision vector of a certain solution, while the value of 1 means selecting that feature and 0 means not. In addition, the first objective function $f_1\left(\mathit{x}\right)$ can be further defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill f_1\left(\mathit{x}\right)=\sum _{i=1}^D{x}_i/D\end{array}\end{array}$$

(2)

where its function values are between 0 and 1, i.e., $\in \{0,1/D,2/D,\dots ,1\}$, denoted as the rate of currently selected features. Moreover, the second objective function $f_2\left(\mathit{x}\right)$ denotes the resultant classification error rate for the previously selected features in $\mathit{x}$, whose function values also discretely range from 0 to 1. Given the results of $TP$ (true positive), $TN$ (true negative), $FP$ (false positive), and $FN$ (false negative), $f_2\left(\mathit{x}\right)$ can be formalized as follows:

$$f_2\left(\mathit{x}\right)={\displaystyle \frac{FP+FN}{TP+TN+FP+FN}}$$

(3)

2.2. Evolutionary Feature Selection Methods

In the past decades, evolutionary feature selection [58] has been roughly categorized into wrapper-based or the filter-based approaches [59,60]. Generally, a wrapper-based approach [61,62] uses a classification model, like SVM (support vector machine) or KNN (K-nearest neighbor) [63], as a “black box” to evaluate the classification accuracy, while the filter-based approach [64,65] is independent of any classifier and ignores the classification results of the currently selected features. Thus, a wrapper-based approach is normally more accurate but may consume a higher computational cost [66,67,68]. In this paper, the wrapper-based approach is adopted for bi-objective feature selection, and in fact, many other such MOEAs have been proposed in the last few years [69]. For example, in 2020, Tian et al. [70] proposed a large-scale MOEA framework, named SparseEA, based on pre-analyzing each feature’s classification performance, which however, would consume a large number of objective function evaluations for high-dimensional datasets. Subsequently, Xu et al. [71] proposed a duplication-analysis-based MOEA, named DAEA, with an efficient reproduction method to generate more valid and diverse offspring, but its tested feature dimensionality has not yet reached 10000. Following the idea of DAEA, Jiao et al. [72] modified the solution duplication handling method and further designed a problem reformulation mechanism, named PRDH, whose applicability across other MOEA frameworks remains unconfirmed. In 2022, Cheng et al. [73] proposed a steering-matrix-based algorithm for high-dimensional classification, named SM-MOEA, which was quiet efficient in tackling large-scale optimization, but its generalization ability still needs to be verified on more datasets, as only 12 datasets were tested in their work.

2.3. Decomposition-Based MOEA Approaches

Since its first introduction in the famous algorithm MOEA/D [74], a great number of decomposition-based MOEAs have been proposed all around the world [75,76,77,78,79]. To be more specific, a decomposition-based approach uses a series of uniformly distributed weight vectors as the aggregation functions to decompose a complex MOP into a set of simpler single-objective optimization problems that are related to each weight vector one by one.Generally speaking, there are three widely-used aggregation functions for decomposition, i.e., the weighted sum (WS) approach [80], the Tchebycheff (TCH) approach [81], and the penalty-based boundary intersection (PBI) approach [82]. The TCH approach is further introduced here, as it was the original decomposition approach adopted in MOEA/D and is also one of the comparison algorithms used in our later experiment. In detail, the TCH approach can be formally defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{cc}\hfill & \hfill minimize{g}^{tch}\left(\mathit{x}\right|\mathit{w},\mathit{z})=\underset{1\le i\le M}{max}\left\{\right|f_i\left(\mathit{x}\right)-z_i|\ast w_i\}\hfill \\ \hfill & \hfill subjectto\mathit{x}=(x_1,x_2,\dots ,x_D),x_i=\{0,1\}\hfill \end{array}\end{array}\end{array}$$

(4)

where $\mathit{w}$ and $\mathit{z}$ respectively denote the current weight vector and the ideal point, while the selection principle is based on a $Max\text{-}Min$ mechanism that first calculates the maximum values and then selects the minimum one.

3. Proposed Algorithm

In this section, we first introduce the general framework of the proposed MTDEA, and then further illustrate its essential components, i.e., initialization, reproduction, environmental selection, and task merging processes.

3.1. General Framework

The general framework of the MTDEA is shown in Algorithm 1, where the population size N and the decision space dimension D (i.e., the total number of features) are input as the primary parameters. In brief, its general framework is similar to traditional MOEAs but brings in multi-task factors and adds a task merging process at the end of every evolutionary generation. In detail, each solution in the population is related with a task number, which are initialized together at the beginning and correlated closely throughout the subsequent evolution. The implicit relationship between solutions and tasks is dynamically maintained by an external array with two rows and N columns, where the first row stores the solution indexes and the second row stores the corresponding task numbers. More specific relationship and constructions of the initial tasks and population are shown in Figure 1, which will be further illustrated in Section 3.2. As a result, the reproduction process of the MTDEA also uses a multi-task mechanism for cooperatively exchanging genetic information between the different tasks, which will be further illustrated in Section 3.3. Moreover, the duplicated decision vectors should first be removed before separately truncating each task-related subpopulation from the current population and offspring, while the detailed selection process based on decomposition will be shown in Section 3.4. At last, the task merging process dynamically decides whether to merge each two tasks or not and will eventually integrate all the tasks into a single task, details of which will be given in Section 3.5.

3.2. Initialization Process

The initialization process of the MTDEA is shown in Algorithm 2, which also invokes Algorithm 3. During initialization, there are a total of three subpopulations to be generated related with different task numbers. The implicit inner relationship between the solutions and tasks is shown in Figure 1, assuming a population size of 100. It can be seen from Figure 1 that the first 25% of solutions are related to task-1, the median 50% to task-2, and the last 25% to task-3, with each solution corresponding one-to-one to a task number at the same index position. Thus, the whole population can be split into three subpopulations, and the generation of a new subpopulation is shown in Algorithm 3, where the so-called distribution axis parameter is input to roughly control the solution distributions in the objective space.

Algorithm 1 $General$  $Framework$  $(N,D)$

Input: population size N, decision space dimension D; Output: final population $Pop$;   1: $[Pop,T]=Initialize(N,D)$; $//$ Algorithm 2   2: while termination criterion is not reached do   3:       $[Po{p}^{*},T^{*}]=Reproduce(Pop,T)$; $//$ Algorithm 4   4:       for each unique task i from T do   5:              $j\leftarrow$ get the quantity of task-i solutions in $Pop$              according to the task numbers in T;   6:              $\psi \leftarrow$ get all the task-i solutions in $Pop$ and $Po{p}^{*}$              according to the task numbers in T and $T^{*}$;   7:              $\psi \leftarrow$ remove duplicated decision vectors in $\psi$;   8:              $\psi =Select(\psi ,i,j)$; $//$ Algorithm 5   9:              replace all the task-i solutions in $Pop$ with $\psi$; 10:       end for 11:       if more than one task exist then 12:              $T=Merge(Pop,T)$; $//$ Algorithm 6 13:       end if 14: end while

Algorithm 2 $Initialize$  $(N,D)$

Input: population size N, decision space dimension D; Output: initial population $Pop$, related tasks T;   1: $Tasks=[1,2,3]$; $//$ task numbers   2: $Axes=[0.25,0.5,0.75]$; $//$ distribution axes   3: $Sizes=[\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$4$}\right.,\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$2$}\right.,\raisebox{1ex}{$N$}\!\left/ \!\raisebox{-1ex}{$4$}\right.]$; $//$ subpopulation sizes   4: $Pop=\varnothing ,T=\varnothing$;   5: for $i=1,2,\dots ,Length\left(Tasks\right)$ do   6:       $Pop=Pop\cup NewPop\left(Sizes\right(i), D, Axes(i\left)\right)$;   7:       $\psi =Ones(1, Sizes(i\left)\right)$; $//$ create a vector of ones   8:       $T=T\cup (\psi .\ast Tasks(i\left)\right)$;   9: end for

Algorithm 3 $NewPop(K,D,A)$

Input: subpopulation size K, decision space dimension D, distribution axis A; Output: new subpopulation $SubPop$;   1: $SubPop=Zeros(K,D)$; $//$ create a matrix of zeros   2: for $i=1,2,\dots ,K$ do   3:       for $j=1,2,\dots ,D$ do   4:             if $\rho <A$ then $//$ $\rho$ is a random probability   5:                  $SubPop(i,j)=1$; $//$ select the jth feature   6:           end if   7:         end for     8: end for

As can be seen from Line 4 in Algorithm 3, the distribution axis A plays a role as the probability threshold, while a larger threshold indicates a higher probability of randomly selecting this feature, which also means that the resultant decision vector is likely to select more features. Reflected in the objective space, this implies that the overall distribution of the newly generated solutions should be more backward in the $f_1$ direction for a larger distribution axis value, and vice versa. Figure 2 gives an intuitive example of how the three subpopulations related to different tasks are likely to be distributed in the objective space. It is shown in Figure 2 that each newly generated subpopulation is assumed to be distributed around their preset axes in the $f_1$ direction. Moreover, the gray arrows in Figure 2 point out the expected evolutionary direction of each subpopulation, corresponding to the specific search bias of each related task, which will be further illustrated in Section 3.4.

3.3. Reproduction Process

The reproduction process of the MTDEA is shown in Algorithm 4, where the current population and the related task set are input. First, a total of N (i.e., $Length\left(Pop\right)$) pairs of parent solutions are randomly selected from the current population. Then, for each pair of parents, a corresponding new offspring will be generated after conducting the so-called valid crossover (from Lines 6 to 10 in Algorithm 4) and bitwise mutation (from Lines 11 to 13 in Algorithm 4) operations. It should be noted that the related task number for a new offspring is the same as its parent $p_1$. Moreover, the valid crossover will not occur if the two parents have different tasks and the random probability $\rho$ fails to reach 0.5, which acts as a key parameter for controlling the genetic information transfer among different tasks (Line 6 in Algorithm 4). Here, it is set to 0.5 for the sake of a relatively uniform random selection, without loss of generality. The so-called valid crossover operation can also be further referred to in our previous work [71], a simple example of which is shown in Figure 3, where two decision vectors are drawn as parents and only the gray parts owning different decision variable values are allowed to swap genes randomly.

Algorithm 4 $Reproduce(Pop,T)$

Input: current population $Pop$, related tasks T; Output: offspring solutions $Po{p}^{*}$, offspring tasks $T^{*}$;   1: $Pairs\leftarrow$ randomly select $Length\left(Pop\right)$ pairs of solutions from $Pop$ as parents; $//$ random mating   2: for $i=1,2,\dots ,Length\left(Pop\right)$ do   3:       $p_1,p_2\leftarrow$ get the ith pair of parents from $Pairs$;   4:       $t_1,t_2\leftarrow$ get the related task numbers of $p_1,p_2$       according to the corresponding indexes in T;   5:       $T^{*}\left(i\right)=t_1$; $//$ assign offspring task number   6:       if $(t_1=t_2)\Vert (t_1\ne t_2\cap \rho <0.5)$ then   7:             $j\leftarrow$ find the indexes of different decision             variable values between $p_1$ and $p_2$;   8:             $j\leftarrow$ uniformly randomly select indexes from j;   9:             $p_1\left(j\right)=p_2\left(j\right)$; $//$ valid crossover 10:       end if 11:       $\rho \leftarrow$ get a set of $Length\left(p_1\right)$ random probabilities; 12:       $j\leftarrow$ get the indexes satisfying $\rho <1/Length\left(p_1\right)$; 13:       $p_1\left(j\right)=\neg p_1\left(j\right)$; $//$ bitwise mutation 14:       $Po{p}^{*}\left(i\right)=p_1$; $//$ get the new offspring 15: end for

3.4. Environmental Selection Process

The environmental selection process of the MTDEA is shown in Algorithm 5, where a certain task-related union population (subpopulation + offspring) is input for further truncation into the required size of subpopulation. The whole selection process is based on decomposition with a set of uniformly distributed weight vectors in the objective space, which selects the best qualified solution for each weight-vector-related aggregation function (the so-called subproblem) one by one, until the required number of solutions have been selected. Combined with the multi-task mechanism, the major differences of the proposed MTDEA from the traditional-decomposition based MOEAs are that it uses different ideal points (i.e., $Zmin$ in Algorithm 5) for different related tasks (also shown in Lines 5 to 11 in Algorithm 5) and uses a normalized modified inverse Tchebycheff (abbreviated as I-TCH) approach as the aggregation function for decomposition. A simple example of how to adaptively adjust the ideal point is given in Figure 4 for a more intuitive explanation.

Algorithm 5 $Select(\psi ,\tau ,K)$

Input: union subpopulation $\psi$, related task number $\tau$, required subpopulation size K; Output: required subpopulation $\psi$;   1: $W\leftarrow$ get a set of K normalized uniformly distributed weight vectors in the objective space for decomposition;   2: $S\leftarrow$ get a K-dimensional vector of false boolean values;   3: $Zmin\leftarrow$ get the best ideal point from $\psi$;   4: $Zmax\leftarrow$ get the worst ideal point from $\psi$;   5: if $\tau ==1$ then   6:       set the $f_1$ objective value of $Zmin$ to zero;   7: else if $\tau ==2$ then   8:       normalize objective values of $\psi$ by $Zmax\text{-}Zmin$;   9: else if $\tau ==3$ then 10:       set the $f_2$ objective value of $Zmin$ to zero; 11: end if 12: for $i=1,2,\dots ,K$ do 13:       $j\leftarrow$ get indexes of all the false boolean values in S; 14:       $fit\leftarrow$ get the fitness of all the solutions in $\psi \left(j\right)$ by       Equation (5), along with $Zmin$ and $W\left(i\right)$ input; 15:       $best\leftarrow$ get the index of the smallest value in $fit$; 16:       set the boolean value of $S\left(j\right(best\left)\right)$ to be true; 17: end for 18: $\psi =\psi \left(S\right)$; $//$ get the final selected solutions

In Figure 4, the ideal point $Zmin$ is drawn as stars for the three different task-related subpopulations. As previously shown in Lines 6 and 10 of Algorithm 5, $Zmin$ for the task-1 subpopulation is transformed by changing its $f_1$ objective value to zero, thereby moving the ideal point to the $f_2$ axis, as shown in Figure 4. The same goes for the ideal point of the task-3 subpopulation, which is moved to the $f_1$ axis by contrast, while that of the task-2 subpopulation has no transformation. In this way, the three different task-related subpopulations can have distinct evolutionary search behaviors, with their exploring preferences adjusted in different directions in the objective space, while task-1 prefers to search more of the sparse areas in the $f_1$ direction, task-3 prefers to search more of the sparse areas in the $f_2$ direction, and task-2 makes a balanced compromise between the other two tasks. It should also be noted that the weight vectors for each subpopulation are uniformly generated and distributed in the objective space, which uses the same method recommended in the classic MOEA/D algorithm [80]. In brief, they are uniformly sampled from a normalized hyperplane in the objective space; more details can be referred to in the literature for the MOEA/D [80].

However, this also requires that the final optimal solution for a weight vector should be obtained as closely as possible to both the weight vector itself and the ideal point $Zmin$. Thus, to better achieve this, the previously mentioned normalized modified I-TCH approach is then adopted in this paper as the aggregation function for decomposition, which can be formally defined as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \begin{array}{cc}\hfill & \hfill minimize{g}^{i\text{-}tch}\left(\mathit{x}\right|\mathit{w},\mathit{z})=\underset{1\le i\le M}{max}\left\{\raisebox{1ex}{$|f_i\left(\mathit{x}\right)-z_i|$}\!\left/ \!\raisebox{-1ex}{$w_i$}\right.\right\}\hfill \\ \hfill & \hfill subjectto\mathit{x}=(x_1,x_2,\dots ,x_D),x_i=\{0,1\}\hfill \end{array}\end{array}\end{array}$$

(5)

where the number of objectives M is set to two, and $\mathit{w}$ or $\mathit{z}$ respectively denotes the current weight vector $W\left(i\right)$ or the ideal point $Zmin$ in Algorithm 5, while $\mathit{x}$ denotes the D-dimensional decision vector of a certain solution. It can be seen that the above Equation (5) is almost the same as the previously introduced Equation (4) but uses $w_i$ as a denominator instead of a multiplier. In fact, one of the comparison algorithms (i.e., MOEA/AWA [83]) in our experiments also adopted the I-TCH approach for decomposition, which also proved its effectiveness, while more details can be found in the following reference [83]. It is also worth noting that Equation (5) can be treated as the final fitness function in this paper, while a smaller value of Equation (5) is preferred for environmental selection.

3.5. Task Merging Process

The task merging process of the MTDEA is shown in Algorithm 6, where the current population and the related task numbers are input. First, it is seen from Algorithm 6 that the merging of tasks is divided into the merging of task-1 and task-2, and the merging of task-3 and task-2. Nevertheless, they share the same merging rule based on the coverage of unique $f_1$ objective values from two different tasks. As adopted in Lines 5 and 12 of Algorithm 6, if the coverage rate is beyond half, or if task-1 or task-3 is totally covered by task-2 in the $f_1$ direction, then the other two tasks will be merged into task-2, and eventually, there will be only a single task-2 remaining for evolution. Moreover, an intuitive picture of the so-called coverage in the $f_1$ direction (also shown as ${\psi }_1$ and ${\psi }_2$ in Algorithm 6) is given in Figure 5 to further explain the above process, which takes the merging of task-1 and task-2 as an example. As can be seen from Figure 5, it is actually quite common in bi-objective feature selection that two different solutions may share the same $f_1$ value, meaning they own the same number of selected features (but probably have different feature combinations). Thus, the coverage of solutions in the $f_1$ direction is naturally suitable for use as a criterion for determining the distribution relationship between two subpopulations during evolution.

Algorithm 6 $Merge(Pop,T)$

Input: current population $Pop$, related tasks T; Output: updated tasks T;   1: ${\psi }_2\leftarrow$ get unique $f_1$ objective values of all the task-2 solutions in $Pop$ according to the task numbers in T;   2: if task-1 still exists then   3:       ${\psi }_1\leftarrow$ get unique $f_1$ objective values of all the task-1       solutions in $Pop$ according to the task numbers in T;   4:       $k\leftarrow$ get the number of all the duplicated elements       between ${\psi }_1$ and ${\psi }_2$;   5:       if $\left(\raisebox{1ex}{$k$}\!\left/ \!\raisebox{-1ex}{$Length\left({\psi }_2\right)$}\right.\right)>0.5\Vert k==Length\left({\psi }_1\right)$ then   6:              set all the task-1 numbers in T to be task-2;   7:       end if   8: end if   9: if task-3 still exists then 10:       ${\psi }_3\leftarrow$ get unique $f_1$ objective values of all the task-3       solutions in $Pop$ according to the task numbers in T; 11:       $k\leftarrow$ get the number of all the duplicated elements       between ${\psi }_3$ and ${\psi }_2$; 12:       if $\left(\raisebox{1ex}{$k$}\!\left/ \!\raisebox{-1ex}{$Length\left({\psi }_2\right)$}\right.\right)>0.5\Vert k==Length\left({\psi }_3\right)$ then 13:              set all the task-3 numbers in T to be task-2; 14:       end if 15: end if

4. Experimental Setups

4.1. Classification Datasets

A total of 20 open-source classification datasets [84] were used as test problems to evaluate the search performance of the compared MOEAs in tackling bi-objective feature selection. More detailed information of these datasets is given in

Table 1. Attributes of each dataset used for test problems.

No.	Dataset	Feature	Sample	Class
1	ORL	1024	400	40
2	Yale	1024	165	15
3	Colon	2000	62	2
4	SRBCT	2308	83	4
5	AR10P	2400	130	10
6	PIE10P	2420	210	10
7	Lymphoma	4026	96	9
9	DLBCL	5469	77	2
8	TOX171	5748	171	4
10	Brain1	5920	90	5
11	Leukemia	7070	72	2
12	CNS	7129	60	2
13	ALLAML	7129	72	2
14	Carcinom	9182	174	11
15	Nci9	9712	60	9
16	Arcene	10,000	200	2
17	Pixraw10P	10,000	100	10
18	Orlraws10P	10,304	100	10
19	Brain2	10,367	50	4
20	Prostate	10,509	102	2

, with the number of total features sorted in ascending order. It can be seen that the number of total features for each dataset varied from 1024 to 10,509, which covered a wide variety of feature dimensions and concentrated on high dimensionality.In addition, the number of samples varied from 50 to 400, and the classes ranged from 2 to 40, also suggesting the comprehensiveness of the test problems.

4.2. Comparison Algorithms

Seven state-of-the-art MOEAs were adopted in this paper as comparison algorithms against the proposed MTDEA, i.e., the NSGA-II (nondominated sorting-based genetic algorithm) [85], MOEA/D (MOEA based on decomposition) [80], MOEA/AWA (MOEA with adaptive weight adjustment) [83], MOEA/HD (MOEA based on hierarchical decomposition) [11], SparseEA (sparse evolutionary algorithm) [70], DAEA (duplication-analysis-based evolutionary algorithm) [71], and PRDH (problem reformation and duplication handling based algorithm) [72]. Among them, the NSGA-II and MOEA/D are among the most classic and well-known MOEAs based on dominance and decomposition, respectively. MOEA/AWA is also a decomposition-based MOEA but further modifies the adjustment of weight vectors in order to handle convex or disconnected Pareto fronts. Moreover, the MOEA/AWA also uses the same inverse Tchebycheff approach as the MTDEA, and thereby having comparative values. The MOEA/HD is a recently published MOEA based on dominance and decomposition, and especially designed for solving MOPs with complex Pareto fronts. The SparseEA, DAEA and PRDH are recently published MOEAs based on dominance and specifically designed for tackling large-scale and discrete MOPs, like bi-objective feature selection in classification.

4.3. Performance Indicators

In this work, multiple performance indicators (i.e., hypervolume [86], minimum classification error, and number of selected features [71]) were used to measure the comprehensive performance of the compared algorithms, in terms of both optimization and classification. More specifically, the hypervolume (HV) indicator was used as the main metric to measure the MOEAs’ general optimization performance, whose reference point was set to $(1,1)$. In addition, the minimum classification error (MCE) indicator and the number of selected features (NSF) indicator were used to measure the MOEAs’ classification performance, related to the best converged solution found in the $f_2$ direction (i.e., the solution with the best classification accuracy). Here, MCE denotes the classification error rate of the solution and NSF denotes the corresponding number of selected features. Normally, a greater HV value means better performance, while smaller MCE and NSF values are preferred by contrast. Finally, Wilcoxon’s test with a $5\%$ significance level was adopted to identify the significant differences between each pair of algorithms, while Friedman’s test was utilized to calculate the overall mean ranks among all algorithms.

4.4. Parameter Settings

In the experiment, all the compared algorithms adopted the same traditional initialization method, for the sake of fairness, while the reproduction methods and other parameter settings taken from the algorithms’ references. All algorithms were coded and run on an open-source MATLAB platform called PlatEMO [87]. For classification, each dataset was randomly divided into training and test subsets, with a proportion of $70/30$, according to the stratified split process [71]. Moreover, a KNN ($K=5$) classification model was utilized with 10-fold cross-validation on the training data, so as to avoid feature selection bias [88]. Last, each experiment was independently run 20 times with randomly preset starting seeds, while the population size for each algorithm was set to 100 and the termination criterion (i.e., the number of objective function evaluations) was set to 10,000 (about 100 generations).

5. Experimental Studies

In this section, we first study the general empirical results of the algorithms on each dataset, in terms of the three performance metrics. Then, we analyze the nondominated solution distributions obtained by each algorithm in the objective space during optimization. Moreover, the proposed MTDEA is further analyzed by comparing it with a modified benchmark algorithm to verify the contribution of its essential components. Finally, the computational time of each algorithm was recorded and compared to analyze the computational efficiency of the proposed algorithm.

5.1. General Performance Studies

The general performances of each algorithm run on all classification datasets are shown in

Table 2. General mean ranks calculated using Friedman’s test for each algorithm in terms of each metric, with best performances marked in gray.

Metric	MTDEA	NSGA-II	MOEA/D	MOEA/AWA	MOEA/HD	SparseEA	DAEA	PRDH
HV	1.012	5.215	3.917	5.290	6.275	7.530	2.192	4.567
MCE	3.615	4.688	5.124	4.787	5.030	4.303	3.740	4.714
NSF	1.000	5.270	3.112	5.640	6.461	7.949	2.110	4.457

,

Table 3. Mean HV performances of each algorithm on each dataset, with the best performance marked in gray, and insignificant differences prefixed by ⋆.

Dataset	MTDEA	NSGA-II	MOEA/D	MOEA/AWA	MOEA/HD	SparseEA	DAEA	PRDH
ORL	7.928e-01	6.103e-01	6.642e-01	6.576e-01	5.996e-01	5.859e-01	7.249e-01	6.350e-01
	±1.80e-02	±1.08e-02	±1.53e-02	±1.79e-02	±1.25e-02	±9.94e-03	±1.75e-02	±1.18e-02
Yale	6.597e-01	4.878e-01	5.127e-01	5.199e-01	4.920e-01	4.811e-01	6.050e-01	5.071e-01
	±3.18e-02	±1.34e-02	±2.62e-02	±2.40e-02	±3.39e-02	±1.64e-02	±2.48e-02	±2.15e-02
Colon	7.746e-01	5.500e-01	6.078e-01	5.466e-01	5.239e-01	4.987e-01	6.680e-01	5.554e-01
	±3.84e-02	±2.65e-02	±4.78e-02	±3.55e-02	±3.30e-02	±2.01e-02	±3.95e-02	±3.10e-02
SRBCT	6.654e-01	2.841e-01	3.178e-01	2.983e-01	2.554e-01	2.479e-01	3.028e-01	2.768e-01
	±1.70e-01	±2.05e-03	±2.03e-03	±2.74e-03	±1.81e-03	±1.69e-03	±1.96e-02	±3.17e-03
AR10P	4.972e-01	3.631e-01	3.709e-01	3.544e-01	3.449e-01	3.223e-01	4.314e-01	3.603e-01
	±2.74e-02	±2.01e-02	±2.26e-02	±1.35e-02	±1.75e-02	±1.11e-02	±2.05e-02	±1.86e-02
PIE10P	8.127e-01	6.023e-01	6.458e-01	6.003e-01	5.883e-01	5.434e-01	6.982e-01	6.056e-01
	±2.00e-02	±1.06e-02	±1.13e-02	±1.29e-02	±1.22e-02	±6.44e-03	±1.57e-02	±9.92e-03
Lymphoma	7.645e-01	5.603e-01	6.007e-01	5.598e-01	5.456e-01	5.067e-01	6.434e-01	5.626e-01
	±2.18e-02	±9.91e-03	±9.76e-03	±1.59e-02	±8.85e-03	±1.07e-02	±1.57e-02	±6.99e-03
TOX171	6.749e-01	4.829e-01	4.876e-01	4.697e-01	4.776e-01	4.575e-01	5.423e-01	4.907e-01
	±2.21e-02	±8.58e-03	±1.97e-02	±1.13e-02	±1.65e-02	±1.19e-02	±1.92e-02	±1.23e-02
DLBCL	8.054e-01	5.852e-01	5.999e-01	5.722e-01	5.735e-01	5.469e-01	6.703e-01	5.982e-01
	±3.06e-02	±2.02e-02	±1.82e-02	±2.17e-02	±1.67e-02	±1.83e-02	±2.12e-02	±1.77e-02
Brain1	6.496e-01	4.718e-01	4.906e-01	4.657e-01	4.535e-01	4.312e-01	5.127e-01	4.744e-01
	±1.38e-02	±3.11e-03	±1.09e-02	±1.06e-02	±1.00e-02	±1.78e-03	±8.53e-03	±2.57e-03
Leukemia	7.626e-01	5.360e-01	5.450e-01	5.295e-01	5.126e-01	4.972e-01	6.022e-01	5.454e-01
	±2.70e-02	±8.95e-03	±1.94e-02	±1.89e-02	±1.79e-02	±9.84e-03	±1.67e-02	±2.04e-02
CNS	5.255e-01	3.781e-01	3.743e-01	3.690e-01	3.707e-01	3.669e-01	4.408e-01	3.844e-01
	±5.88e-02	±3.28e-02	±3.32e-02	±3.09e-02	±3.25e-02	±1.57e-02	±3.03e-02	±2.94e-02
ALLAML	7.278e-01	5.205e-01	5.358e-01	5.084e-01	5.060e-01	4.853e-01	5.826e-01	5.280e-01
	±2.93e-02	±1.52e-02	±1.34e-02	±1.60e-02	±1.44e-02	±1.52e-02	±1.83e-02	±1.32e-02
Carcinom	7.199e-01	5.180e-01	5.233e-01	5.098e-01	5.091e-01	4.871e-01	5.809e-01	5.193e-01
	±1.40e-02	±1.09e-02	±1.55e-02	±1.00e-02	±1.10e-02	±8.18e-03	±1.18e-02	±7.20e-03
Nci9	3.570e-01	2.406e-01	2.616e-01	2.544e-01	2.370e-01	2.254e-01	2.707e-01	2.451e-01
	±2.33e-02	±2.54e-02	±2.94e-02	±2.80e-02	±2.21e-02	±2.00e-02	±2.75e-02	±2.71e-02
Arcene	5.132e-01	3.625e-01	3.724e-01	3.649e-01	3.445e-01	3.374e-01	3.859e-01	3.647e-01
	±4.15e-03	±1.10e-03	±1.85e-03	±2.23e-03	±1.24e-03	±1.29e-03	±2.75e-03	±1.71e-03
Pixraw10P	8.104e-01	5.795e-01	5.911e-01	5.773e-01	5.640e-01	5.407e-01	6.327e-01	5.846e-01
	±6.33e-03	±2.22e-03	±9.88e-03	±7.09e-03	±7.64e-03	±1.57e-03	±9.37e-03	±2.65e-03
Orlraws10P	7.491e-01	5.390e-01	5.447e-01	5.328e-01	5.297e-01	5.057e-01	5.951e-01	5.444e-01
	±1.43e-02	±7.53e-03	±9.58e-03	±1.13e-02	±8.00e-03	±3.88e-03	±8.65e-03	±4.77e-03
Brain2	5.603e-01	3.903e-01	3.824e-01	3.816e-01	3.782e-01	3.687e-01	4.335e-01	3.898e-01
	±2.84e-02	±2.15e-02	±2.43e-02	±2.91e-02	±2.12e-02	±1.66e-02	±2.82e-02	±1.83e-02
Prostate	6.456e-01	4.629e-01	4.599e-01	4.574e-01	4.559e-01	4.419e-01	5.204e-01	4.693e-01
	±2.38e-02	±1.29e-02	±1.51e-02	±1.58e-02	±1.16e-02	±8.71e-03	±1.53e-02	±1.52e-02

,

Table 4. Mean MCE performance of each algorithm on each dataset, with the best performances marked in gray, and insignificant differences prefixed by ⋆.

Dataset	MTDEA	NSGA-II	MOEA/D	MOEA/AWA	MOEA/HD	SparseEA	DAEA	PRDH
ORL	1.375e-01	⋆ 1.442e-01	⋆ 1.437e-01	1.475e-01	1.517e-01	1.471e-01	⋆ 1.304e-01	1.487e-01
	±1.49e-02	±2.01e-02	±1.35e-02	±1.30e-02	±2.03e-02	±1.33e-02	±1.49e-02	±1.39e-02
Yale	2.944e-01	3.478e-01	3.622e-01	3.367e-01	3.400e-01	⋆ 3.133e-01	⋆ 2.989e-01	3.500e-01
	±3.60e-02	±2.53e-02	±3.82e-02	±3.90e-02	±5.20e-02	±2.78e-02	±3.01e-02	±3.13e-02
Colon	1.421e-01	2.132e-01	2.079e-01	2.079e-01	2.342e-01	1.974e-01	1.684e-01	2.132e-01
	±4.86e-02	±4.35e-02	±6.50e-02	±5.53e-02	±5.26e-02	±3.77e-02	±5.01e-02	±4.97e-02
SRBCT	2.840e-01	6.400e-01	6.400e-01	6.400e-01	6.400e-01	6.400e-01	6.400e-01	6.400e-01
	±2.15e-01	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16
AR10P	4.688e-01	⋆ 4.900e-01	5.200e-01	4.962e-01	5.150e-01	5.075e-01	⋆ 4.700e-01	5.063e-01
	±3.52e-02	±3.38e-02	±3.77e-02	±2.47e-02	±3.08e-02	±2.00e-02	±3.10e-02	±3.02e-02
PIE10P	7.750e-02	9.667e-02	1.033e-01	9.583e-02	1.017e-01	1.017e-01	⋆ 8.583e-02	1.058e-01
	±2.25e-02	±1.28e-02	±2.27e-02	±1.42e-02	±1.42e-02	±1.07e-02	±1.82e-02	±1.56e-02
Lymphoma	1.200e-01	1.350e-01	⋆ 1.300e-01	⋆ 1.333e-01	⋆ 1.300e-01	⋆ 1.300e-01	⋆ 1.250e-01	1.333e-01
	±2.27e-02	±1.31e-02	±1.49e-02	±2.16e-02	±1.03e-02	±1.84e-02	±2.39e-02	±1.08e-02
TOX171	2.038e-01	2.236e-01	2.406e-01	2.330e-01	2.274e-01	⋆ 2.132e-01	⋆ 2.179e-01	2.208e-01
	±2.97e-02	±1.53e-02	±3.56e-02	±2.06e-02	±2.84e-02	±2.38e-02	±2.84e-02	±2.04e-02
DLBCL	7.174e-02	⋆ 5.870e-02	⋆ 7.174e-02	⋆ 6.957e-02	⋆ 6.304e-02	3.913e-02	4.130e-02	4.130e-02
	±3.80e-02	±3.53e-02	±2.92e-02	±3.28e-02	±2.63e-02	±3.43e-02	±2.98e-02	±2.98e-02
Brain1	2.593e-01	⋆ 2.593e-01	⋆ 2.593e-01	⋆ 2.593e-01	⋆ 2.593e-01	⋆ 2.593e-01	⋆ 2.593e-01	⋆ 2.593e-01
	±1.20e-02	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00
Leukemia	1.091e-01	1.318e-01	1.455e-01	1.364e-01	1.455e-01	⋆ 1.273e-01	⋆ 1.205e-01	⋆ 1.227e-01
	±3.73e-02	±1.40e-02	±2.80e-02	±2.95e-02	±2.80e-02	±1.87e-02	±2.67e-02	±3.64e-02
CNS	4.000e-01	⋆ 4.139e-01	⋆ 4.417e-01	⋆ 4.167e-01	⋆ 4.167e-01	⋆ 3.833e-01	⋆ 3.833e-01	⋆ 4.083e-01
	±7.77e-02	±6.11e-02	±5.55e-02	±6.11e-02	±5.84e-02	±3.07e-02	±4.73e-02	±5.49e-02
ALLAML	1.455e-01	⋆ 1.568e-01	⋆ 1.591e-01	1.727e-01	⋆ 1.636e-01	⋆ 1.500e-01	⋆ 1.500e-01	⋆ 1.523e-01
	±3.79e-02	±2.75e-02	±2.33e-02	±2.80e-02	±2.28e-02	±2.99e-02	±2.60e-02	±2.22e-02
Carcinom	1.356e-01	⋆ 1.423e-01	1.500e-01	⋆ 1.394e-01	1.500e-01	⋆ 1.385e-01	⋆ 1.327e-01	1.471e-01
	±1.59e-02	±1.91e-02	±2.30e-02	±1.51e-02	±2.03e-02	±1.60e-02	±1.64e-02	±1.29e-02
Nci9	6.289e-01	6.579e-01	⋆ 6.342e-01	⋆ 6.368e-01	⋆ 6.421e-01	6.553e-01	⋆ 6.316e-01	⋆ 6.526e-01
	±3.18e-02	±4.68e-02	±5.26e-02	±5.09e-02	±4.39e-02	±4.00e-02	±4.83e-02	±4.95e-02
Arcene	4.333e-01	⋆ 4.333e-01	⋆ 4.333e-01	⋆ 4.333e-01	⋆ 4.333e-01	⋆ 4.333e-01	⋆ 4.333e-01	⋆ 4.333e-01
	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16	±1.14e-16
Pixraw10P	3.333e-02	⋆ 3.333e-02	⋆ 3.333e-02	⋆ 3.333e-02	⋆ 3.333e-02	⋆ 3.333e-02	⋆ 3.333e-02	⋆ 3.333e-02
	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00	±0.00e+00
Orlraws10P	1.033e-01	⋆ 1.050e-01	⋆ 1.117e-01	⋆ 1.067e-01	⋆ 1.050e-01	⋆ 1.017e-01	⋆ 1.033e-01	⋆ 1.017e-01
	±1.84e-02	±1.22e-02	±1.63e-02	±1.37e-02	±1.22e-02	±7.45e-03	±1.03e-02	±7.45e-03
Brain2	3.500e-01	3.767e-01	3.967e-01	3.833e-01	3.900e-01	⋆ 3.700e-01	⋆ 3.733e-01	3.833e-01
	±3.67e-02	±3.91e-02	±4.03e-02	±5.24e-02	±3.91e-02	±3.40e-02	±4.54e-02	±2.96e-02
Prostate	2.355e-01	⋆ 2.419e-01	2.613e-01	⋆ 2.387e-01	⋆ 2.435e-01	⋆ 2.274e-01	⋆ 2.258e-01	⋆ 2.371e-01
	±3.33e-02	±2.45e-02	±2.94e-02	±2.65e-02	±1.95e-02	±1.65e-02	±2.56e-02	±2.82e-02

and

Table 5. Mean NSF performance of each algorithm on each dataset, with best performances marked in gray, and insignificant differences prefixed by ⋆.

Dataset	MTDEA	NSGA-II	MOEA/D	MOEA/AWA	MOEA/HD	SparseEA	DAEA	PRDH
ORL	1.301e+02	3.498e+02	2.771e+02	3.257e+02	3.516e+02	4.001e+02	2.272e+02	3.054e+02
	±2.37e+01	±2.55e+01	±1.95e+01	±8.52e+01	±1.56e+01	±3.34e+01	±3.23e+01	±1.35e+01
Yale	1.368e+02	3.340e+02	2.682e+02	3.149e+02	3.291e+02	3.848e+02	2.311e+02	2.966e+02
	±2.19e+01	±2.99e+01	±1.46e+01	±5.93e+01	±1.68e+01	±2.03e+01	±4.03e+01	±1.94e+01
Colon	2.518e+02	6.990e+02	5.509e+02	7.224e+02	7.374e+02	8.752e+02	4.804e+02	6.866e+02
	±2.74e+01	±1.50e+01	±5.59e+01	±4.67e+01	±2.43e+01	±3.37e+01	±3.17e+01	±1.98e+01
SRBCT	2.712e+02	8.142e+02	6.096e+02	7.275e+02	9.884e+02	1.034e+03	7.004e+02	8.581e+02
	±5.61e+01	±1.24e+01	±1.23e+01	±1.66e+01	±1.10e+01	±1.03e+01	±1.19e+02	±1.93e+01
AR10P	3.889e+02	9.154e+02	7.826e+02	9.657e+02	9.311e+02	1.081e+03	6.920e+02	8.866e+02
	±4.58e+01	±1.77e+01	±2.61e+01	±5.67e+01	±2.85e+01	±3.27e+01	±7.48e+01	±1.65e+01
PIE10P	3.518e+02	9.076e+02	7.648e+02	9.489e+02	9.400e+02	1.085e+03	6.721e+02	8.814e+02
	±3.11e+01	±2.51e+01	±2.78e+01	±8.51e+01	±2.50e+01	±2.95e+01	±3.75e+01	±1.65e+01
Lymphoma	6.319e+02	1.600e+03	1.412e+03	1.607e+03	1.688e+03	1.890e+03	1.239e+03	1.598e+03
	±4.38e+01	±2.11e+01	±2.43e+01	±7.12e+01	±4.38e+01	±1.56e+01	±7.37e+01	±2.49e+01
TOX171	1.126e+03	2.510e+03	2.376e+03	2.594e+03	2.530e+03	2.768e+03	2.064e+03	2.465e+03
	±7.98e+01	±6.23e+01	±3.48e+01	±8.09e+01	±3.03e+01	±3.54e+01	±6.63e+01	±4.24e+01
DLBCL	8.355e+02	2.300e+03	2.154e+03	2.342e+03	2.361e+03	2.626e+03	1.838e+03	2.287e+03
	±5.62e+01	±2.54e+01	±9.42e+01	±7.18e+01	±4.26e+01	±2.31e+01	±7.16e+01	±2.78e+01
Brain1	9.804e+02	2.492e+03	2.332e+03	2.544e+03	2.648e+03	2.838e+03	2.143e+03	2.470e+03
	±9.27e+01	±2.65e+01	±9.29e+01	±9.00e+01	±8.53e+01	±1.52e+01	±7.27e+01	±2.19e+01
Leukemia	1.201e+03	3.041e+03	2.893e+03	3.081e+03	3.184e+03	3.423e+03	2.541e+03	3.010e+03
	±5.41e+01	±3.44e+01	±1.06e+02	±9.67e+01	±5.71e+01	±3.21e+01	±8.82e+01	±4.22e+01
CNS	1.375e+03	3.102e+03	2.939e+03	3.193e+03	3.172e+03	3.449e+03	2.550e+03	3.064e+03
	±8.44e+01	±7.67e+01	±6.44e+01	±8.03e+01	±4.51e+01	±4.15e+01	±5.91e+01	±8.10e+01
ALLAML	1.273e+03	3.085e+03	2.930e+03	3.112e+03	3.182e+03	3.450e+03	2.566e+03	3.043e+03
	±7.06e+01	±2.61e+01	±4.67e+01	±7.95e+01	±4.63e+01	±3.33e+01	±5.94e+01	±4.01e+01
Carcinom	1.853e+03	4.097e+03	3.981e+03	4.240e+03	4.149e+03	4.493e+03	3.480e+03	4.079e+03
	±9.64e+01	±3.82e+01	±6.53e+01	±7.39e+01	±3.06e+01	±3.26e+01	±1.01e+02	±8.56e+01
Nci9	1.777e+03	4.288e+03	4.083e+03	4.230e+03	4.600e+03	4.727e+03	3.890e+03	4.245e+03
	±1.07e+02	±3.25e+01	±3.29e+01	±2.61e+01	±2.86e+01	±1.76e+01	±4.30e+01	±1.69e+01
Arcene	1.686e+03	4.421e+03	4.240e+03	4.377e+03	4.748e+03	4.875e+03	3.996e+03	4.380e+03
	±7.54e+01	±1.99e+01	±3.35e+01	±4.04e+01	±2.25e+01	±2.34e+01	±5.00e+01	±3.10e+01
Pixraw10P	1.807e+03	4.426e+03	4.295e+03	4.452e+03	4.603e+03	4.866e+03	3.823e+03	4.368e+03
	±7.19e+01	±2.52e+01	±1.12e+02	±8.04e+01	±8.66e+01	±1.79e+01	±1.06e+02	±3.00e+01
Orlraws10P	1.971e+03	4.581e+03	4.463e+03	4.648e+03	4.697e+03	5.023e+03	3.890e+03	4.536e+03
	±6.24e+01	±3.63e+01	±4.45e+01	±8.97e+01	±5.78e+01	±2.67e+01	±7.47e+01	±3.49e+01
Brain2	2.038e+03	4.642e+03	4.584e+03	4.734e+03	4.729e+03	5.084e+03	3.935e+03	4.591e+03
	±7.39e+01	±4.55e+01	±1.21e+02	±9.91e+01	±4.01e+01	±3.71e+01	±8.19e+01	±4.58e+01
Prostate	2.081e+03	4.713e+03	4.587e+03	4.820e+03	4.796e+03	5.154e+03	4.024e+03	4.670e+03
	±9.25e+01	±5.72e+01	±5.63e+01	±8.07e+01	±4.31e+01	±4.61e+01	±1.16e+02	±6.92e+01

. First of all, it is suggested from the overall mean ranks of each algorithm on all datasets, in terms of the Friedman’s test as shown in Table 2, that the proposed MTDEA always ranked first for all three performance metrics (HV, MCE, and NSF). In contrast, the DAEA generally fell behind the MTDEA and ranked second in terms of all three metrics, while the SparseEA seemed to perform much better in terms of MCE than for the other two metrics. Generally, the highest HV rank being for the MTDEA implied its comprehensive advantages in both diversity and convergence in multi-objective optimization, while the highest MCE rank being for MTDEA implied its overall superiority in finding solutions with the best classification accuracy.In addition, the highest NSF rank being for the MTDEA suggests its promising computational efficiency, because the number of selected features generally affects the complexity of classification models for learning (normally a larger number of selected features leads to a higher computational cost of classification). The computational time of each algorithm will be further analyzed in Section 5.4.

The more detailed performance of each algorithm on every dataset, in terms of all three metrics, is presented in Table 3, Table 4 and Table 5, respectively. It is seen from Table 3, Table 4 and Table 5 that the MTDEA generally performed the best on all datasets in Table 3 and Table 5 in terms of the HV and NSF metrics, but it encountered a few insignificant losses in Table 4 in terms of the MCE metric. To be more specific, in Table 4, although the MTDEA performed the best on 14 out of 20 datasets, it lost out slightly on the 6 others, either against the DAEA or SparseEA, which are two recently published MOEAs specially designed for tackling bi-objective feature selection or large-scale discrete optimization. The very few insignificant setbacks of the MTDEA for the MCE performance were mainly because the MTDEA emphasizes the dynamic balance between diversity and convergence, while MCE mainly focuses on finding the solution with the best classification accuracy. In addition, as the “no free lunch” theory says, it is also impossible for a single method to best solve all kinds of problems.

5.2. Nondominated Solution Distributions

For a more intuitive view of the performances, Figure 6 illustrates the nondominated solution distributions of each algorithm, in terms of Pareto curves in the objective space, with the median HV performances run on each dataset. We only show the nondominated solutions instead of the whole population, for a clearer representation of the diversity and convergence states. It is also worth noting that the nondominated solutions shown in each subfigure of Figure 6 look quite sparse for most datasets, due to the limited number of training samples and the large number of features. General speaking, it is suggested from Figure 6 that the proposed MTDEA performed the best on the majority of the classification datasets, with a promising overall diversity and convergence.

In detail, the MTDEA generally obtained the largest number of nondominated solutions with best diversity, while the solution distributions in the $f_1$ and $f_2$ directions of objective space were overall the most converged. However, there are still a few unsatisfactory performances shown in some subfigures, such as Figure 6a,h,i, where the MTDEA was not the best (but still good) of all the algorithms in the $f_2$ objective direction (i.e., the classification error rate). Nevertheless, the few flaws of the MTDEA are generally insignificant and it was generally close to the best algorithms in the $f_2$ objective direction, while the MTDEA always exhibited the best performance in the $f_1$ objective direction (i.e., the selected feature rate). It should also be noted that although we choose the performance for the median HV values obtained by each algorithm for the sake of fairness, they have a certain degree of random fluctuation and can only be seen as a supplement to the previously studied statistical empirical results in Section 5.1.

5.3. Component Contribution Analyses

To further confirm the effectiveness of the proposed MTDEA, especially to verify the contributions of its essential component (i.e., the dynamic multi-task framework with adaptive ideal points for decomposition), rather than adopting other existing methods (such as the I-TCH based decomposition approach and normalization), the MTDEA and its Baseline algorithm were compared in terms of all three performance metrics, as shown in

Table 6. Mean HV, MCE, and NSF performances of the MTDEA against the Baseline algorithm, with best performances marked in gray and insignificant differences prefixed by ⋆.

Dataset	HV Metric		MCE Metric		NSF Metric
	MTDEA	Baseline	MTDEA	Baseline	MTDEA	Baseline
ORL	7.9280e-01	6.2015e-01	1.3750e-01	⋆ 1.4250e-01	1.3005e+02	3.3955e+02
	±1.797e-02	±2.020e-02	±1.493e-02	±1.402e-02	±2.373e+01	±2.964e+01
Yale	6.5971e-01	4.9766e-01	2.9444e-01	3.3556e-01	1.3675e+02	3.3045e+02
	±3.181e-02	±2.479e-02	±3.596e-02	±3.808e-02	±2.190e+01	±1.961e+01
Colon	7.7459e-01	5.4779e-01	1.4211e-01	2.1316e-01	2.5180e+02	7.0630e+02
	±3.839e-02	±2.682e-02	±4.860e-02	±4.345e-02	±2.739e+01	±1.718e+01
SRBCT	6.6537e-01	2.8661e-01	2.8400e-01	6.4000e-01	2.7120e+02	7.9880e+02
	±1.696e-01	±1.828e-03	±2.152e-01	±1.139e-16	±5.614e+01	±1.110e+01
AR10P	4.9725e-01	3.5777e-01	4.6875e-01	4.9750e-01	3.8895e+02	9.2410e+02
	±2.744e-02	±1.439e-02	±3.524e-02	±2.420e-02	±4.576e+01	±2.474e+01
PIE10P	8.1273e-01	5.9713e-01	7.7500e-02	1.0083e-01	3.5175e+02	9.1285e+02
	±2.001e-02	±9.652e-03	±2.247e-02	±1.144e-02	±3.107e+01	±2.042e+01
Lymphoma	7.6449e-01	5.6584e-01	1.2000e-01	⋆ 1.3167e-01	6.3190e+02	1.5825e+03
	±2.178e-02	±8.253e-03	±2.269e-02	±1.701e-02	±4.382e+01	±2.530e+01
TOX171	6.7490e-01	4.8274e-01	2.0377e-01	2.2736e-01	1.1264e+03	2.5053e+03
	±2.214e-02	±1.347e-02	±2.974e-02	±2.247e-02	±7.979e+01	±8.390e+01
DLBCL	8.0543e-01	5.7887e-01	7.1739e-02	⋆ 7.6087e-02	8.3550e+02	2.2779e+03
	±3.059e-02	±2.118e-02	±3.805e-02	±3.419e-02	±5.622e+01	±3.618e+01
Brain1	6.4956e-01	4.7324e-01	2.5926e-01	⋆ 2.5926e-01	9.8040e+02	2.4799e+03
	±1.384e-02	±5.628e-03	±1.202e-02	±0.000e+00	±9.274e+01	±4.795e+01
Leukemia	7.6257e-01	5.4152e-01	1.0909e-01	⋆ 1.3182e-01	1.2007e+03	2.9922e+03
	±2.699e-02	±1.669e-02	±3.731e-02	±2.912e-02	±5.407e+01	±3.736e+01
CNS	5.2552e-01	3.7420e-01	4.0000e-01	⋆ 4.2500e-01	1.3754e+03	3.0789e+03
	±5.878e-02	±2.658e-02	±7.774e-02	±4.862e-02	±8.443e+01	±6.460e+01
ALLAML	7.2778e-01	5.2572e-01	1.4545e-01	⋆ 1.5227e-01	1.2732e+03	3.0622e+03
	±2.933e-02	±1.071e-02	±3.789e-02	±2.224e-02	±7.057e+01	±4.950e+01
Carcinom	7.1986e-01	5.2429e-01	1.3558e-01	⋆ 1.3462e-01	1.8533e+03	4.0858e+03
	±1.404e-02	±8.217e-03	±1.588e-02	±1.395e-02	±9.638e+01	±6.919e+01
Nci9	3.5698e-01	2.4945e-01	6.2895e-01	⋆ 6.4737e-01	1.7768e+03	4.2062e+03
	±2.326e-02	±2.105e-02	±3.183e-02	±3.856e-02	±1.071e+02	±4.303e+01
Arcene	5.1318e-01	3.6631e-01	4.3333e-01	⋆ 4.3333e-01	1.6858e+03	4.3514e+03
	±4.154e-03	±1.934e-03	±1.139e-16	±1.139e-16	±7.540e+01	±3.511e+01
Pixraw10P	8.1039e-01	5.8583e-01	3.3333e-02	⋆ 3.3333e-02	1.8071e+03	4.3545e+03
	±6.335e-03	±4.056e-03	±0.000e+00	±0.000e+00	±7.186e+01	±4.601e+01
Orlraws10P	7.4909e-01	5.4469e-01	1.0333e-01	⋆ 1.0333e-01	1.9710e+03	4.5222e+03
	±1.430e-02	±6.752e-03	±1.842e-02	±1.026e-02	±6.243e+01	±3.812e+01
Brain2	5.6026e-01	3.8877e-01	3.5000e-01	3.8333e-01	2.0377e+03	4.6294e+03
	±2.838e-02	±1.692e-02	±3.667e-02	±2.962e-02	±7.393e+01	±8.246e+01
Prostate	6.4555e-01	4.6144e-01	2.3548e-01	⋆ 2.4839e-01	2.0812e+03	4.6716e+03
	±2.378e-02	±1.361e-02	±3.326e-02	±2.585e-02	±9.247e+01	±4.519e+01

. In this paper, the so-called Baseline algorithm removed all the multi-task contents from Algorithm 1, replacing Algorithms 2 and 4 respectively with traditional initialization and reproduction methods, which are commonly used in most MOEAs [80,85], while its environmental selection process still used the modified and normalized I-TCH based decomposition approach as previously introduced. It is suggested from Table 6 that the MTDEA generally showed outstanding performance, with significant advantages over the Baseline algorithm on almost all datasets. The only insignificant loss for the MTDEA took place on the Carcinom dataset in terms of the MCE metric, which was slightly worse than the Baseline algorithm. By contrast, the performance of the MTDEA in terms of the HV and NSF metrics were the best on all datasets compared with the Benchmark algorithm. Thus, it was demonstrated that the essential component of the MTDEA made an overall positive contribution in improving the performance in finding optimal or near-optimal solutions in the large-scale search space of the high-dimensional datasets.

5.4. Computational Time Analyses

The mean computational time for each algorithm run on each dataset was recorded and calculated in seconds, as shown in

Table 7. Mean computational time in seconds for each algorithm run on each dataset, with best times marked in gray, and insignificant differences prefixed by ⋆. Moreover, if MTDEA performed second best of all, it is also marked in a lighter gray color.

Dataset	MTDEA	NSGA-II	MOEA/D	MOEA/AWA	MOEA/HD	SparseEA	DAEA	PRDH
ORL	7.289e+02	1.071e+03	9.947e+02	9.512e+02	1.084e+03	1.065e+03	8.724e+02	1.053e+03
	±2.43e+01	±3.74e+01	±3.94e+01	±3.00e+01	±4.59e+01	±2.32e+01	±3.81e+01	±2.42e+01
Yale	1.434e+02	1.942e+02	1.860e+02	1.910e+02	2.032e+02	2.028e+02	1.790e+02	2.022e+02
	±4.69e+00	±4.97e+00	±3.54e+00	±3.57e+00	±6.39e+00	±4.44e+00	±4.73e+00	±4.74e+00
Colon	6.170e+01	8.630e+01	7.946e+01	8.715e+01	8.672e+01	8.419e+01	8.861e+01	9.450e+01
	±1.06e+00	±1.55e+00	±1.58e+00	±1.85e+00	±2.60e+00	±1.84e+00	±1.74e+00	±1.65e+00
SRBCT	1.653e+02	2.327e+02	2.078e+02	2.154e+02	2.618e+02	2.254e+02	2.078e+02	2.558e+02
	±5.26e+00	±5.64e+00	±2.87e+00	±5.34e+00	±7.01e+00	±5.98e+00	±1.03e+01	±3.74e+00
AR10P	4.898e+02	6.246e+02	6.008e+02	6.226e+02	6.349e+02	5.974e+02	5.872e+02	6.314e+02
	±2.02e+01	±9.67e+00	±1.52e+01	±1.60e+01	±1.30e+01	±1.74e+01	±1.11e+01	±1.02e+01
PIE10P	9.958e+02	1.462e+03	1.382e+03	1.469e+03	1.539e+03	1.411e+03	1.198e+03	1.415e+03
	±3.44e+01	±9.16e+01	±6.56e+01	±1.03e+02	±9.56e+01	±6.73e+01	±5.89e+01	±5.14e+01
Lymphoma	6.800e+02	9.464e+02	8.585e+02	9.201e+02	9.785e+02	8.230e+02	8.358e+02	8.834e+02
	±1.93e+01	±5.54e+01	±2.14e+01	±5.38e+01	±7.62e+01	±5.04e+01	±3.06e+01	±1.62e+01
TOX171	2.417e+03	3.248e+03	3.065e+03	3.128e+03	3.300e+03	2.301e+03	2.826e+03	3.013e+03
	±1.96e+02	±2.29e+02	±2.00e+02	±2.37e+02	±2.51e+02	±1.68e+02	±3.77e+01	±1.10e+02
DLBCL	8.218e+02	1.135e+03	1.083e+03	1.109e+03	1.159e+03	9.196e+02	1.032e+03	1.064e+03
	±3.99e+01	±7.51e+01	±7.73e+01	±7.66e+01	±8.80e+01	±7.54e+01	±2.52e+01	±1.94e+01
Brain1	1.224e+03	1.731e+03	1.715e+03	1.726e+03	1.789e+03	1.266e+03	1.498e+03	1.593e+03
	±1.08e+02	±1.44e+02	±1.56e+02	±1.42e+02	±1.59e+02	±1.08e+02	±8.45e+01	±6.91e+01
Leukemia	1.150e+03	1.648e+03	1.584e+03	1.705e+03	1.708e+03	1.080e+03	1.379e+03	1.537e+03
	±9.20e+01	±1.36e+02	±1.28e+02	±1.89e+02	±1.28e+02	±9.02e+01	±5.30e+01	±8.69e+01
CNS	9.885e+02	1.199e+03	1.145e+03	1.161e+03	1.217e+03	8.565e+02	1.077e+03	1.122e+03
	±8.98e+01	±8.43e+01	±8.29e+01	±8.87e+01	±9.35e+01	±6.03e+01	±2.52e+01	±2.03e+01
ALLAML	1.175e+03	1.677e+03	1.648e+03	1.705e+03	1.731e+03	1.086e+03	1.407e+03	1.558e+03
	±9.54e+01	±1.43e+02	±1.51e+02	±1.90e+02	±1.42e+02	±8.30e+01	±5.50e+01	±8.15e+01
Carcinom	4.527e+03	5.767e+03	5.719e+03	5.692e+03	5.860e+03	3.349e+03	4.934e+03	5.280e+03
	±4.36e+02	±5.05e+02	±4.51e+02	±4.65e+02	±6.06e+02	±3.03e+01	±1.35e+02	±2.00e+02
Nci9	1.348e+03	1.940e+03	1.891e+03	1.926e+03	2.276e+03	1.228e+03	1.818e+03	1.867e+03
	±9.02e+01	±1.58e+02	±9.31e+01	±8.43e+01	±3.22e+02	±1.69e+02	±4.41e+01	±1.14e+02
Arcene	5.231e+03	7.230e+03	6.937e+03	7.047e+03	7.745e+03	4.329e+03	8.062e+03	6.646e+03
	±1.97e+02	±5.42e+02	±4.00e+02	±3.64e+02	±9.04e+02	±4.43e+02	±1.38e+03	±3.93e+02
Pixraw10P	2.564e+03	3.430e+03	3.382e+03	3.429e+03	3.694e+03	1.989e+03	3.845e+03	3.276e+03
	±1.83e+02	±1.94e+02	±2.31e+02	±2.45e+02	±3.92e+02	±2.61e+02	±5.98e+02	±2.56e+02
Orlraws10P	2.693e+03	3.628e+03	3.547e+03	3.598e+03	4.208e+03	⋆ 2.602e+03	3.944e+03	3.400e+03
	±1.63e+02	±3.12e+02	±2.06e+02	±2.57e+02	±8.63e+02	±2.36e+02	±6.29e+02	±2.76e+02
Brain2	1.230e+03	1.789e+03	1.708e+03	1.723e+03	1.978e+03	1.170e+03	1.775e+03	1.661e+03
	±6.56e+01	±1.49e+02	±1.66e+02	±1.44e+02	±3.51e+02	±6.17e+01	±2.42e+02	±1.03e+02
Prostate	2.857e+03	3.906e+03	3.762e+03	3.775e+03	4.505e+03	⋆ 2.676e+03	3.546e+03	3.426e+03
	±1.36e+02	±1.63e+02	±1.51e+02	±2.99e+02	±1.02e+03	±7.03e+02	±1.71e+02	±1.46e+02

. In Table 7, we not only mark the best performance in gray but also mark the second best performance in a lighter gray color, in order to provide more comprehensive analyses. First, it can be seen from Table 7 that the MTDEA performed the best on 9 out of 20 datasets, and it also performed second best on the remaining 11 datasets. Thus, it can be concluded that the MTDEA had a generally high computational efficiency, spending much less time than the traditional algorithms in tackling feature selection. Moreover, when looking into the 11 second best performances marked in the lighter gray color, it is seen that the MTDEA only just lost to SparseEA in terms of computational time, while most of the losses were on very high-dimensional datasets. In fact, the reason why the SparseEA could spend even less time than the proposed MTDEA running on those datasets was mainly due to its pre-analysis process before the normal evolution. In brief, the SparseEA pre-analyzed each feature’s classification performance, which actually consumed much less time than the subsequent normal evolution, because only a single feature was selected for classification.

6. Conclusions

This paper proposed a multi-task decomposition-based evolutionary algorithm (named MTDEA) for solving the discrete bi-objective feature selection problem with binary-coding, especially for large-scale classification data. The proposed MTDEA not only adopts an adaptive decomposition-based MOEA framework with transformed ideal points for more efficient evolution, but also has a multi-task mechanism with dynamically merged tasks for more diverse cooperation. In this work, we focused on studying the adaptive integration of the multi-task framework with the decomposition-based evolutionary approach. To be more specific, first, a dynamic multi-task framework was designed, which initializes different search tasks for three separate subpopulations and then eventually merges them into a single-task population at the later evolutionary stage. Furthermore, an adaptive decomposition-based evolutionary approach was also designed, which cooperates with the above multi-task framework and adaptively adjusts the global ideal point for each multi-task subpopulation. In this way, the search performance of the MTDEA was significantly enhanced, while the population diversity and the classification convergence could also be delicately balanced. It is suggested from the comprehensive empirical studies that the MTDEA significantly outperformed the seven other state-of-the-art MOEAs on a total of 20 high-dimensional classification datasets, in terms of three different performance metrics, with a better distribution diversity and classification convergence. Moreover, it was also shown that the most essential component of the MTDEA, i.e., the dynamic multi-task framework with adaptive ideal points for decomposition, made an overall positive contribution to the improvement in the algorithm performance.

In our future work, it is planned to study the adaptive setting of initial parameters for the multi-task framework, probably based on the dynamic analysis of the optimization environment of different tested datasets.Moreover, we would also like to further study the performance of the proposed MTDEA on more high-dimensional datasets, as well as its applicability to other kinds of discrete optimization problems such as neural network construction and community node detection.