*4.3. Optimization Behavior*

Based on the previous results, for this experiment, we selected 50 · 10<sup>3</sup> and 10 · 10<sup>6</sup> function evaluations as the stopping criteria for the KNP and the TSP, respectively. As shown before, since the solutions stop improving by that point for any of the approaches, we can reduce the computational effort without losing generality in the analysis to be performed. Moreover, in this second experiment, we ran the complete set of instances and did the comparison at the end of the executions, once the corresponding stopping criterion was reached. Note that all the algorithms were executed 100 times. In order to compare the results obtained by the multi-objective approaches with those achieved by the single-objective optimizers, we calculated the extreme solutions in the Pareto optimal set, which correspond to the best solution attained for each objective function.

The results shown in Tables 5 and 6 correspond to the KNP problem, considering objective 1 and objective 2, respectively. Similarly, the results in Tables 7 and 8 correspond to the TSP. To facilitate the analysis, the mean and median solution values for each problem-instance-algorithm have been normalized as relative measures. Such relative solution values are expressed as a percentage with respect to the best corresponding solution. For each problem instance we fixed the best solution as the best value found for a particular objective across the complete set of related executions. For each instance and objective, we have performed a total of 500 executions (5 algorithms × 100 executions each). From this total of 500 values, we fixed the best one as our reference to calculate the percentage of solution quality obtained by each proposal (as shown in the tables). Furthermore, for each instance, the cells containing the best median results have a gray background. Finally, the last column shows, for each instance considered, whether statistically significant differences arose when comparing the best-performing multi-objective approach against the best single-objective method by using the statistical comparison procedure described at the beginning of this section. The best-performing schemes are those that exhibit the best mean and median of the objective function for each test case. If any statistically significant differences exist, i.e., the p-value obtained from the statistical comparison procedure is lower than the significance level, an 'S' if shown if the corresponding single-objective algorithm provides a better mean and median of the corresponding objective function. If the best mean and median are provided by the corresponding multi-objective approach, an 'M' is shown. Finally, for those test cases where the two algorithms exhibit no statistically significant differences, a '-' is shown.




**Table 5.** *Cont*.

#### **Table 6.** Results for KNP instances (objective 2).



**Table 6.** *Cont*.

In the case of the KNP, we see that, in most test cases, the SOEAs obtain the best results, especially gGA, for both objective functions (see Tables 5 and 6). In fact, for those cases, gGA is statistically superior to the corresponding multi-objective algorithms. However, the results of the best-performing MOEAs are very close to those obtained by the best-performing SOEAs. Particularly, we should note the behavior of NSGA-II when optimizing objective 1 of the strongly correlated instances Set2/STRONG (see Table 5). NSGA-II not only provides the best solutions, but it is also statistically superior to gGA in all instances belonging to that group. For those instances, NSGA-II is followed by MOEA/D and SMS-EMOA in the ranking.

With regard to objective 1, Figure 5 shows more information on this ranking, and also that gGA is close to the SMS-EMOA but never exceeds it, ranking fourth. We see that eES is ranked last, well behind the remaining algorithms. In general, for objective 1, the SOEAs are statistically superior in 79% of the instances, the MOEAs in 19%, while in 2% of the instances, the algorithms did not exhibit statistically significant differences between the two approaches. Table 5 also shows that MOEA/D ranks second, with 38%, surpassing the eES in these cases. Table 6 shows the KNP results for objective 2, where we can see that gGA again yielded the best results in 93% of the instances, 1% for MOEAs, while 6% present no statistically significant differences.

In this case, eES swapped the second position in the ranking with MOEA/D (for the Set2/UNCOR and Set2/WEAK instances), where MOEA/D ranks second in 27% of the cases. As a result, we can conclude that, when dealing with strongly correlated instances of the KNP, NSGA-II provides the best results, and in fact has to be executed only once, rather than the multiple executions required with a single-objective approach, like gGA, which would have to be executed twice, one run per objective function being optimized. The above would result in significant savings in terms of the computational resources required to solve this type of instance.

**Figure 5.** Boxplots showing the results for the KNP (strongly correlated instances) achieved by the different single-objective and multi-objective approaches at the end of 100 repetitions of the runs. Some instances were omitted because of space restrictions. However, all graphics can be found in the repository associated with this paper.


**Table 7.** Results for TSP instances (objective 1).

**Table 8.** Results for TSP instances (objective 2).


Regarding the TSP, the results for objective 1 (Table 7) and objective 2 (Table 8), show hardly any differences. In both cases, NSGA-II was the best-performing approach, not only considering almost all small instances, but also some large ones. Furthermore, in those cases where NSGA-II was superior, the differences were statistically significant compared the corresponding best-performing single-objective approach. As in the case of the strongly correlated instances of the KNP, for those particular instances of the TSP, it is better to run a multi-objective approach, such as NSGA-II, instead of running a single-objective algorithm. As a first approach, decision makers usually tend to perform a transformation of a multi-objective problem into a single-objective one, in the case they are interested in a particular objective of a multi-objective problem. The said transformation is carried out either by performing a scalarization of the different objective functions or by redefining objective functions as constraints. Bearing the above in mind, although practitioners are only focused on one of the objective functions of a multi-objective problem, the quality of the solutions attained by the direct application of a multi-objective optimizer could be higher in comparison to the quality of the solutions achieved by a single-objective algorithm executed for each of the objective functions independently. As a result, from the practical point of view, the application of a multi-objective solver to a multi-objective problem could be a much better option rather than performing a transformation of the multi-objective problem into a single-objective one to solve it through a single-objective approach.

Finally, we note that for most instances with a size between 150 and 300, MOEAs are dominated by SOEAs. In larger instances, the SMS-EMOA tends to be superior to the other approaches. In general, and considering both objective functions, the MOEAs are statistically superior in 69% of the instances, SOEAs in 21%, while 10% exhibit no statistically significant differences, with the NSGA-II being the best-ranked algorithm, followed by gGA, and finally by SMS-EMOA, eES, and MOEA/D. Moreover, if we consider how MOEAs behave with the TSP problem, we see that for problem instances with sizes of 100, 300, 500, 750 and 1000 (see Figures 3 and 4, Tables 6 and 7), MOEAs—especially SMS-EMOA— can perform better than SOEAs as the size of the instances increases. Figure 6 provides more statistical information. For example, note the significant difference in the behavior of SMS-EMOA between small and large instances.

**Figure 6.** *Cont*.

**Figure 6.** Boxplots showing the results for the TSP achieved by the different single-objective and multi-objective approaches at the end of 100 repetitions of the runs. Some instances were omitted because of space restrictions. However, all graphics can be found in the repository associated with this paper.
