A Weakly Pareto Compliant Quality Indicator

Abstract

In multi- and many-objective optimization problems, the optimization target is to obtain a set of non-dominated solutions close to the Pareto-optimal front, well-distributed, maximally extended and fully filled. Comparing solution sets is crucial in evaluating the performance of different optimization algorithms. The use of performance indicators is common in comparing those sets and, subsequently, optimization algorithms. Therefore, an effective performance indicator must encompass these features as a whole and, above all, it must be Pareto dominance compliant. Unfortunately, some of the known indicators often fail to properly reflect the quality of a solution set or cost a lot to compute. This paper demonstrates that the Degree of Approximation (DOA) quality indicator is a weakly Pareto compliant unary indicator that gives a good estimation of the match between the approximated front and the Pareto-optimal front.

1. Introduction

The optimized design of industrial applications is often problematic because of the simultaneous occurrence of many conflicting targets [1,2,3]. In real-world optimization problems, the decision maker needs to have a wide range of solutions to choose from [4]. Some optimization methods solve a single function obtained by aggregating different objective functions [5,6,7]. The choice of weights is the major weakness to this approach [5]. Other multi- and many-objective optimization algorithms (MOOAs) search for a non-dominated solution set [8,9,10], i.e., a set of multiple alternative solutions. This set is the Approximation Set in the decision space and the Approximated Pareto Front (APF) in the objective functions space. The main goal of such algorithms is to provide an APF matching the Pareto-optimal front (POF). The problem is to assess how well the approximated front fits the optimal one [11]. The notion of optimization algorithms performance involves evaluating the quality of the solution and the required computational effort [12]. This proves troublesome in the case of multi- and many-objective optimization problems (MOOPs): a good approach would be to use a quality indicator (QI), i.e., a function of the APF that simplifies the quantitative performance comparison of different optimization algorithms. The simplest comparison method would be to check whether one APF is better than another with respect to the Pareto dominance relations [11]. Thus, a QI must be able to account for Pareto dominance to properly compare two different algorithms. This is known as completeness with respect to Pareto dominance relations, and is the most desired property of a QI. More specifically, assuming there are two MOOAs providing, respectively, two non-dominated set A and B, a (weakly) Pareto compliant QI must comply with the following relation: if A dominates B then QI(A) must be better than QI(B). On the other hand, if there exists a scenario where A dominates B, but the QI(A) is worse than QI(B), then the QI is not Pareto compliant, therefore it could provide misleading results when it is used to compare MOOAs.

Moreover, when APFs are incomparable with respect to Pareto dominance relations, more information is needed to compare the APFs provided by different MOOAs. Generally, a good MOOA should [13,14,15]:

• minimize the APF distance from the POF;
• obtain a good (usually uniform) distribution of the solutions found;
• maximize the APF extension i.e., for each objective the non-dominated solutions should cover a wide range of values (best case: the global optimum of each objective function must be found);
• maximize the APF density, i.e., high cardinality for the approximation set is desirable.

Each goal represents a desired feature of the APF: in the following we refer to them as closeness, distribution, extension and cardinality, respectively. It is worth noticing that, given a scenario where:

• A is closer to the POF than B;
• the solutions in A are better distributed than the ones in B;
• A is more extended than B;
• the size of A is greater than the size of B,

it is very probable that A dominates B, or, at least B does not dominate A.

Although it is improbable that the two APFs will be incomparable with respect to Pareto dominance relations, when this event occurs (an example is shown in Figure 1) QI(A) should be better than QI(B). In other words, the QI should reveal that A outperforms B with respect to all features and, consequently, A is preferable to B even though the two APFs are incomparable.

A unary QI (UQI) estimates a non-dominated solution set quality by means of a real number [16]; then it is useful to estimate the effectiveness of a MOOA. Several known UQIs have no or limited completeness as regards Pareto dominance relations and are unable to take into account all the features listed previously. Few UQIs overcome these limitations despite needing much computational effort.

This paper demonstrates the ≻-completeness of the UQI, called Degree of Approximation (DOA) [17]. Moreover, it proves its ability to take into account all four features.

To our knowledge, Hypervolume [1] is the only ⊳-complete UQI (then Hypervolume is a Pareto compliant UQI), and for this reason it is considered the best UQI for comparing optimization algorithms. Nevertheless, the relation A⊲B differs from A≺B since the former accounts for the case in which A contains some solutions of B but the probability of this specific event is very low, and it can be considered null when the objective functions’ space belongs to the set of real numbers. Therefore, DOA can be used to evaluate the performance of optimization algorithms instead of the Hypervolume since it is proven that DOA is ≻-complete (then DOA is a weakly Pareto compliant UQI). Note that the calculation of Hypervolume is difficult as the number of objective functions increases, while the calculation of DOA is usually very simple and fast even in the case of many-objective optimization. DOA calculation needs the knowledge of the POF, while the Hypervolume indicator needs to know the reference point. However, it is not a limitation when they are applied to compare the performance of different MOOAs. In fact, the best way to perform the comparisons is using properly designed benchmark problems for which the POF is known [18]: Schaffer’s benchmark [19], Fonseca’s benchmark [20], Poloni’s benchmark [21], Kursawe’s benchmark [22], ZDT1-6 [13], and DTLZ1-7 [23].

In the Evolutionary Multi-Objective community, UQIs are also adopted as the guidance mechanism during the optimization process, e.g., SMS-EMOEA [24] exploits the Hypervolume indicator during optimization. The use of a given UQI in indicator-based optimization algorithms enables one to implicitly scalarize the multi-objective problem. In this case, the compliance with the dominance relation of the indicator is one reason for the slow convergence rates of elitist multi- and many-objective evolutionary algorithms (MOEAs) [25]. On the other hand, DOA is devised to evaluate the performance of the final set of non-dominated solutions and it is not devised to be included as guidance mechanism of a MOOA. Hence, in the DOA’s application field, Pareto compliance is a suitable and necessary feature.

The paper is organized as follows. Section 2 recalls the definitions and terminology typically used in multi-objective optimization related to the Pareto dominance concept. Section 3 outlines the characteristics of a QI and presents a review of the most common UQIs. Section 4 describes DOA in detail, while Section 5 and Section 6 mathematically demonstrate its ≻≻-completeness and ≻-completeness, respectively. Section 7 proves DOA compatibility with respect to the not better dominance relation. Finally, Section 8 validates DOA with some examples to highlight its accounting for closeness, distribution, extension and cardinality. Conclusions and future works are drawn in Section 9 and minor details of the proof in Section 6 have been reported in the appendices.

2. Definitions and Terminology

2.1. Multi- and Many-Objective Optimization Problem

Solving a MOOP means finding the optimal and feasible parameter configurations. A feasible solution (configuration) is called decision vector (x = x₁,x₂,...,x_m) and is a point in the decision space(X). An objective vector(y = y₁,y₂,...,y_m), that is a point in the objective space (Y), is linked to each decision vector by means of evaluating function f. So, a MOOP, with m decision variables (parameters to be set), n targets (objective functions to be optimized), and c constraints (ℓ equality and c-ℓ are inequality constraints), can be mathematically represented as follows.

Maximize or minimize:

$$y=f\left(x\right)=\left(f_1\left(x\right),f_2\left(x\right),\dots ,f_n\left(x\right)\right)$$

(1)

subject to:

$$\begin{array}{l}{g}_i\left(x\right)=0i=1,2,\dots ,\ell \\ g_i\left(x\right)\ge 0i=\ell +1,\ell +2,\dots ,c\end{array}$$

(2)

where:

$$\begin{array}{l}x=\left(x_1,x_2,\dots ,x_m\right)\in X\\ y=\left(y_1,y_2,\dots ,y_n\right)\in Y\\ y_i=f_i\left(x\right)i=1,2,\dots ,n\end{array}$$

(3)

Without loss of generality, in the following it is assumed that each objective function has to be minimized.

2.2. Pareto Dominance

Usually, in real-world MOOPs, there is no single parameter configuration that simultaneously optimizes all objective functions, i.e., a point does not exist in the decision space that is a global optimum. Thus, solving a real-world MOOP means offering the designer a set of alternative optimal solutions in the Pareto dominance sense.

Pareto dominance—A decision vector x¹dominates another decision vector x² iff:

$$f_i\left(x^1\right)\le f_i\left(x^2\right)i=1,2,\dots ,n\mathrm{and}\exists i:f_i\left(x^1\right)<f_i\left(x^2\right)$$

(4)

This relation is denoted as x¹ ≺ x². When one or more of these relations is not satisfied, x¹ does not dominate x²; this condition is denoted as x¹ ⊀ x². It is worth noticing that, for a single objective function, the standard relation less than is generally used to define the corresponding minimization problem, while the symbol ≺ represents a natural extension of < in the case of MOOPs [26].

Pareto optimality—A decision vector x’ is said to be Pareto-optimal iff:

$$\cancel{\exists }x\in X:x\prec x\prime$$

(5)

The set that groups this kind of solution is known as a Pareto-optimal set, and all the solutions of this set are alternative, with no one solution being dominated by the other solutions.

In addition to dominance, other types of relation between the solutions can be defined:

strictly dominance: a decision vector x¹ strictly dominates another decision vector x² (denoted as x¹ ≺≺ x²) iff:

$$f_i\left(x^1\right)<f_i\left(x^2\right)i=1,2,\dots ,n$$

(6)

weakly dominance: a decision vector x¹ weakly dominates the decision vector x² (denoted as x¹ ≼ x²) iff:

$$f_i\left(x^1\right)\le f_i\left(x^2\right)i=1,2,\dots ,n$$

(7)

Finally, when x¹ is better than x² with respect to a subset of objective functions but x² is better than x¹ with respect to another subset, the two solutions are said incomparable, denoted as x¹||x² (or x²||x¹):

$$\begin{gathered} x^1⋠x^2\wedge x^2⋠x^1 \\ \Rightarrow x^1\left|\right|x^2\left(or{x}^2\left|\right|x^1\right) \end{gathered}$$

(8)

Table 1. Dominance relations between two solutions [11].

Symbol	Relation	Description
x1≺≺x2	strictly dominance x1 strictly dominates x2	x1 is better than x2 with respect to each objective function
x1≺x2	dominance x1 dominates x2	x1 is not worse than x2 with respect to each objective function and x1 is better than x2 by at least one objective function
x1≼x2	weakly dominance x1 weakly dominates x2	x1 is not worse than x2 with respect to each objective function
x1||x2	incomparability x1 and x2 are incomparable	x1 and x2 do not weakly dominate each other

resumes the dominance relations. It is worth to note that a relation may imply other relations:

x¹ ≺≺ x² ⇒ x¹ ≺ x² ⇒ x¹ ≼ x²

(9)

x¹ ⋠ x² ⇒ x¹ ⊀ x² ⇒ x¹ ⊀⊀ x².

(10)

By relating the solutions of one APF A to those of another APF B it is possible to extend the dominance relations between two solutions to two APFs.

Table 2. Dominance relations between two APFs [11].

Symbol	Relation	Description
A≺≺B	A strictly dominates B	each solution belonging to B is strictly dominated by a solution belonging to A
A≺B	A dominates B	each solution belonging to B is dominated by a solution belonging to A
A⊲B	A is better than B	each solution belonging to B is weakly dominated by a solution belonging to A, and A≠B
A≼B	A weakly dominates B	each solution belonging to B is weakly dominated by a solution belonging to A
A||B	A and B are incomparable	A and B do not weakly dominate each other

shows the relations between two APFs.

3. Quality Indicator

3.1. Definitions

A quality indicator QI is a function q: S→ ℝ, where S is the objective functions space, that assigns a real value to a set of APFs belonging to S related to a MOOP. When the function q has just one argument (i.e., one APF), the quality indicator is called unary, when it has two arguments (i.e., two APFs) it is called binary, and so on.

The aim of a QI is to compare APFs and so QIs are mainly used to indicate if a MOOA works any better than others. Some QIs can also be applied as the acceptance criterion to the selection operator of the stochastic search algorithms [27], but DOA is not devised for such a scope.

3.2. Comparison Methods

This paper focuses on the use of QIs for evaluating the performances of different optimization algorithms. To do this the QI results must be interpreted by means of an interpretation function E: ℝ^q→ Bool, where q depends on the size of the QI set. Figure 2 shows some examples of interpretation functions (A and B are two APFs).

Finally, the combination of a quality indicator, I, and an interpretation function, E, is called a comparison method [11], and is referred to as C_I,E: C_I,E(A,B) = E(I(A),I(B)).

3.3. Compatibility and Completeness

Usually, one or a set of QIs can be useful to compare different optimization algorithms to figure out which works better on a particular class of problems.

Non–dominated solutions are preferred to the dominated ones from the designer’s point of view. Then, when a comparison method shows that APF A is preferable to APF B, A must be better than B. In a similar way, when A is better than B, a comparison method must indicate that A is preferable to B. Such features are known as ⊳-compatibility and ⊳-completeness [11].

Let ► be an arbitrary dominance relation among those defined in Table 2 (≻≻ or ≻ or ⊳). A comparison method C_I,E is said ►-compatible if for each possible pair of APFs A and B:

$$\begin{array}{l}{C}_{I,E}(A,B)\text{ is true}\Rightarrow \\ A◀B\end{array}$$

(11)

A comparison method C_I,E is said ►-complete if for each possible pair of APFs A and B:

$$\begin{array}{l}A◀B\\ \Rightarrow C_{I,E}(A,B)\text{ is true}.\end{array}$$

(12)

It has been demonstrated [11] that a comparison method based on a UQI (or on a finite combination of UQIs) that is both ⊳-compatible and ⊳-complete cannot exist. Moreover, Pareto dominance is sufficient but not necessary to consider an APF preferable to another: there are pairs of APFs with considerable quality difference which are considered, by Pareto dominance relations, as not comparable [16]. Hence, if a comparison method based on UQI were ⊳-compatible, the indicator could not provide any preference in the case of two incomparable APFs. Therefore, it would be better if the UQI were only compatible with ⋫ [28] and it should take into account all the features (closeness, distribution, extension, cardinality) that are desirable for an APF.

Finally, while a comparison method ⊳-complete is necessary (i.e., when APF A is better than APF B the comparison method must highlight it), when a comparison method shows that A is preferable to B, one of the following two cases must hold:

• A is better than B (A⊲B);
• A and B are incomparable and A outperforms B with respect to closeness, distribution, extension and cardinality.

3.4. Closeness, Distribution, Extension, and Cardinality

The main target of an optimization algorithm to solve a MOOP is to find an APF as similar as possible to the POF. Hence, as said before, the APF must be:

• close to the POF; Figure 3 represents the extreme cases: an APF exhibiting good closeness only, and an APF with all good features but not close to the POF;
• well distributed (usually uniform); Figure 4 shows an APF exhibiting a uniform distribution only and an APF with all good features but not uniformly distributed;
• very extended (in the best case the global optimum of each objective function belongs to the APF); Figure 5 shows an APF with only a good extension and one with all good features but not extended;
• of high cardinality; Figure 6 shows an APF with good cardinality only and an APF with all good features but poor cardinality.

Figure 7 shows an APF with all the desired features. A good QI must take into account all these features to give a correct measure of APF quality.

Table 3. Summary of selected UQIs and features that influence their value.

Indicator	Closeness	Distribution	Extension	Cardinality
Average Distance from Reference Set [30]	⚫	⚫	⚫	⚫
Chi-Square-Like Deviation Measure [33]		⚫
Completeness Indicator [31,32]	⚫	⚫	⚫	⚫
Enclosing Hypercube [11]			⚫
Generational Distance [34]	⚫
Hypervolume [1]	⚫	⚫	⚫	⚫
Inverted Generational Distance [29]	⚫	⚫	⚫	⚫
M1* [13]	⚫
M2* [13]		⚫
M3* [13]			⚫
Maximum Pareto Front Error [34]	⚫
Outer Diameter [26]			⚫
Overall Nondominated Vector Generation [34]				⚫
Overall Pareto Spread [35]			⚫
Potential Function [27]	⚫	⚫	⚫	⚫
Seven Points Average Distance [36]	⚫		⚫
Spacing [37]		⚫
Unary ε-Indicator [11,26]	⚫
Uniform Distribution [38]		⚫
Worst Distance from Reference Set [30]	⚫
Δ [8]		⚫	⚫
Δp [39,40]	⚫	⚫	⚫	⚫

points out if a specific feature partially or totally ⚫ affects the value of some UQIs. A heuristic approach has been applied to determine whether a feature (closeness, distribution, extension, cardinality) affects the QI value. In particular, an APF B obtained by improving a given feature of another APF A is expected to have an indicator value better than that of A when the indicator is sensitive to this feature. For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature. An indicator is partially affected by a feature when it sometimes improves and other times does not change.

The Inverted Generational Distance indicator (IGD) [29] and a similar indicator using a weighted-scalar distance function called Average Distance from Reference Set indicator (ADRS) [30], the Completeness indicator [31,32], the Potential Function indicator [27] and the Hypervolume indicator [1] account for all the features but they present some drawbacks.

The IGD and ADRS indicators have the same complexity of DOA, but they are ≻≻-complete only [11]. The Δ_p indicator [39] can be considered as the combination of slight variations of the Generational Distance and the IGD indicator [25]. Therefore, it presents the advantage and limitations of these indicators, that is it accounts for all the features but it is ≻≻-complete only.

The Completeness and Potential Function indicators are as ≻-complete as the DOA indicator. Nevertheless, the Completeness indicator cannot be directly computed, but can be estimated by drawing samples from the feasible set and computing completeness for these samples. The confidence interval for the true value can be evaluated with any reliability value, given sufficiently large samples [26]. For the Potential Function indicator similar considerations hold. Hence, the drawback of both indicators is the high computational cost.

To our knowledge, Hypervolume is the only ⊳-complete UQI, and so is considered the best UQI for comparing optimization algorithms. Nevertheless, the relation A⊲B differs from A≺B since the former accounts for the case in which B contains some solutions of A but the probability of this specific event is very low, and it can be considered null when the objective functions’ space belongs to the set of real numbers.

Moreover, Hypervolume running time grows exponentially with the number of objective functions [41,42,43,44]. The most obvious method for calculating Hypervolume is the inclusion-exclusion algorithm, with complexity O(n2^m), where n is the number of objectives and m is the number of APF points. The fastest methods for calculating Hypervolume (e.g., LebMeasure [45], Hypervolume by Slicing Objectives algorithm [46]) lead to a O(m²n³)complexity. The DOA indicator has a lower computational cost, presenting a O(nMm) complexity, where M is the number of POF points.

4. The Weakly Pareto Compliant Quality Indicator

The comparison method based on DOA and its associated interpretation function is ≻-complete. In detail, for a given POF and an APF A, DOA is computed as follows.

First, given a solution i belonging to the POF, D_i,A, is determined from the sub-set of A containing the solutions dominated by i (Figure 8). Hence, if the number of solutions belonging to D_i,A is not null, for each approximated solution a∊D_i,A the Euclidean distance df_i,a between a and i is computed as:

$$d{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^n{\left[f_{k,a}-f_{k,i}\right]}^2}},$$

(13)

with:

n 

number of objective functions,

f_k,a

value of k-th objective function of the approximated solution a,

f_k,i

value of k-th objective function of optimal solution i.

Euclidean distance d_i,A (Figure 9) between i and the nearest approximated solution belonging to D_i,A is computed in the objective function space as:

$$d_{i,A}=\{\begin{array}{ll}\min \left(d{f}_{i,a}\right)a\in D_{i,A}& if\left|D_{i,A}\right|>0\\ \infty & if\left|D_{i,A}\right|=0\end{array},$$

(14)

where |D_i,A| is the number of solutions belonging to D_i,A.

Another quantity r_i,A (similarly to d_i,A) is computed for i considering the solutions of A not dominated by i (i.e., A\D_i,A):

$$r_{i,A}=\{\begin{array}{ll}\min \left(r{f}_{i,a}\right)a\in A\backslash D_{i,A}& if\left|A\backslash D_{i,A}\right|>0\\ \infty & if\left|A\backslash D_{i,A}\right|=0\end{array},$$

(15)

where rf_i,a is a reduced distance (Figure 10) between i and a non-dominated solution a of A (i.e., ∀a ∊ A: i || a), i.e., computed only for objectives k with f_k,a ≥ f_k,i:

$$r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^n{\left[\max \left(0,f_{k,a}-f_{k,i}\right)\right]}^2}}.$$

(16)

Note that, rf_i,a is equal to df_i,a when a $\in$ D_i,A. Moreover, defining n_a (n_a < n) as the number of functions for which the f_k,a − f_k,i ≥ 0 (f_k,a ≥ f_k,i, k = 1,..,n_a and f_k,a < f_k,i, k = n_a + 1,..,n) expression (16) can be rewritten as:

$$r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_a\left(k\right)-f_i\left(k\right)\right]}^2+{\displaystyle \sum _{k=n_a+1}^{n}0}}}.$$

(17)

Finally, defining

s_i,A = min(d_i,A,r_i,A)

(18)

the DOA indicator for the APF A is computed as:

$$DOA(A)=\frac{1}{\left|POF\right|}{\displaystyle \sum _{i=1}^{\left|POF\right|}{s}_{i,A}}.$$

(19)

DOA and IGD present some similarities. More specifically, if d_i,A, Equation (14), is computed considering all solutions in A regardless of the dominance relation with i, and only this quantity is considered then the IGD indicator is obtained, that is:

$$IGD(A)=\frac{1}{\left|POF\right|}{\displaystyle \sum _{i=1}^{\left|POF\right|}{d}_{i,A}}.$$

(20)

The IGD indicator is not ≻-complete while DOA is ≻-complete, as will be proved in the following. Therefore, the use of r_i,A is the enabling key for DOA ≻-completeness.

While the Hypervolume indicator needs to know the reference point, the DOA calculation needs the knowledge of the POF, like the IGD indicator. This is not a drawback in MOOA benchmarking which is usually carried out for problems with known POF. More specifically, the Pareto optimal solutions and the analytical expression of the objective functions are usually known. Therefore, the POF is obtained by points uniformly discretizing the decision parameter intervals where the Pareto optimal solutions are located. The value of DOA does not depend on the number of points in the POF if |POF|>>|APF|.

Considering two APFs A and B, the proposed quality indicator needs interpreting [11] to affirm either that A is preferable to B or B is preferable to A or A and B are equivalent: the proposed obvious interpretation function is illustrated in Figure 11. Moreover, DOA changes with arbitrary scaling of the objective functions, since the DOA indicator is a distance-based metric, while the relationship between DOA(A) and DOA(B) does not change.

In the following, it is demonstrated that A≺B implies A is preferable to B, that is DOA(A) < DOA(B), in order to affirm that DOA is a ≻-complete quality indicator.

For the sake of clarity, the ≻≻-completeness of DOA is demonstrated before proving its ≻-completeness.

5. ≻≻-Completeness

DOA is a ≻≻-complete quality indicator if DOA(A)<DOA(B) for any pair of APF A and B, with A≺≺B.

In the hypothesis that A≺≺B, each solution of B is strictly dominated by, at least, one solution of A. To demonstrate that DOA(A)<DOA(B) is sufficient to prove that s_i,A is always lesser than s_i,B for each point i ∊ POF. In other words,

• if

  s_i,A < s_i,B ∀ i ∊ POF,

  (21)
• then

  $$DOA(A)=\frac{1}{\left|POF\right|}{\displaystyle \sum _{i=1}^{\left|POF\right|}{s}_{i,A}}<\frac{1}{\left|POF\right|}{\displaystyle \sum _{i=1}^{\left|POF\right|}{s}_{i,B}}=DOA(B).$$

  (22)

Considering a point i ∊POF, in the following, b indicates the solution belonging to B which provides s_i,B and a a solution of A that strictly dominates b (a≺≺b); only four scenarios are possible (see Figure 12):

• A1. i≺≺b Λ i≼a
• B1. i≺≺b Λ i||a
• C1. i≺b Λ i||a
• D1. i||b Λ i||a

Note that the other scenarios i≼b Λ i≼a and i||b Λ i≼a are not possible because a≺≺b: in fact, from either i≼b Λ a≺≺b and i||b Λ a≺≺b follows i⋠a.

Moreover, it is worth mentioning the following:

Remark 1.

i≼a implies that s_i,A ≤ df_i,a, in detail:

• s_i,A = df_i,a iff s_i,A = d_i,A Λ d_i,A = df_i,a;
• s_i,A < df_i,a either if s_i,A = d_i,A Λ d_i,A = df_i,a* < df_i,a (where a*∊A and a*≠a) or if s_i,A = r_i,A (this implies that r_i,A < d_i,A ≤ df_i,a).

Remark 2.

i||a implies that s_i,A ≤ rf_i,a, in detail:

• s_i,A = rf_i,a iff s_i,A = r_i,A Λ r_i,A = rf_i,a;
• s_i,A < rf_i,a either if s_i,A = r_i,A Λ r_i,A = rf_i,a* < rf_i,a (where a*∊A and a*≠a) or if s_i,A = d_i,A (this implies that d_i,A < r_i,A ≤ rf_i,a).

Finally, the inequality s_i,A < s_i,B will be proved for the four scenarios A1–D1: this inequality naturally implies the ≻≻-completeness of the DOA indicator.

5.1. A1. i≺≺b Λ i≼a

In this case, i strictly dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i≼a implies that s_i,A ≤ df_i,a (see Remark 1).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that df_i,a < df_i,b.

Recalling that

i≼a ⇒

f_k,i ≤ f_k,a, ∀ k = 1,…, n

a≺≺b ⇒

f_k,a < f_k,b, ∀ k = 1,…, n

the following inequalities hold:

$$\begin{array}{l}0\le f_{k,a}-f_{k,i}<f_{k,b}-f_{k,i}\forall k=1,..,n\\ \Rightarrow \\ d{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^n{\left[f_{k,a}-f_{k,i}\right]}^2}}<\sqrt{{\displaystyle \sum _{k=1}^n{\left[f_{k,b}-f_{k,i}\right]}^2}}=d{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le d{f}_{i,a}<d{f}_{i,b}=s_{i,B}\end{array}.$$

(23)

□

5.2. B1. i≺≺b Λ i||a

In this case, i strictly dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i||a implies that s_i,A ≤ rf_i,a (see Remark 2).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that rf_i,a < df_i,b. Proof is given in the next section because scenario C encompasses scenario B.

5.3. C1. i≺b Λ i||a

In this case, i dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i || a implies that s_i,A ≤ rf_i,a (see Remark 2).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that rf_i,a < df_i,b. Ordering the n objective functions of solution a in such a way that the first n_a (with n_a < n) are greater than those of i and recalling that:

i||a  ⇒

f_k,i < f_k,a, ∀ k = 1,…, n_a

f_k,i ≥ f_k,a, ∀ k = n_a + 1,…, n

i≺b ⇒

f_k,i ≤ f_k,b, ∀ k = 1,…, n

a≺≺b ⇒

f_k,a < f_k,b, ∀ k = 1,…, n

then the following inequalities hold:

$$\begin{array}{l}0<f_{k,a}-f_{k,i}<f_{k,b}-f_{k,i}\forall k=1,..,n_a\\ 0\le f_{k,b}-f_{k,i}\forall k=n_a+1,..,n\\ \Rightarrow \\ r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,a}-f_{k,i}\right]}^2+{\displaystyle \sum _{k=n_a+1}^{n}0}}}<\\ \sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,b}-f_{k,i}\right]}^2}+{\displaystyle \sum _{k=n_a+1}^n{\left[f_{k,b}-f_{k,i}\right]}^2}}=d{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le r{f}_{i,a}<d{f}_{i,b}=s_{i,B}.\end{array}$$

(24)

□

5.4. D1. i||b Λ i||a

In this case, i and b are incomparable then s_i,B = r_i,B = rf_i,b, because b is the solution which provides s_i,B. Moreover, i||a implies that s_i,A ≤ rf_i,a (see Remark 2).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that rf_i,a < rf_i,b.

Ordering the n objective functions of solution a in such a way that the first n_a are greater than those of i, ordering the n objective functions of solution b in such a way that the first n_b are greater than those of i (with n_a ≤ n_b < n, since a≻≻b Λ i||a implies n_a ≤ n_b, while i||b implies n_b < n) and recalling that:

i||a ⇒

f_k,i < f_k,a, ∀ k = 1,…, n_a

f_k,i ≥ f_k,a, ∀ k = n_a + 1,…, n

i||b ⇒

f_k,i < f_k,b, ∀ k = 1,…, n_b

f_k,i ≥ f_k,b, ∀ k = n_b + 1,…, n

a≺≺b  ⇒

f_k,a < f_k,b, ∀ k = 1,..., n

then the following inequalities hold:

$$\begin{array}{l}\begin{array}{ll}0<f_{k,a}-f_{k,i}<f_{k,b}-f_{k,i}& \forall k=1,..,n_a\\ 0<f_{k,b}-f_{k,i}& \forall k=n_a+1,..,n_b\\ 0\ge f_{k,b}-f_{k,i}& \forall k=n_b+1,..,n\end{array}\\ \Rightarrow \\ r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,a}-f_{k,i}\right]}^2+{\displaystyle \sum _{k=n_a+1}^{n_b}0}+{\displaystyle \sum _{k=n_b+1}^{n}0}}}<\\ \sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,b}-f_{k,i}\right]}^2}+{\displaystyle \sum _{k=n_a+1}^{n_b}{\left[f_{k,b}-f_{k,i}\right]}^2+{\displaystyle \sum _{k=n_b+1}^{n}0}}}=r{f}_{i,b}\\ \Rightarrow s_{i,A}\le r{f}_{i,a}<r{f}_{i,b}=s_{i,B}\end{array}.$$

(25)

□

6. ≻-Completeness

In this section is proved that DOA is a ≻-complete quality indicator. Consider any pair of APF A and B, with A≺B, the ≻-completeness of DOAis demonstrated by proving that s_i,A is never greater than s_i,B (for each point i∊POF) and always exists a point i*∊ POF for which s_i*,A is lesser than s_i*,B:

• if

  s_i,A ≤ s_i,B ∀ i ∊ POF Λ ∃ i*∊POF: s_i*,A < s_i*,B

  (26)
• then

  $$\mathrm{DOA}(\mathrm{A})=\frac{1}{\left|POF\right|}\left(s_{i*,A}+{\displaystyle \sum _{i=2}^{\left|POF\right|}{s}_{i,A}}\right)<\frac{1}{\left|POF\right|}\left(s_{i*,B}+{\displaystyle \sum _{i=2}^{\left|POF\right|}{s}_{i,B}}\right)=\mathrm{DOA}(\mathrm{B}).$$

  (27)

In the following, b indicates the solution belonging to B which provides s_i,B for a point i ∊ POF and a a solution of A that dominates b (a≺b). Moreover, the ≻-completeness of DOA is proved in the worst and most general case, i.e., when ∀ b∊B ∄ a∊A: a≺≺b (i.e., a≺b Λ a⊀⊀b, limit case); only five scenarios are possible (see Figure 13):

• A2. i≺≺b Λ i≺a
• B2. i≺≺b Λ i||a
• C2. i≺b Λ i≼a
• D2. i≺b Λ i||a
• E2. i||b Λ i||a

Note that the scenario i||b Λ i≼a is not possible because a≺b: in fact, from i||b Λ a≺b follows i⋠a. Moreover, scenario A2 does not include i = a, differently from section 5, because in this case i≺≺b while a≺b Λ a⊀⊀b. Analogously, differently from section 5, when i≺b the scenario i≼a is possible. Finally, the scenario i ≼ b is not considered because A≺B, in fact i = b would lead to the absurd a≺b=i.

In order to demonstrate the ≻-completeness of DOA, it is proved that the inequality s_i,A < s_i,B is verified ∀i for the three scenarios A2, B2 and C2. While for the remaining two scenarios D2 and E2 we will prove that the following two sufficient conditions hold:

α.

s_i,A ≤ s_i,B

β.

∃ i* ∊ POF: s_i*,A < s_i*,B.

For the sake of simplicity, the proof of β will be given in Appendix A.

6.1. A2. i≺≺b Λ i≺a

In this case, i strictly dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i≺a implies that s_i,A ≤ df_i,a (see Remark 1).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that df_i,a < df_i,b.

Recalling that:

i≺a ⇒

f_k,i ≤ f_k,a, ∀ k = 1,…, n

a≺b ⇒

f_k,a ≤ f_k,b, ∀ k = 1,…, n Λ ∃ j: f_j,a < f_j,b

the following inequalities hold:

$$\begin{array}{l}0\le f_{k,a}-f_{k,i}\le f_{k,b}-f_{k,i}\forall k=1,..,n\\ 0\le f_{k,a}-f_{k,i}<f_{k,b}-f_{k,i}k=j\\ \Rightarrow \\ d{f}_{i,a}=\sqrt{{\displaystyle \sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^n{\left[f_{k,a}-f_{k,i}\right]}^2+{\left[f_{j,a}-f_{j,i}\right]}^2}}<\\ \sqrt{{\displaystyle \sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^n{\left[f_{k,b}-f_{k,i}\right]}^2+{\left[f_{j,b}-f_{j,i}\right]}^2}}=d{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le d{f}_{i,a}<d{f}_{i,b}=s_{i,B}.\end{array}$$

(28)

□

6.2. B2. i≺≺b Λ i||a

In this case, i strictly dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i||a implies that s_i,A ≤ rf_i,a (see Remark 2).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that rf_i,a < df_i,b.

Ordering the n objectives f of solution a in such a way that the first n_a (with n_a < n) are greater than those of i and recalling that:

i||a ⇒

f_k,i < f_k,a, ∀ k = 1,…, n_a

f_k,i ≥ f_k,a, ∀ k = n_a + 1,…, n

i≺≺b  ⇒

f_k,i < f_k,b, ∀ k = 1,..., n

a≺b ⇒

f_k,a ≤ f_k,b, ∀ k = 1,…, n

then the following inequalities hold:

$$\begin{array}{l}0<f_{k,a}-f_{k,i}\le f_{k,b}-f_{k,i}\forall k=1,..,n_a\\ 0<f_{k,b}-f_{k,i}\forall k=n_a+1,..,n\\ \Rightarrow \\ r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,a}-f_{k,i}\right]}^2+{\displaystyle \sum _{k=n_a+1}^{n}0}}}<\\ \sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,b}-f_{k,i}\right]}^2}+{\displaystyle \sum _{k=n_a+1}^n{\left[f_{k,b}-f_{k,i}\right]}^2}}=d{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le r{f}_{i,a}<d{f}_{i,b}=s_{i,B}.\end{array}$$

(29)

□

6.3. C2. i≺b Λ i≼a

In this case, i dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i≼a implies that s_i,A ≤ df_i,a (see Remark 1).

So, in order to demonstrate that s_i,A < s_i,B it is sufficient to demonstrate that df_i,a < df_i,b.

Recalling that:

i≼a ⇒

f_k,i ≤ f_k,a, ∀ k = 1,…, n

a≺b ⇒

f_k,a ≤ f_k,b, ∀ k = 1,…, n Λ ∃ j: f_j,a < f_j,b

then the following inequalities hold:

$$\begin{array}{l}0\le f_{k,a}-f_{k,i}\le f_{k,b}-f_{k,i}\forall k=1,..,n\\ 0\le f_{k,a}-f_{k,i}<f_{k,b}-f_{k,i}\forall k=j\\ \Rightarrow \\ d{f}_{i,a}=\sqrt{{\displaystyle \sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^n{\left[f_{k,a}-f_{k,i}\right]}^2+{\left[f_{j,a}-f_{j,i}\right]}^2}}<\\ \sqrt{{\displaystyle \sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^n{\left[f_{k,b}-f_{k,i}\right]}^2+{\left[f_{j,b}-f_{j,i}\right]}^2}}=d{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le d{f}_{i,a}<d{f}_{i,b}=s_{i,B}\end{array}.$$

(30)

□

6.4. D2. i≺b Λ i||a

In this case, i dominates b then s_i,B = d_i,B = df_i,b, because b is the solution which provides s_i,B. Moreover, i||a implies that s_i,A ≤ rf_i,a (see Remark 2).

So, in order to demonstrate that s_i,A ≤ s_i,B it is sufficient to demonstrate that rf_i,a ≤ df_i,b.

Ordering the n objectives f of solution a in such a way that the first n_a (with n_a < n) are greater than those of i and recalling that:

i||a ⇒

f_k,i < f_k,a, ∀ k = 1,…, n_a

f_k,i ≥ f_k,a, ∀ k = n_a + 1,…, n

i≺b ⇒

f_k,i ≤ f_k,b, ∀ k = 1,…, n Λ ∃ h: f_h,i < f_h,b

a≺b ⇒

f_k,a ≤ f_k,b, ∀ k = 1,…, n Λ ∃ j: f_j,a < f_j,b

then the following inequalities hold

$$\begin{array}{l}0<f_{k,a}-f_{k,i}\le f_{k,b}-f_{k,i}\forall k=1,..,n_a\\ 0\le f_{k,b}-f_{k,i}\forall k=n_a+1,..,n\\ \Rightarrow \\ r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,a}-f_{k,i}\right]}^2}+{\displaystyle \sum _{k=n_a+1}^{n}0}}\le \\ \sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,b}-f_{k,i}\right]}^2}+{\displaystyle \sum _{k=n_a+1}^n{\left[f_{k,b}-f_{k,i}\right]}^2}}=d{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le r{f}_{i,a}\le d{f}_{i,b}=s_{i,B}.\end{array}$$

(31)

□

Note that rf_i,a is strictly lesser than df_i,b when 1 ≤ j ≤ n_a, since

$${\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,a}-f_{k,i}\right]}^2}<{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,b}-f_{k,i}\right]}^2}.$$

(32)

Otherwise s_i,A ≤ s_i,B. In particular, rf_i,a = df_i,b iff f_k,a = f_k,b ∀ k = 1,…,n_a Λ f_k,i = f_k,b ∀ k = n_a + 1,…,n. In this case, obviously, f_h,i < f_h,a = f_h,b with 1 ≤ h ≤ n_a and f_j,a < f_j,i = f_j,b with j > n_a. As said before, the proof that ∃ i*∊POF: s_i*,A < s_i*,B has been reported in Appendix A.

6.5. E2. i||b Λ i||a

In this case, i and b are incomparable then s_i,B = r_i,B = rf_i,b, because b is the solution which provides s_i,B. Moreover, i||a implies that s_i,A ≤ rf_i,a (see Remark 2).

So, in order to demonstrate that s_i,A ≤ s_i,B, it is sufficient to demonstrate that rf_i,a ≤ rf_i,b. Ordering the n objectives f of solution a in such a way that the first n_a are greater than those of i, ordering the n objectives f of solution b in such a way that the first n_b are greater than those of i (with n_a ≤ n_b < n, since a≺b Λ i||a implies n_a ≤ n_b, while i||b implies n_b < n) and recalling that:

i||a ⇒

f_k,i < f_k,a, ∀ k = 1,…, n_a

f_k,i ≥ f_k,a, ∀ k = n_a + 1,…, n

i||b ⇒

f_k,i < f_k,b, ∀ k = 1,…, n_b

f_k,i ≥ f_k,b, ∀ k = n_b + 1,..,n

a≺b ⇒

f_k,a ≤ f_k,b, ∀ k = 1,…, n Λ ∃ j: f_j,a < f_j,b

then the following inequalities hold:

$$\begin{array}{l}\begin{array}{ll}0<f_{k,a}-f_{k,i}\le f_{k,b}-f_{k,i}& \forall k=1,..,n_a\\ 0<f_{k,b}-f_{k,i}& \forall k=n_a+1,..,n_b\\ 0\ge f_{k,b}-f_{k,i}& \forall k=n_b+1,..,n\end{array}\\ \Rightarrow \\ r{f}_{i,a}=\sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,a}-f_{k,i}\right]}^2+{\displaystyle \sum _{k=n_a+1}^{n_b}0}+{\displaystyle \sum _{k=n_b+1}^{n}0}}}\le \\ \sqrt{{\displaystyle \sum _{k=1}^{n_a}{\left[f_{k,b}-f_{k,i}\right]}^2}+{\displaystyle \sum _{k=n_a+1}^{n_b}{\left[f_{k,b}-f_{k,i}\right]}^2+{\displaystyle \sum _{k=n_b+1}^{n}0}}}=r{f}_{i,b}\\ \Rightarrow \\ s_{i,A}\le r{f}_{i,a}\le r{f}_{i,b}=s_{i,B}.\end{array}$$

(33)

□

It is worth noting that:

• rf_i,a < rf_i,b if n_b ≠ n_a;
• rf_i,a ≤ rf_i,b if n_b = n_a.

In the last case, rf_i,a = rf_i,b iff f_k,a = f_k,b ∀ k = 1,..,n_a. In this case, obviously, f_j,a < f_j,b ≤ f_j,i whit j > n_a. As said before, the proof that ∃ i*∊POF: s_i*,A < s_i*,B has been reported in Appendix A.

7. ⋫-Compatibility

In this section it is proved that DOA is a ⋫-compatible quality indicator. The following remarks are necessary.

Remark 3.

A solution a’ belonging to A does not influence DOA(A) if, ∀ i∊POF:

• s_i,A < df_i,a’ when a’ is dominated by i;
• s_i,A < rf_i,a’ when a’ is not dominated by i.

In fact, in this case s_i,A ≠ df_i,a’ and s_i,A ≠ rf_i,a’, by which follows that DOA(A) does not change its value if a’ is moved off from A. These means that an APF B = A\{a’} has the same quality indicator value, i.e., DOA(A) = DOA(B). Using DOA seems B equivalent to A while A, having one more solution, is better than B [11]. Then the proposed method is not a ⊳-complete quality indicator. On the other hand, DOA together with its interpretation function is a ≻-complete comparison method, as it is demonstrated before.

Remark 4.

The relation A⊲B is equivalent to assume that B = C ⋃ D, where C ⊊ A, D ⋂ A = Ø and ∀ b∊D ∃ a∊A: a≺b. Note that, when C = Ø then A≺B. Hence, the difference between case A≺B and A⊲B is that in the latter B could contains some solutions of A. Moreover, obviously, only the last case includes B ⊊ A (i.e., D = Ø). For these reasons when A⊲B, DOA(A) is never greater than DOA(B).

Remark 5.

The relation A||B involves that DOA(A) can be greater or less than or equal to DOA(B).

For two generic APF A and B, the ≻-completeness of DOA, together with Remarks 3–5, imply that if DOA(A) < DOA(B) then A≺B or A⊲B or A||B, i.e., it is sure that B⋪A. Then DOA is a ⋫-compatible unary quality indicator.

8. DOA Validation

To demonstrate that DOA takes into account all features (closeness, distribution, extension, and cardinality) three typical POFs have been considered:

• convex and connected;
• non-convex and connected;
• convex and disconnected.

In particular, the DOA unary quality indicator has been computed for different examples of APFs. Both POFs and APFs are drawn in Figure 14, Figure 15 and Figure 16.

Table 4. DOA Evaluation for some typical POF (symbol in brackets is the marker associated to the APF in Figure 14, Figure 15 and Figure 16).

POF	APF	Closeness	Distribution	Extension	Cardinality	DOA
Convex and connected (see Figure 14)	APF1(◊)	poor	poor	poor	poor	0.71040
	APF2(+)	good	poor	poor	poor	0.16940
	APF3(○)	good	good	poor	poor	0.16287
	APF4(□)	good	good	good	poor	0.10253
	APF5(●)	good	good	good	good	0.06431
Non-convex and connected (see Figure 15)	APF1(◊)	poor	poor	poor	poor	0.79167
	APF2(+)	good	poor	poor	poor	0.24303
	APF3(○)	good	good	poor	poor	0.23465
	APF4(□)	good	good	good	poor	0.09301
	APF5(●)	good	good	good	good	0.06993
Convex and disconnected (see Figure 16)	APF1(◊)	poor	poor	poor	poor	0.69510
	APF2(+)	good	poor	poor	poor	0.16866
	APF3(○)	good	good	poor	poor	0.16810
	APF4(□)	good	good	good	poor	0.07254
	APF5(●)	good	good	good	good	0.06331

shows the values of DOA for the APFs considered.

Let us examine the strategy adopted to choose the examples of APF for each POF: the APF B obtained by improving a given feature of APF A is expected to have a better indicator value than that of A if the indicator is sensitive to this feature. For example, if an APF is gradually moved towards the POF and the indicator increasingly improves, then the indicator is influenced by the closeness feature. The validation was performed according to these considerations. Hence, from an APF (APF1, indicated in the figures by symbol◊) with only poor features, the second APF (APF2, indicated by +) is created by improving APF1 closeness that is by converging APF1 on the POF. Therefore, APF2 has better closeness than APF1 but the same distribution, extension and cardinality. The third APF (APF3, indicated by ○) is obtained by improving the distribution of APF2 by uniformly distributing the solutions of APF2. APF3 has better distribution than APF2 but the same closeness, extension and cardinality. The fourth APF (APF4, indicated by □) is created by improving the extension of APF3, preserving the other features. Finally, the fifth APF (APF5, indicated by ●) is created by adding new points to APF4, and hence improving the cardinality of APF5 with respect to the fourth APF. It is worth noting that it does not matter in which order the different features are added to the initial APF1, and the order used in Table 4 simply follows the features’ description given in the Introduction.

Whatever the characteristics of the POF, such a strategy highlights that the value of DOA decreases when one feature improves.

The results in Table 4 highlight the ≻-completeness of DOA too. In fact, the first APF is dominated by the others and it has a DOA value worse than those of the other APFs.

9. Conclusions and Future Work

Evaluating the performance of MOOAs is very difficult because their comparison involves comparing APFs. QIs are used to measure the goodness of the APF provided by different optimization algorithms to highlight which works better.

Therefore, a QI must be able to account for Pareto dominance to properly compare two different algorithms. Moreover, when APFs are incomparable, further data (closeness, distribution, extension and cardinality) must be taken into account to compare the APFs provided by different MOOAs. Few UQIs are ≻-complete and able to account for the aforementioned features, yet they require great computational effort.

This paper has described the DOA unary quality indicator, which could be very useful in assessing the performance of an MOOA by estimating the match between the approximation front found by the MOOA and the optimal one. It has been proven that it is ≻-complete, ⋫-compatible, and requires little computational cost. Moreover, a numerical validation was carried out to demonstrate that it accounts for closeness, distribution, extension, and cardinality. An implementation of the DOA indicator is available online: http://wwwelfin.diees.unict.it/esg/DOA.html.

The major drawback of the proposed indicator is that it is not ⊳-complete. Even though the dominance relation A ⊲ B differs from A ≺ B since the former accounts for the case in which B contains some solutions of A, the probability of this specific event is next to nil. Nevertheless, a future development of this work is to properly modify DOA to make it ⊳-complete, maintaining low computational complexity. In this perspective, the underlying reasons leading from a not ≻-complete indicator, IGD, to the similar one proposed in this paper, DOA, having ≻-completeness can be the enabling keys.

Moreover, an integral formulation of DOA can be developed to deal with continuous POFs that can be expressed by an analytical function.

Finally, innovative studies have proposed interesting strategies to use UQIs requiring the knowledge of the POF as the guidance mechanism for indicator-based MOEAs. Therefore, the use of DOA for indicator-based evolutionary algorithms is another attractive development of the work.