Novel Features and Neighborhood Complexity Measures for Multiclass Classification of Hybrid Data

Abstract

The present capabilities for collecting and storing all kinds of data exceed the collective ability to analyze, summarize, and extract knowledge from this data. Knowledge management aims to automatically organize a systematic process of learning. Most meta-learning strategies are based on determining data characteristics, usually by computing data complexity measures. Such measures describe data characteristics related to size, shape, density, and other factors. However, most of the data complexity measures in the literature assume the classification problem is binary (just two decision classes), and that the data is numeric and has no missing values. The main contribution of this paper is that we extend four data complexity measures to overcome these drawbacks for characterizing multiclass, hybrid, and incomplete supervised data. We change the formulation of Feature-based measures by maintaining the essence of the original measures, and we use a maximum similarity graph-based approach for designing Neighborhood measures. We also use ordering weighting average operators to avoid biases in the proposed measures. We included the proposed measures in the EPIC software for computational availability, and we computed the measures for publicly available multiclass hybrid and incomplete datasets. In addition, the performance of the proposed measures was analyzed, and we can confirm that they solve some of the biases of previous ones and are capable of natively handling mixed, incomplete, and multiclass data without any preprocessing needed.

1. Introduction

Several disciplines, such as Pattern Recognition, Machine Learning, Computational Intelligence, and Artificial Intelligence share an interest in automatic knowledge management strategies [1]. The latter is becoming an active research area useful for several daily-life aspects such as environmental concerns (i.e., clothing sustainability [2], fault detection in the subsea [3], and pollution analysis [4]), education (teacher training [5] and other educational applications [6]), and health (disease prediction [7], secure Internet of Things in healthcare [8] and other healthcare challenges [9]), among others.

There are numerous supervised classification algorithms, such as Neighborhood-based classifiers [10], Decision trees [11], Neural networks [12], Support vector machines [13], Associative classifiers [14], and Logical-combinatorial classifiers [15]. However, because of the No Free Lunch theorems [16], no algorithm will outperform all others for all problems and performance measures. That is why meta-learning, as a knowledge management technique, is the focus of several research efforts [17,18,19].

Most meta-learning strategies are based on determining data characteristics, usually by computing data complexity measures [20,21]. Such measures aim at describing data characteristics [22,23,24] related to size, shape, density, and others. However, the majority of the data complexity measures in the literature assume the classification problem is binary (just two decision classes), and that the data is numeric and has no missing values.

Unfortunately, in many real-life applications, such assumptions are not fulfilled. Real data is often hybrid (described by both numeric and categorical attributes) and can present an absence of information. In addition, several problems have multiple possible outcomes or decisions to make; they have multiple decision classes.

As stated before, data complexity measures [20] are usually defined for numeric, complete, and binary classification problems. Therefore, our aim is to extend the measures for characterizing multiclass, hybrid, and incomplete supervised data. The main contributions of this paper are the following:

• We extend four data complexity measures for the multiclass classification scenario and for dealing with hybrid and incomplete data.
• We include the proposed four measures in the EPIC software [25,26] for computational availability.
• We compute the proposed measures for publicly available multiclass hybrid and incomplete datasets.

This paper is organized as follows: Section 2 reviews some of the related works on data complexity measures. Section 3 introduces the extended data complexity measures, and Section 4 shows some of the properties of the proposed measures, as well as their computation over publicly available datasets. Finally, we present some conclusions and future works.

2. Related Works

This section reviews the existing data complexity measures of the Feature-based, Linearity, and Neighborhood taxonomies for dealing with single-label supervised classification. Section 2.1 offers the preliminary concepts. Section 2.2 details Feature-based complexity measures. Section 2.3 explains Linearity measures, and Section 2.4 covers Neighborhood measures.

2.1. Preliminaries

Let U be a universe of instances, described by a set of features $A=\left\{A_1,\cdots ,A_m\right\}$. Each attribute $A_i$ has a definition domain, which can be Boolean, numeric, or categorical, and can have missing values (denoted by ?). Let $x\in U$ be an instance. Its value in the i-th feature is denoted by $x\left[i\right]$. Let us have a decision attribute or label and a set of decision classes $d=\left\{d_1,\cdots ,d_c\right\}$. The true label value of the instance $x\in U$ is denoted by $class\left(x\right)$. If the decision class $d$ has two values, we have a binary single-label classification. On the other hand, if $d$ has more than two values, we have a multi-class single-label classification problem (Figure 1).

Figure 1. Different scenarios in supervised classification while considering a decision label.

Since 2002, there has been an interest in assessing the complexity of supervised data for automatic classification problems [23]. Several data complexity measures have been introduced [27].

An established taxonomy of data complexity measures is the following [27]:

• Feature-based measures describe how useful the features are to separate the classes.
• Linearity measures try to determine the extent of linear separation between the classes.
• Neighborhood measures characterize class overlapping in neighborhoods.
• Network measures focus on the structural information of the data.
• Dimensionality measures estimate data sparsity.
• Class imbalance measures consider the number of instances in the classes.

In the following analysis, we survey some of the most used data complexity measures of the Feature-based, Linearity, and Neighborhood taxonomies and describe their advantages and disadvantages and their potential to be extended for multidimensional classification.

2.2. Feature-Based Measures

Feature-based measures consider the individual power of the features regarding the classification task. They are usually easy to compute, but they do not consider feature dependencies.

2.2.1. Maximum Fisher’s Discriminant Ratio (F1)

F1 is a measure devoted to assessing the discriminant power of individual features. It is given by:

$$\mathrm{F}1=\frac{1}{1+ma{x}_{i=1}^m{r}_i}$$

(1)

where $r_i$ is a discriminant ratio for the i-th feature. Several formulations of $r_i$ exist in the literature, all considering means and standard deviation of classes. Let ${\mu }_1$ and ${\mu }_2$ be the means of the classes, and let ${\sigma }_1$ and ${\sigma }_2$ be the standard deviations. The original formulation by Ho and Basu [23] is:

$$r_i=\frac{{({\mu }_1-{\mu }_2)}^2}{{{\sigma }_1}^2+{{\sigma }_2}^2}$$

(2)

The F1 measure has several drawbacks. It assumes data is numeric and complete (no missing values) and that the classification problem is binary. In addition, it assumes that the linear boundary is perpendicular to one of the feature axes. If features are separable but with an oblique line, it does not capture such information.

2.2.2. Volume of the Overlapping Region (F2)

The original F2 formulation [23] has the same disadvantages as F1 and fails in classes where one or more features occupy different intervals. It is also susceptible to noise in the data.

$$\begin{array}{c}\mathrm{F}2={\displaystyle {\displaystyle \prod }_i^m}\frac{overlap\left(fi\right)}{range\left(fi\right)}={\displaystyle {\displaystyle \prod }_i^m}\frac{\max \{0,\min \max \left(fi\right)–\max \min \left(fi\right)\}c}{\max \max \left(fi\right)–\min \min \left(fi\right)}\\ \min \max (fi)=\min (\max (f{i}^{c1}),\max (f{i}^{c2}))\\ \max \min (fi)=\max (\min (f{i}^{c1}),\min (f{i}^{c2}))\\ \max \max (fi)=\max (\max (f{i}^{c1}),\max (f{i}^{c2}))\\ \min \min (fi)=\min (\min (f{i}^{c1}),\min (f{i}^{c2}))\end{array}$$

(3)

where $\max \left(f{i}^{cj}\right)$ and $\min \left(f{i}^{cj}\right)$ are the maximum and minimum values is a feature for class c_j ∈ {1, 2}, respectively.

F2 measure has the same disadvantages as F1 but also fails in classes where one or more features occupy different intervals. It is also susceptible to noise in the data. To overcome some of these drawbacks, Cummins introduced a modified version (F2’) [28].

2.2.3. Maximum Individual Feature Efficiency (F3)

F3 measures the capability of individual features for class separation as follows:

$$\mathrm{F}3=mi{n}_{i=1}^m\frac{n_o\left(A_i\right)}{n}$$

(4)

where $n_o\left(A_i\right)$ is the number of overlapping instances according to $A_i$, $I$ is the indicator function, and $n$ is the number of instances ($\left|U\right|$), as:

$$n_o\left(A_i\right)={{\displaystyle \sum }}_{x\in U}I(x\left[i\right]>\max \min \left(A_i\right)\wedge x\left[i\right]<\min \max \left(A_i\right))$$

(5)

In the same manner as F1 and F2, the F3 measure assumes data is numeric, complete, and has only two decision classes.

2.3. Linearity Measures

Linearity measures aim at determining if it is possible to separate the classes by a hyperplane under the hypothesis that a linearly separable problem is simpler than a non-linear one.

2.3.1. Sum of the Error Distance by Linear Programming (L1)

This measure computes the sum of the distances of the instances incorrectly classified as:

$$\mathrm{L}1=1-\frac{1}{1+SumErrorDist}=\frac{SumErrorDist}{1+SumErrorDist}$$

(6)

where an optimization process of a Support Vector Machine (SVM), which is used to compute the desired hyperplane, determines ${\epsilon }_j$ values.

2.3.2. Error Rate of Linear Classifier (L2)

L2 computes the error ratio of the linear SVM classifier. Let $h\left(x\right)$ be the linear classifier, L2 is computed as:

$$\mathrm{L}2=\frac{{{\displaystyle \sum }}_{i=1}^{n}I\left(h\left(x_i\right)\ne y_i\right)}{n}$$

(7)

The main issues with L1 and L2 these measures are:

• It assumes two classes, as it uses an SVM;
• It assumes data is numeric and complete;
• It uses a fixed classifier (SVM) to establish the linearity of the data.

2.3.3. Non-linearity of Linear Classifier (L3)

The L3 measure starts by obtaining an interpolated set of instances of cardinality $l$. Let $h_T\left(x\right)$ be the linear classifier obtained using the training data and $x_i^{\prime }$ be the interpolated instances; L3 is as follows:

$$\mathrm{L}3=\frac{1}{l}{\displaystyle \sum }_{\mathrm{i}=1}^l\mathrm{I}({\mathrm{h}}_{\mathrm{T}}(x_i^{\prime })\ne class\left(x_i^{\prime }\right))$$

(8)

According to [27], L3 is sensitive to how the data from a class are distributed in the border regions and also on how much the convex hulls that delimit the classes overlap. In addition, L3 shares the disadvantages of L1 and L2.

2.4. Neighborhood Measures

The data complexity measures based on neighborhoods aim to capture the structure of the classes and/or the shape of the decision boundaries. Such measures use distances or dissimilarity functions between instances and, therefore, are bounded by $O\left(n^2\right)$.

2.4.1. Fraction of Borderline Points (N1)

A Minimum Spanning Tree is built and denoted as $MST=\left(U,E\right)$. Each vertex corresponds to an instance, and the edges are weighted according to the distance between them. N1 is obtained by calculating the percentage of instances in vertices $\left(x_i,x_j\right)\in U\times U$ involved in edges that connect instances of opposite classes in the generated tree, as:

$$\mathrm{N}1=\frac{1}{n}{\displaystyle \sum }_{i=1}^{n}I(\left(x_i,x_j\right)\in E\wedge class\left(x_i\right)\ne class\left(x_j\right))$$

(9)

The N1 measure is not deterministic, due to the fact that there can be multiple MSTs for the same dataset. However, its formulation is suitable for multiclass data and allows the use of different functions for computing the dissimilarities in hybrid and incomplete data.

2.4.2. Ratio of Intra/Extra Class Nearest Neighbor Distance (N2)

This measure calculates the ratio of two sums: (i) the sum of the distances between each example and its nearest neighbor of the same class (intra class); and (ii) the sum of the distances between each example and its nearest neighbor of another class (extra class)

$$intra\_extra=\frac{{{\displaystyle \sum }}_{i=1}^{n}d(x_i,same\_NN\left(x_i\right))}{{{\displaystyle \sum }}_{i=1}^{n}d(x_i,diff\_NN\left(x_i\right))}$$

(10)

where $d(x_i,same\_NN\left(x_i\right)$ is the distance of $x_i$ to its nearest neighbor of the same class and $d(x_i,diff\_NN\left(x_i\right)$ is the distance of $x_i$ to its nearest enemy. N2 is computed as:

$$\mathrm{N}2=1-\frac{1}{1+intra\_extra}=\frac{intra\_extra}{1+intra\_extra}$$

(11)

2.4.3. Error Rate of the Nearest Neighbor Classifier (N3)

This measure calculates the global error of the nearest neighbor classifier as:

$$\mathrm{N}3=\frac{{{\displaystyle \sum }}_{i=1}^{n}I\left(NN\left(x_i\right)\ne class\left(x_i\right)\right)}{n}$$

(12)

The main issue with this measure is that it is biased towards the majority class and is unsuitable for imbalanced data.

2.4.4. Non-Linearity of the Nearest Neighbor Classifier (N4)

This measure is similar to L3 but uses NN instead of the linear classifier, as:

$$\mathrm{N}4=\frac{1}{l}{\displaystyle \sum }_{i=1}^{l}I(N{N}_T(x_i^{\prime })\ne class\left(x_i^{\prime }\right))$$

(13)

where $l$ is the number of interpolated instances.

The main problem with this instance is that it is unsuitable for hybrid and incomplete data (due to data interpolation).

2.4.5. Fractions of Hyperspheres Covering Data (T1)

The T1 measure aims to capture the topological structure of the data by recursively computing the hyperspheres needed to cover the data, without instances of different classes in the same hypersphere, as:

$$\mathrm{T}1=\frac{\#Hyperspheres\left(T\right)}{n}$$

(14)

where $\#Hyperspheres\left(T\right)$ is the number of hyperspheres needed to cover the data.

The T1 measure assumes data is numeric and complete and has a recursive procedure for computing the hyperspheres with high computational complexity.

2.4.6. Local Set Average Cardinality (LSC)

LSC is a measure of local characterization. It considers the number of instances of the same class surrounding an instance. Then, it considers the average surrounding.

$$\begin{gathered} \mathrm{LSC}=1-\frac{1}{n^2}{{\displaystyle \sum }}_{i=1..n}\left|LS\left(x_i\right)\right| \\ \mathrm{LS}\left(x_i\right)=\left\{x_j|d\left(x_i,x_j\right)<d\left(x_i,diff\_NN\left(x_i\right)\right)\right\} \end{gathered}$$

(15)

This measure can complement N1 and L1 by also revealing the narrowness of the between-class margin [27]. It handles multiclass data, and its ability to handle hybrid and incomplete data will depend on the dissimilarity function used.

Table 1 summarizes the reviewed data complexity measures, considering their proportion, boundaries, computational complexity, and ability to deal with multiclass, hybrid, and incomplete data.

Table 1. Description of the reviewed data complexity measures.

Measure	Proportion	Boundaries	Complexity [27]	Multiclass	Hybrid	Missing Values
F1	Direct	(0, 1]	$O\left(m\ast n\right)$	Yes *	No	No
F2	Direct	(0, 1]	$O\left(m\ast n\ast c\right)$	No	No	No
F3	Direct	[0, 1]	$O\left(m\ast n\ast c\right)$	No	No	No
L1	Direct	[0, 1)	$O\left(n^2\right)$	No	No	No
L2	Direct	[0, 1]	$O\left(n^2\right)$	No	No	No
L3	Direct	[0, 1]	$O\left(n^2+m\ast l\ast c\right)$	No	No	No
N1	Direct	[0, 1]	$O\left(m\ast n^2\right)$	Yes	Yes +	Yes +
N2	Direct	[0, 1)	$O\left(m\ast n^2\right)$	Yes	Yes +	Yes +
N3	Direct	[0, 1]	$O\left(m\ast n\right)$	Yes	No	No
N4	Direct	[0, 1]	$O\left(m\ast n\ast l\right)$	Yes	No	No
T1	Direct	[0, 1]	$O\left(m\ast n^2\right)$	Yes	No	No
LSC	Direct	$\left[0,1-1/n\right]$	$O\left(m\ast n^2\right)$	Yes	Yes+	Yes +

* Some formulations admit multiclass, as in [27]. ⁺ Depending on the dissimilarity function used.

As shown, Feature-based measures do not handle multiclass nor hybrid and incomplete data, as well as Linearity measures. On the other hand, some Neighborhood measures are applicable for such scenarios if using appropriate dissimilarity functions, while others (N3, N4, and T1) do not.

It is important to mention that with the increase in the applicability of machine-learning techniques, the need for validating such techniques has also increased [29]. To such end, formal methods have been developed [30]. Data complexity measures are between two phases of the machine learning cycle: data understanding and data preparation (Figure 2). Depending on the complexity assessment, data preparation techniques can be used more wisely.

Figure 2. Placement of data complexity assessment in the CRISP-DM-KD cycle as in [31].

Unfortunately, to the best of our knowledge, no formal method has been developed to deal with the task of assessing data complexity, and it remains a gap in the scientific research on the topic.

3. Proposed Approach and Results

Our hypothesis is that extending data complexity measures for the multiclass scenario, having hybrid and incomplete data, is possible (Figure 3).

Figure 3. Proposed approach for developing novel data complexity measures.

In the following, we describe the proposed extended measures. It is important to mention that all proposed measures are able to deal with multiclass, hybrid, and incomplete data.

3.1. Extended Feature-Based Measures

3.1.1. Extended Maximum Fisher’s Discriminant Ratio (F1_ext)

We wanted to maintain the idea behind F1 as a way of assessing the discriminant power of individual features. It is given by:

$$\begin{gathered} \mathrm{F}1\_\mathrm{ext}=\frac{1}{1+ma{x}_{i=1}^m{r}_i} \\ {r}_{i\_num}=\frac{{{\displaystyle \sum }}_{j=1..c}\left[\left|\left\{x\in U|class\left(x\right)=j\right\}\right|\ast {\left({{\mu }_j}^i-{\mu }^i\right)}^2\right]}{{{\displaystyle \sum }}_{j=1..c}{{\displaystyle \sum }}_{\left\{x\in U|class\left(x\right)=j\right\}}{\left(x\left[i\right]-{\mu }_j{}^i\right)}^2} \\ {r}_{i\_cat}=\frac{\left|overlap\_cat\left(A_i\right)\right|}{\left|range\_cat\left(A_i\right)\right|} \\ \mathrm{overlap}\_\mathrm{cat}\left(A_i\right)=\left\{v:\exists x,y\in U,v=x\left[i\right]=y\left[i\right]\ne ?\wedge class\left(x\right)\ne class\left(y\right)\right\} \\ \mathrm{range}\_\mathrm{cat}\left(A_i\right)=\left\{v:x\in U,x\left[i\right]=v\wedge x\ne ?\right\} \end{gathered}$$

(16)

For numerical features, ${\mu }^i$ is the mean of feature $A_i$ and ${{\mu }_j}^i$ is the mean of feature $A_i$ considering only the instances in $\left\{x\in U|class\left(x\right)=j\right\}$. Both means are computed, disregarding the instances with missing values (?).

Our definition of overlap for categorical feature values extends [28] by enumerating values appearing in different classes and disregarding missing values, and our definition of range considers all possible values of feature A_i. Using an efficient implementation, the computational complexity of our proposal is bounded by $O\left(m\ast n\right)$.

The proposed measure is able to deal with multiclass hybrid and incomplete data, presenting an advance for data complexity analysis. However, as its predecessor F1, for numerical data, our measure assumes that the linear boundary is unique and perpendicular to one of the feature axes. If a feature is separable but with more than one line, it does not capture such information (Figure 4).

Figure 4. Example of instances linearly separable (a) by one line and (b) by three lines. Note the values of F1_ext are 0.14 and 0.24, respectively.

3.1.2. Extended Volume of the Overlapping Region (F2_ext)

We extended the original F2 measure by using different overlapping ranges for numeric and categorical data. For numeric data, our formulation is close to the original but with extensions.

$$\mathrm{F}2\_\mathrm{ext}=m\ast {{\displaystyle \prod }}_{i=1..m}{r}_i$$

(17)

where:

$$\begin{gathered} r_i=\{\begin{array}{cc}{r}_{i\_cat}& \mathrm{if}\mathrm{feature}i\mathrm{is}\mathrm{categorical}\\ r_{i\_num}& \mathrm{if}\mathrm{feature}i\mathrm{is}\mathrm{numeric}\end{array} \\ {r}_{i\_num}=\frac{\left|overlap\_num\left(A_i\right)\right|}{\left|range\_num\left(A_i\right)\right|}=\frac{\max \{0,\min \max \left(A_i\right)–\max \min \left(A_i\right)\}}{\max \max \left(A_i\right)–\min \min \left(A_i\right)} \\ \begin{array}{l}\min \max (A_i)=\underset{j=1..c}{\min }\underset{x\in U}{\max }\left\{x[i]\ne ?|x\wedge class(x)=j\right\}\\ \max \min (A_i)=\underset{j=1..c}{\max }\underset{x\in U}{\min }\left\{x[i]\ne ?|x\wedge class(x)=j\right\}\\ \max \max (A_i)=\underset{j=1..c}{\max }\underset{x\in U}{\max }\left\{x[i]\ne ?|x\wedge class(x)=j\right\}\\ \min \min (A_i)=\underset{j=1..c}{\min }\underset{x\in U}{\min }\left\{x[i]\ne ?|x\wedge class(x)=j\right\}\end{array} \\ {r}_{i\_cat}=\frac{\left|overlap\_cat\left(A_i\right)\right|}{\left|range\_cat\left(A_i\right)\right|} \end{gathered}$$

(18)

Both minimum and maximum values are computed, disregarding the instances with missing values (?). Our formulation solves the problem of dealing with multiple classes, as well as with missing and hybrid data. Using an efficient implementation, the computational complexity of our proposal is bounded by $O\left(m\ast n\right)$.

As pointed out by Lorena [27], the F2 value can become very small depending on the number of operands in Equation (17); that is, it is highly dependent on the number of features a dataset has. Our extension does not avoid this situation. It has the same problems as F1_ext (Figure 5).

Figure 5. Example of the curse of the dimensionality for F2 measure. (a) Dataset with two overlapping instances. Assume all attributes have the same values. (b) Results for one attribute. (c) Results for two attributes. (d) Results for ten attributes. Note the values of F2_ext values rapidly decrease as the number of attributes increase, even though are only two overlapping instances in all cases.

3.1.3. Extended Maximum Individual Feature Efficiency (F3_ext)

We inspire in the extension to the F3 measure in [28], but we maintain the idea of [27] to the measure providing lower values for simpler problems. Our proposal is as follows:

$$\mathrm{F}3\_\mathrm{ext}=mi{n}_{i=1}^m\frac{n_o\left(A_i\right)}{n}$$

(19)

where $n_o\left(A_i\right)$ is the number of overlapping instances according to $A_i$, as:

$$\begin{gathered} n_o\left(A_i\right)=\{\begin{array}{cc}\left|overlap\_cat\left(A_i\right)\right|& \mathrm{if}\mathrm{feature}i\mathrm{is}\mathrm{categorical}\\ \left|intersect\left(A_i\right)\right|& \mathrm{if}\mathrm{feature}i\mathrm{is}\mathrm{numeric}\end{array} \\ intersect\left(A_i\right)=\{\begin{array}{cc}n& i\mathrm{f}\min \max \left(A_i\right)\ne \min \max \left(A_i\right)\\ \left\{x|\left(\begin{array}{c}(x\left[i\right]>maxmin\left(A_i\right)\wedge \\ (x\left[i\right]<minmax\left(A_i\right)\end{array}\right)\right\}& \mathrm{otherwise}\end{array} \end{gathered}$$

(20)

where $I$ is the indicator function.

For this measure, our formulation solves the problem of dealing with multiple classes, as well as with missing and hybrid data. In addition, it solves the F3 drawback of not penalizing attributes having the same (or very similar) values for all instances (Figure 6).

Figure 6. The drawback of F3 is solved by F3_ext. We use a two-dimensional dataset, having zero for all instances in the $A_2$ attribute, with two overlapping instances. Note the values of F3 and F3_ext are 0.00 and 0.50, respectively.

Using an efficient implementation, the computational complexity of our proposal is bounded by $O\left(m\ast n\right)$.

3.2. Linearity Measures

Linearity measures are based on the idea of designing a hyperplane able to separate decision classes. Due to the fact that we are working with categorical data and with incomplete data, there is no direct way of using the notions of planes with such data. For future works, we will be working with other topological ideas resembling Linearity and able to deal with hybrid and incomplete data.

3.3. Neighborhood Measures

To extend Neighborhood measures, we propose using a dissimilarity function able to deal with mixed and incomplete data, such as HEOM [32]. We also propose using the normalized version (NHEOM) to guarantee the distance function to return values in [0, 1]. Let $ma{x}^i$ and $mi{n}^i$ be the maximum and minimum values of the numeric attribute $A_i$. The NHEOM is as follows:

$$\begin{gathered} NHEOM(x,y)=\frac{\sqrt{{\Sigma }_{i=1}^{m}dis{s}_i{\left(x\left[i\right],y\left[i\right]\right)}^2}}{\sqrt{m}} \\ dis{s}_i\left(x\left[i\right],y\left[i\right]\right)=\{\begin{array}{cc}1& \mathrm{if}x\left[i\right]=?\vee y\left[i\right]=?\\ overlap\left(x\left[i\right],y\left[i\right]\right)& \mathrm{if}{A}_i\mathrm{is}\mathrm{categorical}\\ rd\_diss\left(x\left[i\right],y\left[i\right]\right)& \mathrm{if}{A}_i\mathrm{is}\mathrm{numeric}\end{array} \\ overlap\left(x\left[i\right],y\left[i\right]\right)=\{\begin{array}{cc}0& \mathrm{if}x\left[i\right]=y\left[i\right]\\ 1& \mathrm{otherwise}\end{array} \\ rd\_diss\left(x\left[i\right],y\left[i\right]\right)=\frac{\left|x\left[i\right]-y\left[i\right]\right|}{ma{x}^i-mi{n}^i} \end{gathered}$$

(21)

As shown in Equation (21), the NHEOM function operates by attribute, and for each attribute, it uses one of three cases: for missing values, it returns one. For complete categorical values, the overlap function considers values similar only if they are equal, and for numerical values, the rd_diss function compares them by considering their difference with respect to the maximum difference between values. The normalization procedure (dividing by the square root of the number of attributes) guarantees NHEOM to be in [0, 1].

We maintain the original formulation of N1, N2, and LSC measures, just by using a dissimilarity function able to deal with hybrid and incomplete data. Regarding the N3 measure, to avoid bias toward the majority class, we change the formulation by considering average by class error.

$$\mathrm{N}3\_\mathrm{ext}=\frac{1}{c}\ast \frac{{{\displaystyle \sum }}_{i=1}^c{{\displaystyle \sum }}_{x\in U|class\left(x\right)=d_i}I\left(NN\left(x\right)\ne class\left(x\right)\right)}{\left|\left\{x\in U|class\left(x\right)=d_i\right\}\right|}$$

(22)

This formulation solves the bias by considering the errors for each decision class (Figure 7). It maintains the ability to handle multiclass, hybrid, and incomplete data.

Figure 7. Solving the N3 bias towards the majority class with the new formulation. (a) Balanced dataset with ten instances, two of them misclassified. (b) Imbalanced dataset with ten instances, two of them misclassified. Note how the proposed measure considers there is a full class misclassified in (b).

Due to the difficulties of interpolation in hybrid and incomplete data, we chose not to extend the N4 measure. Similarly, the T1 measure was not considered because of the impossibility of computing hyperspheres with categorical data.

4. Discussion

For discussion, we first analyze the behavior of the proposed measures over synthetic data (Section 4.1), and we compute the measures over publicly available datasets (Section 4.2). All experiments were executed in a Lenovo ThinkPad X1 laptop, with Windows 10 operating system, Intel(R) Core(TM) i7-8550U CPU at 1.80 GHz and 16 GB of RAM. The laptop was not exclusively dedicated to the experiments (all were executed on low priority).

4.1. Synthetic Data

We first supply three explanatory examples for the computation of the proposed measures. The first two of them use two-dimensional datasets, with no missing values (Figure 8 and Figure 9), to be able to visualize the data distribution, and the second example consists of a synthetic hybrid and incomplete dataset, adapted from the well-known play tennis dataset (Figure 10). The results of the data complexity measures over the example datasets are shown in Table 2.

Figure 8. Iris2D dataset. Note that classes are compact, balanced, and fully separated.

Figure 9. Clover dataset. Classes are separated but imbalanced and with disjoints. The dataset resembles a clover flower.

Figure 10. Tennis dataset. This is a modified version of the well-known tennis dataset by Quinlan. Note that it has hybrid numeric and categorical data with missing values.

Table 2. Results of the data complexity measures for synthetic datasets.

Measure	Iris2D	Clover	Tennis
F1_ext	0.059	0.980	0.667
F2_ext	0.000	0.374	0.007
F3_ext	0.000	0.624	0.200
N1	0.020	0.054	0.286
N2	0.023	0.261	0.555
N3_ext	0.000	0.102	0.267
LSC	0.504	0.938	0.893

4.2. Real Data

In this section, we compute the measures over publicly available multiclass hybrid and incomplete datasets. We add the proposed measures to the EPIC software [25,26]. Table 3 summarizes the measures used in the experiments, clarifying their type, proportion, boundaries, computational complexity, and whether or not it is a newly proposed method.

Table 3. Description of the proposed data complexity measures and each measure’s ability to deal with hybrid and incomplete data.

Type	Measure	Proportion	Boundaries	Complexity	New
Feature-based	F1_ext	Direct	[0, 1]	$O\left(m\ast n\right)$	Yes
	F2_ext	Direct	[0, 1]	$O\left(m\ast n\right)$	Yes
	F3_ext	Direct	[0, 1]	$O\left(m\ast n\right)$	Yes
Neighborhood-based	N1	Direct	[0, 1]	$O\left(m\ast n^2\right)$	No
	N2	Direct	[0, 1]	$O\left(m\ast n^2\right)$	No
	N3_ext	Direct	[0, 1]	$O\left(m\ast n^2\right)$	Yes
	LSC	Direct	$\left[0,1-1/n\right]$	$O\left(m\ast n^2\right)$	No

We selected 15 datasets publicly available in the KEEL repository [33]. All datasets correspond to real-life hybrid and incomplete problems (Table 4); all of them are partitioned using stratified five-fold cross-validation.

Table 4. Description of the real datasets used.

Name	Numeric Attributes	Categorical Attributes	Instances	Classes	%Missing	Non-Error Rate
automobile	15	10	205	6	26.83	0.44
bands	19	0	539	2	32.28	0.63
breast	0	9	286	2	3.15	0.66
cleveland	13	0	303	5	1.98	0.36
crx	6	9	690	2	5.36	0.58
dermatology	34	0	366	6	2.19	0.96
hepatitis	19	0	155	2	48.39	0.60
horse-colic	8	15	368	2	98.10	0.53
housevotes	0	16	435	2	46.67	0.93
mammographic	5	0	961	2	13.63	0.75
marketing	13	0	8993	9	23.54	0.23
mushroom	0	22	8124	2	30.53	1.00
post-operative	0	8	90	3	3.33	0.37
saheart	8	1	462	2	0.00	0.51
wisconsin	9	0	699	2	2.29	0.93

We also provide the non-error rate (NER) results for the Nearest Neighbor classifier for each dataset. NER is computed as [34]:

$$NER=\frac{{\Sigma }_{g=1}^{G}S{n}_g}{G}$$

(23)

where

$$S{n}_g=\frac{c_{gg}}{n_g}$$

(24)

The NER measure assumes a confusion matrix of $g$ classes (Figure 11), and it is robust for imbalanced data.

Figure 11. Confusion matrix of c classes.

Table 5 presents the results of the data complexity measures’ computation for Feature-based and Neighborhood measures, and Table 6 shows the execution time (in milliseconds). The complex dataset according to each measure is highlighted in bold.

Table 5. Results for Feature-based and Neighborhood and Dimensionality measures.

Name	F1_ext	F2_ext	F3_ext	N1	N2	N3_ext	LSC
automobile	0.587	0.000	0.000	0.546	0.528	0.520	0.991
bands	0.757	0.000	0.009	0.415	0.450	0.368	0.993
breast	0.490	0.000	0.009	0.311	0.464	0.292	0.991
cleveland	0.735	0.000	0.000	0.588	0.546	0.601	0.993
crx	0.525	0.000	0.004	0.392	0.396	0.394	0.994
dermatology	0.056	0.000	0.000	0.054	0.343	0.040	0.874
hepatitis	0.560	0.000	0.000	0.277	0.364	0.284	0.956
horse-colic	0.178	0.000	0.370	0.419	0.267	0.393	0.993
housevotes	0.889	0.000	0.011	0.088	0.341	0.072	0.893
mammographic	0.911	0.001	0.670	0.252	0.345	0.254	0.993
marketing	0.804	0.644	0.000	0.728	0.706	0.724	1.000
mushroom	0.733	0.000	0.000	0.002	0.269	0.000	0.876
post-operative	0.727	0.000	0.029	0.408	0.489	0.453	0.987
saheart	0.818	0.063	0.005	0.406	0.475	0.424	0.988
wisconsin	0.325	0.161	0.141	0.065	0.280	0.061	0.747

Table 6. Execution time (in milliseconds) to compute the data complexity measures.

Name	F1_ext	F2_ext	F3_ext	N1	N2	N3_ext	LSC
automobile	2.400	3.600	2.800	1056.000	1055.400	949.400	1120.000
bands	2.200	0.000	10.200	4035.200	4407.000	4164.600	3746.800
breast	0.000	0.000	0.000	161.200	204.000	188.000	303.200
cleveland	1.200	0.000	2.000	1297.400	1374.600	1458.200	1402.200
crx	1.400	0.000	2.400	2836.600	3269.600	3329.400	3130.400
dermatology	1.200	0.000	4.600	1490.200	1491.600	1528.400	1467.600
hepatitis	0.600	0.000	5.200	609.400	526.200	539.400	631.400
horse-colic	0.800	0.200	1.600	1276.800	1249.600	1316.000	1555.400
housevotes	0.800	0.000	0.800	849.600	879.400	879.000	808.600
mammographic	1.000	0.000	2.000	3524.200	3600.600	3611.400	3371.200
marketing	35.200	5.600	67.600	664,776.000	726,545.800	729,344.000	646,769.200
mushroom	9.800	4.800	9.200	123,978.800	129,301.400	122,140.800	122,648.000
post-operative	0.000	0.000	0.000	27.800	20.400	27.400	24.800
saheart	0.600	0.000	0.800	547.000	531.400	550.000	542.600
wisconsin	1.400	0.000	2.800	3440.400	3394.000	3137.600	2905.400

As shown, the hardest dataset is marketing. This complexity is shown in the low values of the non-error rate obtained in Table 4. F2_ext measure offers little information, being close to zero for 14 of the studied datasets. F3_ext measure points out that for most datasets, there is at least one attribute with low overlapping for 12 of the analyzed datasets.

The datasets with no clear separation are horse-colic, mammographic, and wisconsin. N1, N2, and N3_ext measures correlate well with the results of the Nearest Neighbor classifier (Table 4), while LSC shows near one value for all datasets.

The Feature-based measures are very fast, in contrast with the Neighborhood-based measures, which depend on the computation of the dissimilarity matrix between instances. For marketing and mushroom datasets, Neighborhood measures took up to two minutes. However, it is important to mention that we used a sequential implementation of the dissimilarity computation, and such time can be significantly diminished with parallel computation.

In addition, for practical purposes, when we want to assess the complexity of a given dataset, we can compute the measures in less than three minutes for the biggest ones. We think this timeframe is suitable for real-world data, and the proposed measures are computationally feasible, even with a sequential implementation.

The limitations of the proposed measures are as follows:

• The F1_ext measure, as its predecessor F1, for numerical data assumes that the linear boundary is unique and perpendicular to one of the feature axes. If a feature is separable but with more than one line, it does not capture such information.
• F2_ext measure can become very small depending on the number of operands in Equation (17); that is, it is highly dependent on the number of features a dataset has.
• Neighborhood measures are bounded by $O\left(m\ast n^2\right)$. For datasets with a huge number of instances, they can be computationally expensive.

5. Conclusions

Our hypothesis related to the fact that extending data complexity measures for the multiclass scenario, having hybrid and incomplete data is possible, has been verified. We have introduced four data complexity measures for multiclass classification problems. All of the proposed measures are able to deal with hybrid (numeric and categorical) and missing data. This will allow knowing the complexity of a complex dataset in advance before using it to train a classifier. We included the proposed measures in the EPIC software [25,26], and we computed the measures for some of the publicly available datasets with satisfactory results.

In experiments with real datasets, it has been found that the Feature-based measures are very fast, in contrast with the Neighborhood-based measures, which depend on the computation of the dissimilarity matrix between instances. Specifically, when performing the experiments on the marketing and mushroom datasets, Neighborhood measures took up to two minutes, while only fractions of a second were invested in the others.

In future work, we want to work with topological ideas resembling Linearity with the intent to design new Linearity-based measures to deal with hybrid and incomplete data. In addition, we want to solve the issue of the F2_ext measure being severely affected by the curse of dimensionality.

Also, in the case of Neighborhood-based measures, we will implement them using parallel computation. This is to reduce the computation time required to obtain the matrix between instances.

A very relevant future work consists of carrying out a deeper analysis of all the complexity measures available in the current state-of-the-art. Then we will apply formal mathematical methods to specify, build, and verify software and hardware systems focused on their application to machine learning solutions. To do this, we will rely heavily on research papers that clearly explain the phases of machine learning and the available formal methods to verify each phase [30].