Evolutionary Mahalanobis Distance-Based Oversampling for Multi-Class Imbalanced Data Classification

Abstract

The number of sensing data are often imbalanced across data classes, for which oversampling on the minority class is an effective remedy. In this paper, an effective oversampling method called evolutionary Mahalanobis distance oversampling (EMDO) is proposed for multi-class imbalanced data classification. EMDO utilizes a set of ellipsoids to approximate the decision regions of the minority class. Furthermore, multi-objective particle swarm optimization (MOPSO) is integrated with the Gustafson–Kessel algorithm in EMDO to learn the size, center, and orientation of every ellipsoid. Synthetic minority samples are generated based on Mahalanobis distance within every ellipsoid. The number of synthetic minority samples generated by EMDO in every ellipsoid is determined based on the density of minority samples in every ellipsoid. The results of computer simulations conducted herein indicate that EMDO outperforms most of the widely used oversampling schemes.

1. Introduction

With advancements in sensor technology and the Internet of things (IoT), vast quantities of sensing data have been collected and analyzed for different applications. Cost-effective sensors are widely used in our everyday lives to collect various types of data for further online or offline analyses and applications. The classification of real-world sensing data is a highly important research topic in the field of data mining and machine learning. However, the data sets collected using sensors or other sensing techniques usually have a skewed class distribution because the number of data points vary greatly between classes. Such data are called imbalanced data. The data utilized in applications, such as anomaly detection in high-speed trains [1,2,3], fault diagnosis of motors [4,5,6], fault detection and diagnosis in manufacturing processes [7,8,9], and medical diagnosis [10,11,12], are usually imbalanced. In imbalanced data sets, at least one class of data has significantly more data points compared with other classes. Learning on imbalanced data results in poor performance, and this problem has thus attracted considerable research attention in recent years. It is mainly because the performance of many conventional learning algorithms is degraded on the skewed class distribution of imbalanced data sets [13].

Balanced class distribution [14] or equal weighting of classification errors for every class [13] is generally assumed in most conventional machine learning algorithms. For instance, 95% and 5% of imbalanced data sets can comprise majority and minority class samples, respectively. With the equal weighting of the classification errors, traditional classification approaches tend to overlook several or most of the minority class samples in the attempt to minimize the overall classification error. Consequently, although the overall classification error rate is low, the classification error rate for the minority class is high. Minority class samples are important in classification in certain applications, such as medical diagnosis, anomaly detection, and fault detection and diagnosis. Majority class samples usually represent normal conditions, and minority class samples represent abnormal conditions, which can be key in such applications. The learning approaches for imbalanced data are designed to increase learning accuracy with respect to minority classes without trading off learning accuracy with respect to majority classes.

The learning approaches for imbalanced data can generally be categorized into three types: cost-sensitive learning, data-level learning, and ensemble learning, which are comprehensively reviewed in [15,16]. Cost-sensitive learning assigns higher misclassification costs to minority class samples than to majority class samples. Studies have proposed various learning approaches that adjust the misclassification cost using kernel functions; these approaches involve the radial basis function [17], matrix-based kernel regression [18], support vector machine [19,20], and deep learning [21,22].

Data-level learning approaches essentially rebalance the skewed data distribution of different classes by removing several majority class samples or by adding new minority class samples, and they can generally be divided into undersampling and oversampling learning approaches. The main advantage of data-level learning approaches is that they are independent of classifiers. They can be considered a type of data preprocessing approach. Therefore, data-level learning approaches can be easily integrated with other imbalanced learning approaches. Undersampling involves removing the majority class samples to ensure that the learning results are not overly biased toward the majority class [23,24,25,26]. Undersampling reduces both the number of samples in and the computational cost of machine learning. However, it tends to reduce the model’s capability to recognize majority classes. By contrast, oversampling involves increasing the number of minority class samples by resampling them or by generating synthetic samples. However, resampling by simply replicating the minority class samples does not improve learning of the decision region of minority classes. The synthetic minority oversampling technique (SMOTE) proposed in [27] selects several samples from a minority class. Searching in the vicinity of the selected samples, it identifies other samples of the minority class to generate new synthetic samples linearly between the two points. SMOTE is the most widely used oversampling technique because of its computational inexpensiveness. However, SMOTE is prone to overgeneralization because it synthesizes new samples through the random selection of minority class samples. Various adaptive sampling methods based on SMOTE have been proposed to overcome its limitation. The adaptive synthetic sampling approach for imbalanced learning algorithm (ADASYN) [28], SMOTEBoost [29], and Borderline SMOTE [30] are effective modified versions of SMOTE. In contrast to SMOTE, other algorithms, such as those in [31,32,33], have been proposed; these algorithms generate synthetic samples by learning the structure underlying the minority samples.

Ensemble learning for incomplete data integrates traditional machine learning approaches, such as boosting [34], bagging [35], and stacking [36], with other cost-sensitive or data resampling imbalanced learning approaches. In [37], SMOTE was integrated with Adaboost [38] to increase the number of minority samples and to assign higher weights to misclassified minority samples. A similar integration of Adaboost with a novel synthetic sampling method was proposed in [39]. The performance of boosting, bagging, and other hybrid techniques applied to imbalanced data has been compared in [40] and [41].

In the methods proposed in [31,32,33], synthetic samples are generated based not on individual minority samples as proposed in SMOTE [30] but on the underlying structure of the minority samples. Recently, a similar oversampling approach called Mahalanobis distance-based oversampling (MDO) was proposed in [42]. MDO generates synthetic samples based on the structure of the principal component space of minority samples. The synthetic samples generated by MDO have the same Mahalanobis distance as that of the considered minority sample. Because the class mean of the synthetic samples generated by MDO is the same as that of the minority class samples, the covariance structure of the minority class samples is preserved. In [43], a scheme called adaptive Mahalanobis distance oversampling (AMDO) was proposed. AMDO integrates generalized singular value decomposition with MDO to solve the oversampling problem encountered in mixed-type imbalanced data sets. Either MDO or AMDO can be utilized as a direct learning approach for solving problems with multi-class imbalanced problems.

The oversampling results obtained from MDO or AMDO are equivalent to those obtained by placing minority class samples and generated synthetic samples into the principal component space. The minority class samples and the synthetic samples can be considered to be included in an ellipsoid centered at the class mean. The orientation of the ellipsoid depends on the covariance structure of the minority class samples. The synthetic samples do not change the covariance structure of the minority class because all the synthetic samples are generated within the ellipsoid. However, both MDO and AMDO use only one ellipsoid to include the minority class samples and synthetic samples. If the decision regions of the minority class are separated, the decision region approximated using only one ellipsoid may overlap with the decision regions of other classes. This is especially true for imbalanced multi-class data. Samples from different classes may be included in a single ellipsoid structure depending on the target minority class samples. The synthetic samples generated by MDO or AMDO are randomly assigned in the single ellipsoid only if they have the same Mahalanobis distance as that of the associated minority sample. When synthetic samples are generated within a single ellipsoid, the generated synthetic samples tend to be placed in the cluster of samples belonging to other classes. This reduces the effectiveness of oversampling. Moreover, certain decision regions (e.g., those that are ring- or belt-shaped) are difficult to approximate with only one ellipsoid.

A novel approach called evolutionary Mahalanobis distance oversampling (EMDO) is proposed in this paper to overcome the limitations of MDO and AMDO. EMDO utilizes multiple ellipsoids to learn the distribution and orientation of minority class samples in parallel. Gustafson and Kessel proposed a clustering algorithm called the Gustafson–Kessel algorithm (GKA) [44], which is similar to the widely used fuzzy c-means [45] clustering approach with Mahalanobis norms. The advantage of the GKA over fuzzy c-means is that it utilizes the Mahalanobis norm instead of the Euclidean norm to learn the underlying sample distribution. However, the GKA assumes a fixed volume before learning the center and orientation of every ellipsoid. The GKA is an effective clustering approach for learning the centers and orientations of data clusters, but it is unsuitable for learning the decision regions of data due to its assumption of a fixed ellipsoid size. The GKA was modified in [46,47] to adaptively learn ellipsoid sizes for pattern recognition problems by using the genetic algorithm with a single objective function. In the proposed EMDO, the GKA is integrated with multi-objective particle swarm optimization (MOPSO) [48,49] to ensure that the centers, orientations, and sizes of multiple ellipsoids, along with the overall misclassification error, are learned in parallel. The misclassification error is defined as the total number of misclassified samples included in a union of multiple ellipsoids. Therefore, EMDO can learn a set of ellipsoids to approximate connected or disconnected complex decision regions with reasonable accuracy. Because multiple ellipsoids are learned in parallel in EMDO, an effective approach is designed to adaptively determine the number of synthetic samples to be generated in every ellipsoid. Similar ideas that design suitable algorithms to search for model parameters for specific applications are shown in [50,51,52].

The technical novelty and main contribution of this paper are summarized as follows.

1)

An effective novel oversampling approach called EMDO is proposed for multi-class imbalanced data problems. Different from the MDO and AMDO approaches that use only one ellipsoid, EMDO learns multiple ellipsoids in parallel to approximate the decision region of the target minority class samples.

2)

MOPSO is utilized along with GKA in EMDO to optimize the parameters, including the centers, orientations, and sizes of multiple ellipsoids approximating the target class of decision regions with reasonable accuracy.

3)

Synthetic minority samples are generated based on the Mahalanobis distance within every ellipsoid learned by EMDO. A novel adaptive approach is proposed to determine the number of synthetic minority samples to be generated based on the density of minority samples in every ellipsoid.

4)

EMDO was evaluated and found to perform better than other widely used oversampling schemes.

The remainder of this paper is organized as follows. Section 2 presents the problem formulation of oversampling for imbalanced data. The GKA is introduced in this section, and it shows that the GKA is suitable to solve the problem formulated herein. Section 3 introduces the proposed multi-objective optimization scheme designed in the EMDO, which uses MOPSO. Section 4 details the method for calculating the number of ellipsoids required to approximate the decision regions of every class. Section 5 describes performance evaluation of EMDO against other widely used oversampling schemes. Finally, Section 6 concludes the study.

2. Problem Statement and GKA

Given a data set $\mathit{S}=\{\left.({\mathit{x}}_{\mathit{i}},y_i)\right|{\mathit{x}}_{\mathit{i}}\in R^d, y_i\in \{1\dots p\},i=1\dots N\}$, every ith sample ${\mathit{x}}_{\mathit{i}}\in \mathit{S}$ belongs to some class $y_i$ among p classes. Let ${\mathit{S}}_{\mathit{j}}\subset \mathit{S}$ be the set containing the samples belonging to class j, j = 1…p. Denote $N_j\equiv \left|{\mathit{S}}_{\mathit{j}}\right|$ as the number of samples in ${\mathit{S}}_{\mathit{j}}$, where $N_{\min }=\underset{j=1\dots p}{\min }(N_j)$, $N_{\max }=\underset{j=1\dots p}{\max }(N_j)$. The data set S is imbalanced if $(N_j/N_{\max })$ is less than a preset imbalance ratio, IR. The value of IR is determined based on the size of S and on the characteristics of the classification problem. Typically, IR ≥ 1.5. ${\mathit{S}}_{\mathit{j}}$ is called a minority set if $(\left|{\mathit{S}}_{\mathit{j}}\right|/N_{\max })<IR$, j = 1…p. An oversampling technique is applied in this study to overcome the skewed distribution of samples in S. If ${\mathit{S}}_{\mathit{j}}$ is a minority set, the synthetic samples belonging to the same jth class are generated in ${\mathit{S}}_{\mathit{j}}$ to form an enlarged set ${\tilde{\mathit{S}}}_{\mathit{j}}$ such that $\left|{\tilde{\mathit{S}}}_{\mathit{j}}\right|=N_{\max }/IR$. Denote ${\tilde{N}}_j$ as the total number of extra synthetic samples generated to balance the minority set ${\mathit{S}}_{\mathit{j}}$,

$${\tilde{N}}_j=(N_{\max }/IR-\left|{\mathit{S}}_{\mathit{j}}\right|).$$

(1)

Note that there can be more than one minority set in a multi-class problem. An oversampling technique is proposed herein to improve classification accuracy on an imbalanced data set. To generate an adequate number of synthetic samples and place them in the minority sets, the distribution of decision regions of every minority set in the d-dimensional feature space must be located. Multiple ellipsoids are utilized in this study to approximate the decision regions of minority sets. EMDO is proposed to learn these ellipsoids and generate synthetic samples in these ellipsoids for oversampling.

Assume that ellipsoids approximate the decision region of the jth class samples. Denote the center of every nth ellipsoid as ${\mathit{v}}_{\mathit{n}}^{\mathit{j}}\in R^d$. The distance between every kth sample ${\mathit{x}}_{\mathit{k}}$ and the ellipsoid center ${\mathit{v}}_{\mathit{n}}^{\mathit{j}}$ is defined in the Mahalanobis form as follows:

$${\lambda }_{nk}^j={({({\mathit{x}}_k-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})}^T{\mathit{M}}_n^j({\mathit{x}}_k-{\mathit{v}}_{\mathit{n}}^{\mathit{j}}))}^{1/2},$$

(2)

where ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}\in R^{d\times d}$ is a norm-inducing matrix. The ellipsoid ${\Phi }_n^j$ is defined as follows by using the Mahalanobis distance defined in (2):

$${\Phi }_n^j({\mathit{x}}_k)={({\mathit{x}}_k-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})}^T{\mathit{M}}_n^j({\mathit{x}}_k-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})=1.$$

(3)

The sample ${\mathit{x}}_{\mathit{k}}$ is inside or on the ellipsoid if ${\Phi }_n^j({\mathit{x}}_{\mathit{k}})\le 1$, but it is outside the ellipsoid if ${\Phi }_n^j({\mathit{x}}_{\mathit{k}})>1$. Let the decision region of the jth class samples in the feature space be denoted as ${\Re }^j$; ${\Re }^j$ is approximated by the union of ${\alpha }^j$ ellipsoids, that is,

$${\Re }^j\cong {\displaystyle \underset{n=1\dots {\alpha }^j}{\cup }{\Phi }_n^j}.$$

(4)

The GKA is used for learning the ${\alpha }^j$ ellipsoids in parallel, given that the size of each ellipsoid is assigned. Denote the size of the ellipsoid ${\Phi }_n^j$ in (3) as ${\xi }_n^j$, n = 1…${\alpha }^j$. The determinant of the norm-inducing matrix ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}$ is inversely proportional to ${\xi }_n^j$. Therefore,

$$\det ({\mathit{M}}_{\mathit{n}}^{\mathit{j}})=1/{\xi }_n^j, n=1\dots {\alpha }^j.$$

(5)

The GKA learns the norm-inducing matrices ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}$ and the ellipsoid centers ${\mathit{v}}_{\mathit{n}}^{\mathit{j}}$ through iteratively calculating an auxiliary fuzzy partition matrix ${\mathit{U}}^{\mathit{j}}\in R^{{\alpha }^j\times N^j}$ by using all $N^j$ samples belonging to the jth class. The element ${\mu }_{nk}^j\in {\mathit{U}}^{\mathit{j}}$ represents the membership value of the kth sample ${\mathit{x}}_{\mathit{k}}$ associated with the nth ellipsoid ${\Phi }_n^j$. The membership values sum to 1 for every ${\mathit{x}}_{\mathit{k}}$, that is,

$${\displaystyle \sum _{n=1}^{{\alpha }^j}{\mu }_{nk}^j}=1, k=1\dots N^j.$$

(6)

The GKA is a fast iterative learning algorithm that efficiently updates the membership values in the fuzzy partition matrix ${\mathit{U}}^{\mathit{j}}$ while learning both the norm-inducing matrix ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}$ and the center ${\mathit{v}}_{\mathit{n}}^{\mathit{j}}$ of every nth ellipsoid. Note that the GKA learns all ${\alpha }^j$ ellipsoids in parallel. Denote the matrices containing the ellipsoid centers and the norm-inducing matrices as ${\mathit{V}}^{\mathit{j}}$ and ${\mathit{M}}^{\mathit{j}}$, respectively, that is, ${\mathit{V}}^{\mathit{j}}=[{\mathit{v}}_1^{\mathit{j}},{\mathit{v}}_2^{\mathit{j}}\dots ,{\mathit{v}}_{{\mathsf{\alpha }}^{\mathit{j}}}^{\mathit{j}}]$ and ${\mathit{M}}^{\mathit{j}}=[{\mathit{M}}_1^{\mathit{j}},{\mathit{M}}_2^{\mathit{j}},\dots ,{\mathit{M}}_{{\mathsf{\alpha }}^{\mathit{j}}}^{\mathit{j}}]$. All elements in the triple $({\mathit{U}}^{\mathit{j}},{\mathit{V}}^{\mathit{j}},{\mathit{M}}^{\mathit{j}})$ are learned by iteratively minimizing the distance in (2) weighted with the membership values in ${\mathit{U}}^{\mathit{j}}$ subject to the constraints in (5) and (6). Let ${\omega }_n^j$, n = 1…${\alpha }^j$ and ${\varpi }_k^j$, k = 1…$N^j$ be the Lagrange multipliers of the constraints in (5) and (6), respectively. The triple $({\mathit{U}}^{\mathit{j}},{\mathit{V}}^{\mathit{j}},{\mathit{M}}^{\mathit{j}})$ is iteratively learned as follows:

$$({\mathit{U}}^{\mathit{j}},{\mathit{V}}^{\mathit{j}},{\mathit{M}}^{\mathit{j}})=\arg \min ({\displaystyle \sum _{n=1}^{{\alpha }^j}{\displaystyle \sum _{k=1}^{N^j}{({\mu }_{nk}^j)}^b{({\lambda }_{nk}^j)}^2}}+{\displaystyle \sum _{n=1}^{{\alpha }^j}{\omega }_n^j(\det ({\mathit{M}}_{\mathit{n}}^{\mathit{j}})}-1/{\xi }_n^j)+{\displaystyle \sum _{k=1}^{N^j}{\varpi }_k^j({\displaystyle \sum _{n=1}^{{\alpha }^j}({\mu }_{nk}^j-1))}},$$

(7)

where b is an adjusted weighting index. The optimization described in (7) is realized by differentiating (7) with respect to ${\mu }_{nk}^j, {\mathit{v}}_{\mathit{n}}^{\mathit{j}}, {\omega }_n^j$, and ${\varpi }_k^j$,and by equating the result to 0.

The parameters are obtained as follows:

$${\mu }_{nk}^j=\frac{1}{{\displaystyle \sum _{i=1}^{{\alpha }^j}{({\lambda }_{nk}^j/{\lambda }_{ik}^j)}^{2/(b-1)}}}, n=1\dots {\alpha }^j, k=1\dots N^j;$$

(8)

$${\mathit{v}}_{\mathit{n}}^{\mathit{j}}=\frac{{\displaystyle \sum _{k=1}^{N^j}{({\mu }_{nk}^j)}^b{\mathit{x}}_{\mathit{k}}}}{{\displaystyle \sum _{k=1}^{N^j}{({\mu }_{nk}^j)}^b}}, n=1\dots {\alpha }^j;$$

(9)

$${\mathit{F}}_{\mathit{n}}^{\mathit{j}}=\frac{{\displaystyle \sum _{k=1}^{N^j}{({\mu }_{nk}^j)}^b({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}}){({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})}^T}}{{\displaystyle \sum _{k=1}^{N^j}{({\mu }_{nk}^j)}^b}}, n=1\dots {\alpha }^j;$$

(10)

$$\mathrm{and}{\mathit{M}}_{\mathit{n}}^{\mathit{j}}={({\zeta }_n^j\det ({\mathit{M}}_{\mathit{n}}^{\mathit{j}}))}^{1/d}{({\mathit{M}}_{\mathit{n}}^{\mathit{j}})}^{-1} n=1\dots {\alpha }^j.$$

(11)

The iteration in GKA is stopped when no significant improvement is made in the fuzzy partition matrix ${\mathit{U}}^{\mathit{j}}$. Let ${({\mathit{U}}^{\mathit{j}})}^{(m)}$ be the fuzzy partition matrix learned in the mth iteration, the norm of the difference between ${({\mathit{U}}^{\mathit{j}})}^{(m)}$ and ${({\mathit{U}}^{\mathit{j}})}^{(m+1)}$ can be defined as

$${\delta }^j=‖{({\mathit{U}}^{\mathit{j}})}^{(m+1)}-{({\mathit{U}}^{\mathit{j}})}^{(m)}‖\equiv \underset{n,k}{\max }\left|{({\mu }_{nk}^j)}^{(m+1)}-{({\mu }_{nk}^j)}^{(m)}\right|.$$

(12)

The GKA iteratively learns ${\mathit{U}}^{\mathit{j}}$, ${\mathit{V}}^{\mathit{j}}$, and ${\mathit{M}}^{\mathit{j}}$ until ${\delta }^j<{\epsilon }^j$, where ${\epsilon }^j$ is a small constant. The flowchart of the GKA is illustrated in Figure 1.

3. Multi-Objective Optimization in EMDO

As depicted in Figure 1 and described in Section 2, the GKA optimizes the centers and norm-inducing matrices of multiple ellipsoids in parallel with a preset size of every ellipsoid. If the ellipsoid size is set inappropriately, the ellipsoids learned by the GKA cannot accurately include all minority class samples. According to (4), each jth-class decision region ${\Re }^j$ is approximated by the union of ${\alpha }^j$ ellipsoids ${\Phi }_n^j$ of size ${\xi }_n^j$, n = 1…${\alpha }^j$. Consider the sets ${\mathsf{\Phi }}^{\mathit{j}}=\{{\Phi }_1^j,{\Phi }_2^j,\dots ,{\Phi }_{{\alpha }^j}^j\}$ and ${\mathsf{\Xi }}^{\mathit{j}}=\{{\xi }_1^j,{\xi }_2^j,\dots ,{\xi }_{{\alpha }^j}^j\}$. The distance between the kth sample ${\mathit{x}}_{\mathit{k}}$ and ${\Re }^j$, denoted $L({\mathit{x}}_{\mathit{k}},{\Re }^j)$, can be defined as the minimum distance between ${\mathit{x}}_{\mathit{k}}$ and the center of each ellipsoid, as follows:

$$L({\mathit{x}}_{\mathit{k}},{\Re }^j)\cong L(x_k,{\mathsf{\Phi }}^{\mathit{j}})=\underset{n=1\dots {\alpha }^j}{\min }{\lambda }_{nk}^j,$$

(13)

where ${\lambda }_{nk}^j$ is defined in (2). The sample ${\mathit{x}}_{\mathit{k}}$ is included in ${\Re }^j$ if $L(x_k,{\Re }^j)\le 1$ and the corresponding class $y_k=j$. Denote a binary function $H(\cdot )$ as follows:

$$H(o)=\left\{\begin{array}{c}1, \mathrm{if} \mathrm{the} \mathrm{logic} \mathrm{statement} o \mathrm{is} \mathrm{true};\\ 0, \mathrm{if} \mathrm{the} \mathrm{logic} \mathrm{statement} o \mathrm{is} \mathrm{false}. \end{array}\right.$$

(14)

The total number of jth-class samples included in the set ${\mathsf{\Phi }}^{\mathit{j}}$ can be calculated as

$$O_{included}^j({\mathsf{\Phi }}^{\mathit{j}})={\displaystyle \sum _{k=1}^{N}H(L({\mathit{x}}_{\mathit{k}},} {\mathsf{\Phi }}^{\mathit{j}})\le 1 \mathrm{and} y_k=j).$$

(15)

It is possible that several samples that do not belong to the jth class are included in the set of ellipsoids ${\mathsf{\Phi }}^{\mathit{j}}$. The total number of samples not belonging to the jth class but included in ${\mathsf{\Phi }}^{\mathit{j}}$ can be calculated as

$$O_{included}^{\backslash j}({\mathsf{\Phi }}^{\mathit{j}})={\displaystyle \sum _{k=1}^{N}H(L({\mathit{x}}_{\mathit{k}},} {\mathsf{\Phi }}^{\mathit{j}})\le 1 \mathrm{and} y_k\ne j),$$

(16)

where $N_{included}^{\backslash j}(\cdot )\le (N-N^j)$.

Referring to (5), the total size of the ellipsoids contained in ${\mathsf{\Phi }}^{\mathit{j}}$ can be calculated as

$${\mathbb{F}}_1({\mathsf{\Xi }}^{\mathit{j}})={\displaystyle \sum _{n=1}^{{\alpha }^j}{\xi }_n^j}.$$

(17)

The proposed EMDO not only minimizes the total ellipsoid sizes but also aims to simultaneously maximize the number of jth class samples included in the set of ellipsoids ${\mathsf{\Phi }}^{\mathit{j}}$ and minimize the number of samples included in ${\mathsf{\Phi }}^{\mathit{j}}$ but not belonging to jth class. The misclassification error can be defined as the summation of the number of jth class samples not included in ${\mathsf{\Phi }}^{\mathit{j}}$, calculated as $(N^j-O_{included}^j({\left.{\mathsf{\Phi }}^{\mathit{j}}\right|}_{{\mathsf{\Xi }}^{\mathit{j}}})$, and the number of samples that are included in ${\mathsf{\Phi }}^{\mathit{j}}$ but that do not belong to the jth class is calculated as $O_{included}^{\backslash j}({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}})$. Therefore, the misclassification error can be defined as

$${\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}})=N^j-O_{included}^j({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}})+O_{included}^{\backslash j}({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}}).$$

(18)

The misclassification error can be utilized as an objective function to optimize the ellipsoid sizes. The set of ellipsoid sizes ${\mathsf{\Xi }}^{\mathit{j}}=[{\xi }_1^j,{\xi }_2^j,\dots ,{\xi }_{{\alpha }^j}^j]$ can be optimized using a multi-objective optimization scheme that minimizes both (17) and (18).

MOPSO is utilized to perform this multi-objective optimization by searching for the best set of ellipsoid sizes ${\mathsf{\Xi }}^{\mathit{j}}$ by minimizing the objective functions ${\mathbb{F}}_1(\cdot )$ and ${\mathbb{F}}_2(\cdot )$. Assume that G particles are utilized in the MOPSO. Denote ${\mathsf{\Xi }}^{\mathit{j}}(k,g)$ as the gth particle in the kth iteration and ${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)$ as the non-dominated solution subject to the following multi-objective optimization. Note that the multi-objective optimization searches for the non-dominated solution of every particle.

$${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)=\underset{{\mathsf{\Xi }}^{\mathit{j}}(k,g), \forall {\mathit{x}}^{\mathit{k}}\in S, g=1\dots G}{Arg\min }({\mathbb{F}}_1({\mathsf{\Xi }}^{\mathit{j}}(k,g)),{\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}(k,g)})).$$

(19)

${\mathsf{\Xi }}^{\mathit{j}}(k,g)$ is defined to be a non-dominated solution, as given by (19), if it is not dominated by any other particle, that is, if both ${\mathbb{F}}_1({\mathsf{\Xi }}^{\mathit{j}}(k,g))\le {\mathbb{F}}_1({\mathsf{\Xi }}^{\mathit{j}}(k,i))$ and ${\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}(k,g)})\le {\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}(k,i)})$, i=1…G, $i\ne g$. Let

$$a_i({\mathsf{\Xi }}^{\mathit{j}}(k,g))=H({\mathbb{F}}_1({\mathsf{\Xi }}^{\mathit{j}}(k,g))>{\mathbb{F}}_1({\mathsf{\Xi }}^{\mathit{j}}(k,i)) \mathrm{or} {\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}(k,g)})>{\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\mathsf{\Xi }}^{\mathit{j}}(k,i)}))$$

(20)

$$A({\mathsf{\Xi }}^{\mathit{j}}(k,g))={\displaystyle \sum _{i=1\dots G,i\ne g}^{}{a}_i({\mathsf{\Xi }}^{\mathit{j}}(k,g))}.$$

(21)

The non-dominated solution obtained using the gth particle, denoted as ${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)$, is updated as ${\mathsf{\Xi }}^{\mathit{j}}(k,g)$ if $A({\mathsf{\Xi }}^{\mathit{j}}(k,g))$=0. However, ${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)$ remains unchanged if $A({\mathsf{\Xi }}^{\mathit{j}}(k,g))$>0.

$${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)=\left\{\begin{array}{c}{\mathsf{\Xi }}^{\mathit{j}}(k,g), \mathrm{if} A({\mathsf{\Xi }}^{\mathit{j}}(k,g))=0;\\ {\overline{\mathsf{\Xi }}}^{\mathit{j}}(g), \mathrm{otherwise}. \end{array}\right.$$

(22)

Only if ${\mathsf{\Xi }}^{\mathit{j}}(k,g)$ is the non-dominated solution of (19) will it be included in the repository ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$, which is the set of all of non-dominated solutions of (19). The best solution achieved by the gth particle, denoted ${\mathsf{\Xi }}_{\mathit{p}\_\mathit{best}}^{\mathit{j}}(g)$, is updated using this non-dominated solution generated by the gth particle, that is, ${\mathsf{\Xi }}_{\mathit{p}\_\mathit{best}}^{\mathit{j}}(g)$ = ${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)$. With reference to (22), if ${\overline{\mathsf{\Xi }}}^{\mathit{j}}(g)$ is not generated in the current kth iteration, ${\mathsf{\Xi }}_{\mathit{p}\_\mathit{best}}^{\mathit{j}}(g)$ remains unchanged.

It is possible that several original non-dominated solutions stored in ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$ become dominated after a new non-dominated solution is included in ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$. A process for filtering out non-dominated solutions, similar to the one proposed in (20) and (21), is executed for all of the non-dominated solutions in ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$. Assume a total of $m_{rep}$ non-dominated solutions are in the repository, including the newly generated one. Each of the $m_{rep}$ non-dominated solutions is compared with all of the other solutions to evaluate whether they are being dominated. Denote ${\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}$ as the mth non-dominated solution. Let

$$a_i({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})=H({\mathbb{F}}_1({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})>{\mathbb{F}}_1({\overline{\mathsf{\Xi }}}_{\mathit{i}}^{\mathit{j}}) \mathrm{or} {\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})})>{\mathbb{F}}_2({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{i}}^{\mathit{j}}}))$$

(23)

$$A({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})={\displaystyle \sum _{i=1\dots m_{rep},i\ne m}^{}{a}_i}({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}).$$

(24)

${\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}$ is no longer a non-dominated solution and is excluded from ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$ if $A({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})$ > 0.

An adaptive grid algorithm [53] is applied to ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$ after the filtering process is completed, as given in (23) and (24), to place all the non-dominated solutions into several grids. The global best particle ${\mathsf{\Xi }}_{\mathit{g}\_\mathit{best}}^{\mathit{j}}$ is randomly selected from among the grids in ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$ by using the roulette wheel selection scheme. After both ${\mathsf{\Xi }}_{\mathit{p}\_\mathit{best}}^{\mathit{j}}(g)$ and ${\mathsf{\Xi }}_{\mathit{g}\_\mathit{best}}^{\mathit{j}}$ are determined, each particle is updated as follows:

$$\mathsf{\tau }(k,g)=\mathsf{\tau }(k-1,g)+c_1{\gamma }_1({\mathsf{\Xi }}_{\mathit{p}\_\mathit{best}}^{\mathit{j}}(g)-{\mathsf{\Xi }}^{\mathit{j}}(k,g))+c_2{\gamma }_2({\mathsf{\Xi }}_{\mathit{g}\_\mathit{best}}^{\mathit{j}}-{\mathsf{\Xi }}^{\mathit{j}}(k,g)),$$

(25)

$${\mathsf{\Xi }}^{\mathit{j}}(k,g)={\mathsf{\Xi }}^{\mathit{j}}(k-1,g)+\mathsf{\tau }(k,g), g=1\dots G,$$

(26)

where $c_1$ and $c_2$ are preset constants and ${\gamma }_1, {\gamma }_2\in [0,1]$ are randomly generated real numbers. If no new non-dominated solution can be successfully included in ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$ after the filtering process for certain preset $K_{thr}$ iterations, MOPSO is saturated, and the iterative learning of particles given by (19)–(26) is stopped.

The optimal solution of the multi-objective optimization problem in (19) is searched from ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$ after the iterative learning process is stopped. Let ${\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}=\{{\overline{\xi }}_{m1}^j,{\overline{\xi }}_{m2}^j,\dots ,{\overline{\xi }}_{m{\alpha }^j}^j\}$ be the mth non-dominated solution in ${\mathit{S}}_{\mathit{rep}}^{\mathit{j}}$. The average density of the ellipsoids associated with ${\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}$ is defined as

$$d({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}})=\frac{O_{included}^j({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}})}{{\displaystyle \sum _{n=1}^{{\alpha }^j}{\overline{\xi }}_{mn}^j}}.$$

(27)

The optimal solution ${({\mathsf{\Xi }}^{\mathit{j}})}^{\ast }$ can be selected based on the average density of the non-dominated solutions because the ellipsoids associated with the optimal solution tend to have a small size but a large number of samples. However, the average density in (27) cannot be directly utilized as the optimization index for selecting the optimal solution. It is modified as the ratio of the total number of jth class samples included in the set ${\mathsf{\Phi }}^{\mathit{j}}$ multiplied with the ratio of the total number of samples not belonging to the jth class but included in ${\mathsf{\Phi }}^{\mathit{j}}$. Let the evaluation index for the mth non-dominated solution ${\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}$ be ${\sigma }^j({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})$, which is defined based on (27) as follows:

$${\sigma }^j({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}})=d({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}})\times \frac{O_{included}^j({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}})}{N^j}\times \frac{(N-N^j-O_{included}^{\backslash j}({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}}))}{N-N^j},$$

(28)

where $N^{\backslash j}({\mathsf{\Phi }}^{\mathit{j}}{|}_{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}})$ denotes the number of samples not belonging to the jth class but included in ${\mathsf{\Phi }}^{\mathit{j}}$. The optimal solution ${({\mathsf{\Xi }}^{\mathit{j}})}^{\ast }$ can be defined as the set of ellipsoid sizes that maximize the index ${\sigma }^j(·)$, that is,

$${({\mathsf{\Xi }}^{\mathit{j}})}^{\ast }=\underset{{\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}, m=1\dots m_{rep}}{Arg\max }{\sigma }^j({\overline{\mathsf{\Xi }}}_{\mathit{m}}^{\mathit{j}}).$$

(29)

After the optimal ellipsoid sizes are determined by MOPSO according to (19) and (29), the GKA is applied based on the optimal ellipsoid sizes ${({\mathsf{\Xi }}^{\mathit{j}})}^{\ast }$ to calculate the other optimal ellipsoid parameters such as the norm-inducing matrix ${({\mathit{M}}^{\mathit{j}})}^{\ast }$ and the ellipsoid centers ${({\mathit{V}}^{\mathit{j}})}^{\ast }$. Note that the orientations of all the ellipsoids approximating the jth class decision are determined by the norm-inducing matrix ${({\mathit{M}}^{\mathit{j}})}^{\ast }$. The proposed MOPSO integrated with the GKA is illustrated in Figure 2.

4. Determining Number of Ellipsoids

The ellipsoid parameters, such as size, centers, and orientation, are optimized using MOPSO integrated with the GKA, as described in Section 2 and Section 3. These ellipsoid parameters are calculated under the condition that the total number of ellipsoids ${\alpha }^j$ used to approximate the jth class decision region is assigned in advance j = 1…p. If ${\alpha }^j$ is too small, the jth class samples might be included in an insufficient number of ellipsoids, resulting in several ellipsoids having large sizes. It is possible that certain samples that do not belong to the jth class are included in these large ellipsoids. Moreover, samples other than those belonging to the jth class might be included because an insufficient number of ellipsoids is assigned to model the jth class samples. A binary-class dataset with 1000 samples in each class is illustrated in Figure 3. The samples in either class 1 or class 2 are randomly generated within the range [0, 3] on X and Y axis, respectively. The problem caused by a small ${\alpha }^j$ is illustrated in Figure 3a. Conversely, the jth class samples might be included in too many ellipsoids if ${\alpha }^j$ is too large. This results in a scenario where several ellipsoids overlap with each another, resulting in the ellipsoids being learned inefficiently, as illustrated in Figure 3b. If a suitable ${\alpha }^j$ is assigned, as illustrated in Figure 3c, the learning result leads to a set of ellipsoids with of the appropriate size, center, and orientation.

To determine the suitable number of ellipsoids, the ellipsoid parameters are optimized using MOPSO integrated with the GKA by setting ${\alpha }^j$ from 1 to an appropriate number q. Denote ${\left.{({\mathsf{\Xi }}_{}^{\mathit{j}})}^{\ast }\right|}_{{\alpha }_{}^j=i}$ as the optimal ellipsoid sizes calculated using MOPSO, according to (29), with ${\alpha }^j$ set to i ellipsoids, i = 1…q. The index ${\sigma }^j({\left.{({\mathsf{\Xi }}^j)}^{\ast }\right|}_{{\alpha }^j=i})$ in (28) is utilized to evaluate the effectiveness and efficiency for different numbers of ellipsoids ${\alpha }^j$. The suitable number of ellipsoids ${\alpha }^j$ can be determined at the value corresponding to the corner of the curve ${\sigma }^j({\left.{({\mathsf{\Xi }}^j)}^{\ast }\right|}_{{\alpha }^j=i})$ with respect to ${\alpha }^j$. Figure 4 shows a typical curve ${\sigma }^j({\left.{({\mathsf{\Xi }}^j)}^{\ast }\right|}_{{\alpha }^j=i})$ with respect to ${\alpha }^j$. According to this figure, ${\alpha }^j=6$ is a suitable choice because the curve corner appears at ${\alpha }^j=6$.

5. Generating Synthetic Samples

Any set of jth class samples with $(\left|{\mathit{S}}_{\mathit{j}}\right|/N_{\max })<IR$, j = 1…p, is considered a minority set. With reference to (1), ${\tilde{N}}_j$ synthetic samples are to be generated and added into the minority set. Recall that the minority set of the jth class samples is approximated by ${\alpha }^j$ ellipsoids. The ${\tilde{N}}_j$ synthetic samples must be proportionally added into each of the ${\alpha }^j$ ellipsoids based on the density $d_n^j$ of every nth ellipsoid, n = 1…${\alpha }^j$. The number of jth class samples in the nth ellipsoid ${\Phi }_n^j$ is defined in a manner similar to (15) as

$$O_{included}^j({\Phi }_n^j)={\displaystyle \sum _{k=1}^{N}H({\lambda }_{nk}^j\le 1 \mathrm{and} y_k=j)}$$

(30)

The number of samples included in the nth ellipsoid but not belonging to the jth class is

$$O_{included}^{\backslash j}({\Phi }_n^j)={\displaystyle \sum _{k=1}^{N}H({\lambda }_{nk}^j\le 1 \mathrm{and} y_k\ne j)}.$$

(31)

The density of ellipsoid ${\Phi }_n^j$ is defined as

$$d({\Phi }_n^j)=\frac{O_{included}^j({\Phi }_n^j)}{{\xi }_n^j}.$$

(32)

The weight of the ellipsoid ${\Phi }_n^j$ for sharing the generated synthetic samples is defined as the reciprocal of the density $d({\Phi }_n^j)$ modified by the ratio of $O_{included}^j({\Phi }_n^j)$ to the total number samples in ${\Phi }_n^j$:

$${\beta }_n^j=\frac{1}{d({\Phi }_n^j)}\times \frac{O_{included}^j({\Phi }_n^j)}{O_{included}^j({\Phi }_n^j)+O_{included}^{\backslash j}({\Phi }_n^j)}.$$

(33)

Denote the number of synthetic samples added to ${\Phi }_n^j$ as ${\tilde{N}}_n^j$, which is determined based on the weight ${\beta }_n^j$ given in (33)

$${\tilde{N}}_n^j={\tilde{N}}^j\times \frac{{\beta }_n^j}{{\displaystyle \sum _{i=1}^{{\alpha }^j}{\beta }_i^j}}, n=1\dots {\alpha }^j.$$

(34)

The scheme for generating synthetic samples for every ellipsoid ${\Phi }_n^j$ is designed to resolve the oversampling problem for the following two scenarios:

(A) $O_{included}^j({\Phi }_n^j)/(O_{included}^j({\Phi }_n^j)+O_{included}^{\backslash j}({\Phi }_n^j))\ge 0.9$

In this case, more than 90% of the samples in ${\Phi }_n^j$ belong to the minority class. The samples other than those belonging to the jth class can be considered noise. The generated synthetic jth class samples do not affect the classification accuracy if they are randomly included in the ellipsoid ${\Phi }_n^j$. According to (2), ${\Delta }_n^j={({\lambda }_{nk}^j)}^2={({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})}^T{\mathit{M}}_{\mathit{n}}^{\mathit{j}}({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})$. Denote ${\mathit{z}}_{\mathit{ni}}^{\mathit{j}}$ as the ith eigenvector of ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}$ corresponding to the ith eigenvalue ${\psi }_{ni}^j$, i = 1…d. Let ${\mathit{Z}}_{\mathit{n}}^{\mathit{j}}=[{\mathit{z}}_{\mathit{n}1}^{\mathit{j}},\dots ,{\mathit{z}}_{\mathrm{nd}}^{\mathit{j}}]$, ${\mathsf{\Psi }}_{\mathit{n}}^{\mathit{j}}=diag({\psi }_{n1}^j,\dots ,{\psi }_{nd}^j)$. Because ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}$ is a positive-definite matrix,

$${\Delta }_n^j={({\lambda }_{nk}^j)}^2={({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})}^T{\mathit{M}}_{\mathit{n}}^{\mathit{j}}({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}}), n=1\dots {\alpha }^j.$$

(35)

Let $p_{ni}^j={({\mathit{z}}_{\mathit{ni}}^{\mathit{j}})}^T({\mathit{x}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})$, i =1…d; then, (35) can be rewritten as

$${\Delta }_n^j={\displaystyle \sum _{i=1}^d{(p_{ni}^j)}^2}{\psi }_{ni}^j, n=1\dots {\alpha }^j.$$

(36)

Note that the sample ${\mathit{x}}_{\mathit{k}}$ is considered to be included in the ellipsoid if ${\Delta }_n^j\le 1$. The ellipsoid has the center ${\mathit{v}}_{\mathit{n}}^{\mathit{j}}$, and all the eigenvectors ${\mathit{z}}_{\mathit{ni}}^{\mathit{j}}$, i = 1…d, are orthogonal to one another. According to (36), every ith orthogonal eigenvector intersects the ellipsoid’s boundary sphere, where ${\Delta }_n^j=1$ at $1/\sqrt{{\psi }_{ni}^j}$ and $-1/\sqrt{{\psi }_{ni}^j}$.

The synthetic samples ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ are randomly generated in ellipsoid ${\Phi }_n^j$. The generated synthetic sample ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ is expressed as a linear combination of eigenvectors because all eigenvectors ${\mathit{z}}_{\mathit{ni}}^{\mathit{j}}$ are orthogonal axes of the ellipsoid, that is,

$${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}={\displaystyle \sum _{i=1}^d{\mathit{z}}_{\mathit{ni}}^{\mathit{j}}}{b}_{ni},$$

(37)

where $b_{ni}$ is the projection of the vector $({\hat{\mathit{x}}}_{\mathit{k}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})$ onto the eigenvector ${\mathit{z}}_{\mathit{ni}}^{\mathit{j}}$. To ensure random generation of the synthetic samples inside ${\Phi }_n^j$, $b_{ni}$ is set to be a random number within the range.

$$-1/\sqrt{{\psi }_{ni}^j}\le b_{ni}^j\le 1/\sqrt{{\psi }_{ni}^j}$$

(38)

because each of the eigenvectors intersects the boundary sphere at $1/\sqrt{{\psi }_{ni}^j}$ and $-1/\sqrt{{\psi }_{ni}^j}$. However, the approach to randomly generate synthetic samples, as expressed in in (37) and (38), does not guarantee that the generated synthetic sample ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ is always in the ellipsoid ${\Phi }_n^j$. For every randomly generated ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$, calculate the Mahalanobis distance according to (2) as

$${\widehat{\lambda }}_{nk}^j={({({\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}})}^T{\mathit{M}}_{\mathit{n}}^{\mathit{j}}({\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}-{\mathit{v}}_{\mathit{n}}^{\mathit{j}}))}^{1/2}.$$

(39)

The generated ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ is in ${\Phi }_n^j$ if ${\widehat{\lambda }}_{nk}^j\le 1$. No additional processing is required if ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ is in ${\Phi }_n^j$. The generated ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ is outside ${\Phi }_n^j$ if ${\widehat{\lambda }}_{nk}^j>1$. Further processing is required if ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ is outside ${\Phi }_n^j$. The multiplication of ${\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ with a random number $\kappa$, where $0<\kappa \le (1/{\widehat{\lambda }}_{nk}^j)$, leads to the multiplicative product $\kappa {\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$ in ${\Phi }_n^j$. Denote the finally determined synthetic samples as ${\tilde{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}$

$${\tilde{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}=\left\{\begin{array}{c}{\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}, \mathrm{if} {\widehat{\lambda }}_{nk}^j\le 1;\\ {\kappa }_1{\hat{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}, \mathrm{if} {\widehat{\lambda }}_{nk}^j>1;\end{array}\right.,$$

(40)

$\mathrm{where} {\kappa }_1 \mathrm{is} \mathit{a} \mathrm{random} \mathrm{number} \mathrm{and} {\kappa }_1\in (0,1/{\widehat{\lambda }}_{nk}^j].$(B) $O_{included}^j({\Phi }_n^j)/(O_{included}^j({\Phi }_n^j)+O_{included}^{\backslash j}({\Phi }_n^j))<0.9$

In this case, more samples not belonging to the jth class are in ${\Phi }_n^j$. The random placement of synthetic samples in ${\Phi }_n^j$, as in the previous case, cannot effectively improve the classification accuracy. Borderline SMOTE [30] is modified to generate synthetic samples in this case. The samples located at the borderline between the clusters belonging and not belonging to the jth class must be first identified using Borderline SMOTE. For every jth class sample ${\mathit{x}}_{\mathit{k}}^{\mathit{j}}\in {\Phi }_n^j$, define the set ${\mathit{S}}_{\mathit{k}}^{\mathit{j}}$ containing all m-nearest neighbors. The m-nearest neighbors of ${\mathit{x}}_{\mathit{k}}^{\mathit{j}}$ are defined as the samples with the m-shortest Mahalanobis distances from ${\mathit{x}}_{\mathit{k}}^{\mathit{j}}$. Note that the Mahalanobis distance is calculated using the same norm-inducing matrix ${\mathit{M}}_{\mathit{n}}^{\mathit{j}}$ as in the case of ${\Phi }_n^j$, that is,

$${\tilde{\lambda }}_{ki}^j={({({\mathit{x}}_{\mathit{k}}^{\mathit{j}}-{\mathit{x}}_{\mathit{i}})}^T{\mathit{M}}_{\mathit{n}}^{\mathit{j}}({\mathit{x}}_{\mathit{k}}^{\mathit{j}}-{\mathit{x}}_{\mathit{i}}))}^{1/2}, \forall {\mathit{x}}_{\mathit{i}}\in {\Phi }_n^j, \mathrm{but} {\mathit{x}}_{\mathit{i}}\ne {\mathit{x}}_{\mathit{k}}^{\mathit{j}}.$$

(41)

The sample with all the m-nearest neighbors belonging to jth class is the sample not on the borderline. Conversely, the borderline sample contains at least one sample among the m-nearest neighbors not belonging to the jth class. Therefore, a sample is a borderline sample if at least one sample in the set of m-nearest neighbors ${\mathit{S}}_{\mathit{k}}^{\mathit{j}}$ does not belong to the jth class for every ${\mathit{x}}_{\mathit{k}}^{\mathit{j}}\in {\Phi }_n^j$.

After the borderline samples are identified, the synthetic samples are generated through random interpolation between the borderline sample ${\mathit{x}}_{\mathit{k}}^{\mathit{j}}$ and any other ${\mathit{x}}_{\mathit{l}}^{\mathit{j}}\in {\mathit{S}}_{\mathit{k}}^{\mathit{j}}$; that is, the synthetic sample is generated as follows:

$${\tilde{\mathit{x}}}_{\mathit{k}}^{\mathit{j}}={\mathit{x}}_{\mathit{k}}^{\mathit{j}}+{\kappa }_2({\mathit{x}}_{\mathit{k}}^{\mathit{j}}-{\mathit{x}}_{\mathit{l}}^{\mathit{j}}), \forall {\mathit{x}}_{\mathit{l}}^{\mathit{j}}\in {\mathit{S}}_{\mathit{k}}^{\mathit{j}},$$

(42)

where ${\kappa }_2$ is a random number and ${\kappa }_2\in [0,1]$.

6. Simulation

The proposed EMDO was evaluated against other multi-class imbalanced data learning algorithms on different numerical data sets. The classifier C4.5 is usually utilized as the classifier to verify the oversampling results for various oversampling approaches. For instance, the oversampling approaches in [30,32,39], and [40,41,42,43] all used C4.5 as the classifier to verify the proposed oversampling schemes. This is mainly due to the fact that the classification results with C4.5 do not change as long as the parameter setting and datasets are fixed. No randomness exists in the classification results with C4.5 for the same parameter setting and dataset. Note that the number of ellipsoids utilized for the minority class is determined using the scheme proposed in Section 4 and is listed in the rightmost column of

Table 1. Characteristics of the data sets for simulations in Example 1.

Data Set	Size	Attributes	Classes	Class Distribution	IRmax	No. of Ellipsoids
Balance	625	4	3	288/49/288	5.88	NA/4/NA
Hayes-Roth	132	4	3	51/51/30	1.7	NA/NA/3
New-Thyroid	215	5	3	150/35/30	5	NA/3/3
Page-Blocks	5472	10	5	4913/329/28/87/115	175.46	NA/9/4/6/9
Dermatology	358	34	6	111/60/71/48/48/20	5.55	NA/4/4/3/3/3
Breast-Tissue	106	9	6	21/15/18/16/14/22	1.57	NA/NA/NA/ NA/4/NA
User-Knowledge-Modelling (UKM)	403	5	5	50/102/129/122	2.58	4/NA/NA/NA
Vertebral-Column	310	6	3	60/150/100	2.5	6/NA/NA
Ecoli	327	7	5	143/77/52/35/20	7.15	NA/6/4/3/3

NA: not available.

and Table 6. The five nearest neighbors are considered for synthetic sample generation in case (B) of Section 5.

The simulations in this study were conducted using five-fold cross-validation with 10 independent runs. Every data set was tested using different oversampling schemes and compared with the proposed EMDO. The minority class with the minimum size was selected to validate the effectiveness and efficiency of the proposed EMDO.

Several evaluation metrics were designed to evaluate the effectiveness and efficiency of the proposed EMDO. The classification accuracy for the jth class is defined as follows:

$$P_j=\frac{T{P}_j}{T{P}_j+F{P}_j},$$

(43)

where $T{P}_j$ is the number of true-positive classified samples, that is, the samples that are correctly classified as belonging to the jth class. $F{P}_j$ is the number of false-positive classified samples, that is, the samples that are incorrectly classified as belonging to the jth class. The metric $P_{avg}$ is defined as the average classification accuracy over all p classes, that is,

$$P_{avg}=\frac{1}{p}{\displaystyle \sum _{j=1}^p{P}_j}.$$

(44)

The metric $P_{\min }$ refers to the classification accuracy defined in (43) for the minority class with the minimum size. To measure the capability of EMDO to separate any pair of classes, the area under curve (AUC) [54,55] is widely used in [56,57]. Denote $A_{m,n}$ as the AUC between class m and class n. The metric AUCm is defined as follows for measuring the capability of EMDO to separate the smallest minority class with the minimum size from the other classes.

$$\mathrm{AUCm}=\frac{1}{p-1}{\displaystyle \sum _{n\ne n’}\frac{A_{n,n’}+A_{n’,n}}{2}}$$

(45)

where n’ denotes the minority class with the minimum size. In addition to AUCm, the average of AUC over all pairs of classes for a multi-class problem, denoted as MAUC, is defined as

$$\mathrm{MAUC}=\frac{2}{p(p-1)}{\displaystyle \sum _{m<n}\frac{A_{m,n}+A_{n,m}}{2}}.$$

(46)

In order to evaluate the imbalance condition of every data set, the maximum imbalance ratio IRmax is defined as follows:

$$I{R}_{\max }=\frac{N_{\max }}{N_{\min }}$$

(47)

where $N_{\min }=\underset{j=1\dots p}{\min }(N_j)$, $N_{\max }=\underset{j=1\dots p}{\max }(N_j)$.

Example 1:

The data sets used in the simulation are the same as those used in [43] for comparing the performance of the EMDO against AMDO and other learning algorithms. The data sets used in [43] were mainly from data repositories such as the ones Knowledge Extraction based on Evolutionary Learning (KEEL) [58] and UCI (University of California, Irvine) Machine Learning Repository [59]. Table 1 describes these data sets. The performance comparison based on different indices are made in

Table 2. Comparison of $P_{\min }$ (%) for every over sampling scheme on different data sets.

Data Set	Baseline	SSMOTE	GCS	ABNC	OSMOTE	MDO	MDO+	AMDO	EMDO
Balance	0.000.00	8.449.18	6.449.83	2.224.97	12.4413.41	2.004.47	0.000.00	10.220.50	20.4110.95
Hayes-Roth	100.000.00	100.000.00	100.000.00	100.000.00	100.000.00	100.000.00	100.000.00	100.000.00	100.000.00
New-Thyroid	83.3311.79	90.0014.91	93.339.13	93.339.13	90.009.13	83.3316.67	96.677.45	100.000.00	100.000.00
Page-Blocks	82.6711.88	78.676.91	93.3314.91	72.6724.99	93.339.13	75.3316.26	93.339.13	96.677.45	96.675.33
Dermatology	95.0011.18	95.0011.18	90.0013.69	100.000.00	90.0013.69	95.0011.18	95.0011.18	100.000.00	100.000.00
Breast-Tissue	60.0027.89	40.0036.51	46.6729.81	60.0027.89	53.3329.81	60.0027.89	53.3329.81	60.0027.89	73.3613.32
UKM	88.0013.04	92.0013.04	90.0010.00	94.008.94	88.0016.43	86.0013.42	86.0013.42	94.008.94	96.005.76
Vertebral-Column	65.0016.03	65.0019.00	60.0019.90	61.6718.26	66.675.89	68.3322.36	66.6728.87	86.679.50	90.676.02
Ecoli	65.0037.91	70.0027.39	55.0032.60	70.0027.39	55.0032.60	75.0030.62	90.0013.69	90.0013.69	90.0012.40
Average	71.00	71.01	70.53	72.65	72.09	71.67	75.67	81.95	85.23
Rank Avg.	6.33	5.89	6.39	5.11	5.78	5.89	5.28	2.56	1.78

The best result is in bold face.

,

Table 3. Comparison of $P_{avg}$ (%) for every oversampling scheme on different data sets.

Data Set	Baseline	SSMOTE	GCS	ABNC	OSMOTE	MDO	MDO+	AMDO	EMDO
Balance	56.381.75	58.382.94	56.654.03	62.333.29	57.862.78	57.282.92	55.451.66	60.372.06	64.383.59
Hayes-Roth	84.916.71	84.675.06	85.335.58	83.527.08	84.976.74	84.916.71	84.976.74	84.976.74	85.612.39
New-Thyroid	88.866.02	91.083.02	93.144.55	93.591.45	92.546.61	89.816.84	94.983.59	96.542.99	96.602.05
Page-Blocks	84.302.14	84.571.88	88.774.49	79.705.57	89.612.98	81.241.79	86.132.48	88.771.92	90.151.87
Dermatology	95.672.05	95.731.83	93.502.71	97.100.75	95.312.45	95.672.05	96.061.62	96.880.26	97.130.20
Breast-Tissue	63.223.74	60.893.94	68.785.77	66.004.88	65.837.05	63.223.74	66.562.17	63.223.74	70.606.28
UKM	92.182.02	92.574.85	91.032.00	94.492.45	91.782.32	91.452.48	91.922.50	94.232.14	95.241.37
Vertebral-Column	76.442.65	77.224.66	75.675.86	76.673.12	77.004.71	78.225.55	76.567.41	81.892.38	85.771.53
Ecoli	74.647.88	72.8112.01	72.7311.35	76.237.14	73.128.72	77.247.03	82.305.21	82.445.08	85.311.62
Average	79.61	79.77	80.62	81.07	80.89	79.89	81.66	83.26	85.65
Rank Avg.	7.00	6.11	6.17	4.78	5.44	6.33	4.89	3.28	1.00

The best result is in bold face.

,

Table 4. Comparison of AUCm (%) for every oversampling scheme on different data sets.

Data Set	Base	SSMOTE	GCS	ABNC	OSMOTE	MDO	MDO+	AMDO	EMDO
Balance	56.951.91	58.123.27	57.103.43	60.522.89	58.583.18	57.402.78	56.600.88	60.611.24	65.614.30
Hayes-Roth	94.342.52	94.251.90	94.502.09	93.822.65	94.362.53	94.342.52	94.362.53	94.362.53	94.672.81
New-Thyroid	91.405.33	93.904.45	95.433.76	95.762.37	94.045.11	91.856.74	97.042.94	98.371.41	98.551.23
Page-Blocks	90.753.57	89.842.16	94.814.66	86.827.87	94.963.16	87.904.66	93.972.91	95.512.07	96.072.24
Dermatology	97.323.46	97.393.27	95.554.17	99.050.28	96.094.09	97.323.46	97.483.23	98.980.26	99.060.22
Breast-Tissue	76.807.31	72.1810.10	75.809.99	78.308.92	76.929.75	76.807.31	77.308.09	76.807.31	82.373.77
UKM	94.143.65	94.945.27	94.193.04	96.412.61	94.014.64	93.333.88	93.553.92	96.453.02	97.262.49
Vertebral-Column	79.253.17	79.965.56	78.386.36	79.425.04	80.042.48	81.136.76	79.548.76	86.041.72	89.072.00
Ecoli	83.0711.54	83.4310.88	79.7811.44	84.789.21	79.7411.00	86.389.56	91.974.61	91.594.19	92.912.92
Average	84.89	84.89	85.06	86.10	85.42	85.16	86.87	88.75	90.62
Rank Avg.	7.00	6.22	6.33	4.89	5.44	6.33	4.89	2.89	1.00

The best result is in bold face.

and

Table 5. Comparison of MAUC (%) for every over sampling scheme on different data sets.

Data Set	Baseline	SSMOTE	GCS	ABNC	OSMOTE	MDO	MDO+	AMDO	EMDO
Balance	67.291.31	68.792.20	67.493.02	71.752.47	68.392.09	67.962.19	66.591.25	70.271.54	73.132.77
Hayes-Roth	88.685.03	88.503.79	89.004.18	87.645.31	88.735.06	88.685.03	88.735.06	88.735.06	89.154.86
New-Thyroid	91.644.52	93.312.27	94.863.41	95.191.09	94.404.96	92.365.13	96.242.69	97.402.24	97.762.14
Page-Blocks	90.191.34	90.351.18	92.982.80	87.313.48	93.511.86	88.271.12	91.331.55	92.981.20	93.851.17
Dermatology	97.401.23	97.441.10	96.101.62	98.260.45	97.191.47	97.401.23	97.630.97	98.130.16	98.260.16
Breast-Tissue	77.932.24	76.532.36	81.273.46	79.602.93	79.504.23	77.932.24	79.931.30	77.932.24	84.075.16
UKM	94.791.34	95.053.23	94.021.33	96.321.64	94.521.54	94.301.65	94.611.67	96.151.42	96.911.05
Vertebral-Column	82.331.99	82.923.50	81.754.39	82.502.34	82.753.53	83.674.16	82.425.55	86.421.78	89.051.33
Ecoli	84.154.93	83.017.50	82.967.10	85.154.46	83.205.45	85.774.39	88.933.26	89.033.18	90.821.01
Average	86.04	86.21	86.71	87.08	86.91	86.26	87.38	88.56	90.33
Rank Avg.	7.00	6.11	6.17	4.72	5.44	6.33	4.89	3.28	1.06

The best result is in bold face.

. Within Table 2, Table 3, Table 4 and Table 5, the algorithms such as SSMOTE refers to Static-SMOTE [60], GCS refers to RESCALE [61], ABNC refers to AdaBoost.NC [62], and OSMOTE refers to OVOSMOTE [63]. The MDO in [42], MDO+ and AMDO in [43] are also compared in Table 2, Table 3, Table 4 and Table 5 with the proposed EMDO. The Baseline algorithm is the classifier C4.5 without any oversampling technique.

To compare the performance of EMDO with those of the other schemes, the rank average of every scheme was calculated. All oversampling schemes were tested on each of the data sets listed in Table 1. The ranking of algorithm performance was based on each of the metrics. For instance, the algorithm with the best performance is ranked first, the algorithm with the second-to-best performance is ranked second, etc. The average rank of every algorithm is then calculated. The schemes with the same metric values share ranks. For instance, if two schemes are ranked second because they have the same metric values, the two schemes share the second and third ranks. These two schemes are thus ranked 2.5. The means and standard deviations of $P_{\min }$ and $P_{avg}$ are listed in Table 2 and Table 3, respectively, for every oversampling scheme, including the proposed EMDO, applied to different data sets. According to Table 2 and Table 3, EMDO outperforms all of the other schemes on every data set. EMDO has the lowest average rank. The results shown in both Table 2 and Table 3 imply that the oversampling performed using EMDO significantly improves the classification accuracy for the smallest minority class. Moreover, the synthetic samples generated for the minority class samples improve the overall average classification accuracy.

For all schemes, the mean and standard deviation are listed in Table 4 and the AUCm defined in (45) and MAUCm defined in (46) are also compared in Table 4 and Table 5, respectively. As indicated in Table 4, EMDO outperform the other schemes in separating the smallest minority class from the other classes for every listed data set. Moreover, according to Table 5, the capability of EMDO to separate all pairs of classes in the multi-class problem is superior to that of the other schemes.

Example 2:

The performance of EMDO is evaluated on the sensory data in this example. Two data sets, Statlog (Shuttle) from UCI and Mafalda [64] from Github are utilized in this example. The data set Statlog (Shuttle) is the set of the recorded sensory data from NASA’s space shuttle while the data set Mafalda is the set of the recorded sensory data from different brands of cars. The characteristics of these two data sets is shown in

Table 6. Characteristics of the data sets for simulations in Example 2.

Data Set	Size	Attributes	Classes	Class Distribution	IRmax	No. of Ellipsoids
Statlog (Shuttle)	58000	9	7	45586/50/171/8903/ 3267/10/13	5684.67	NA/3/4/5/5/ 3/1
Mafalda	23762	14	3	17757/2990/3015	5.94	NA/11/11

NA: not available.

. Table 6 shows that both data sets are extremely imbalanced because the maximum imbalance ratios IR_max are as high as 5684.7 and 5.94, respectively. Four indices $P_{\min }$, $P_{avg}$, AUCm, and MAUC are calculated and compared in

Table 7. Comparison of the performance with and without EMDO for imbalanced sensory data.

Data Set		Pmin	Pavg	AUCm	MAUC
Statlog (Shuttle)	w/ EMDO	89.3313.73	96.601.95	99.660.47	99.320.94
	w/o EMDO	6048.99	93.217.42	77.656.17	92.801.66
Mafalda	w/ EMDO	57.3818.33	72.349.33	76.399.31	78.198.22
	w/o EMDO	33.1712.13	60.7710.53	65.257.88	67.806.99

for both data sets with classifier C4.5. The classification results are greatly improved with oversampling scheme EMDO compared with the results without EMDO. EMDO helps improve classification results for both highly imbalanced data sets according to the four evaluation indices listed in Table 7.

7. Conclusions

EMDO was demonstrated to outperform competing oversampling approaches in simulations. EMDO performed well because it approximates the decision region of the target minority class with reasonable accuracy by using a set of ellipsoids. In problems involving multi-class imbalanced data, EMDO performs exceptionally well if the decision region of the minority class is separated in the feature space. EMDO can learn the sizes, centers, and orientations of the ellipsoids that approximate the minority class decision region by using the underlying distribution of minority class samples. IoT is a key emerging technology, and imbalanced data will become an increasingly common problem as the number of IoT sensors increases. The proposed EMDO is suitable for solving such multi-class imbalanced data classification problems. One of the future works related to this study involves applying EMDO to address the problem of imbalanced data encountered in real-world IoT sensing data. Although EMDO is a data-level learning approach, it can easily be integrated with other cost-sensitive methods to increase the effectiveness and efficiency of learning. Further studies on variants of integration can be another direction for future research.