SMOTE-ENC: A Novel SMOTE-Based Method to Generate Synthetic Data for Nominal and Continuous Features

Abstract

Real-world datasets are heavily skewed where some classes are significantly outnumbered by the other classes. In these situations, machine learning algorithms fail to achieve substantial efficacy while predicting these underrepresented instances. To solve this problem, many variations of synthetic minority oversampling methods (SMOTE) have been proposed to balance datasets which deal with continuous features. However, for datasets with both nominal and continuous features, SMOTE-NC is the only SMOTE-based oversampling technique to balance the data. In this paper, we present a novel minority oversampling method, SMOTE-ENC (SMOTE—Encoded Nominal and Continuous), in which nominal features are encoded as numeric values and the difference between two such numeric values reflects the amount of change of association with the minority class. Our experiments show that classification models using the SMOTE-ENC method offer better prediction than models using SMOTE-NC when the dataset has a substantial number of nominal features and also when there is some association between the categorical features and the target class. Additionally, our proposed method addressed one of the major limitations of the SMOTE-NC algorithm. SMOTE-NC can be applied only on mixed datasets that have features consisting of both continuous and nominal features and cannot function if all the features of the dataset are nominal. Our novel method has been generalized to be applied to both mixed datasets and nominal-only datasets.

1. Introduction

Class imbalance is one of the major obstacles in classification problems to achieve high accuracy for minority class data points. In the real world, datasets are very often heavily skewed; e.g., in medical datasets [1] for cancer detection, benign tumors significantly outnumber cancerous tumors [2,3,4,5,6,7], while in financial datasets [8] legitimate transactions outnumber fraudulent transactions [9]. When statistical models are constructed on these extremely skewed datasets, the model tends to predict every instance as a member of the overrepresented class, making the model inefficient when used to identify underrepresented class instances, which is especially critical when classifying minority instances is the point of interest.

In order to solve this problem, in machine learning algorithms, different approaches are adopted which broadly can be divided into three categories: tuning cost function [10], sampling [11] and modifying learning algorithm [12]. Among these, sampling techniques are one of the widely accepted methods to balance the skewed dataset. Two of the earliest proposed sampling methods were random undersampling [13] and random oversampling: the former samples majority class data points to form a representative set of similar size to the underrepresented class and discards the rest while the latter replicates some of the minority class data points to balance the dataset. However, by adding additional copies to some minority class instances, random oversampling leads to overfitting and by removing samples removes potentially valuable data [14], while random undersampling causes underfitting. Hence, both of them exhibit poor generalization of unseen data.

As an alternative, in 2002, Chawla et al. [15] proposed a novel oversampling method, SMOTE, where instead of replicating existing instances, new instances were synthesized to balance the dataset. In SMOTE, for each underrepresented instance, a predetermined number of neighbors were calculated, then some minority class instances were randomly chosen for synthetic data point creation. Finally, artificial observations were created along the line between the selected minority instance and its closest neighbors. Chawla et al. experimented with SMOTE on a wide range of datasets which showed that SMOTE can improve classifier performance for minority class. By virtue of being trained on more underrepresented examples, SMOTE decreased generalization errors.

Since then, different variations of SMOTE have been proposed [16,17,18,19,20,21]. Based on the SMOTE method, Han et al., proposed BorderLine-SMOTE, [22] where minority instances close to the class boundary were synthesized. In 2004, Batista et al. [11] proposed SMOTE-ENN and SMOTE-Tomek, which considered distribution overlap and class boundaries while generating synthetic data. In 2008, He et al., proposed ADASYN [23], which focused on difficult minority class instances and simulated synthetic data by understanding pattern of those instances. In 2016, Torres et al., proposed a variant of SMOTE, SMOTE-D [24], which, unlike SMOTE, deterministically synthesized artificial data points for underrepresented classes. All of these methods were developed considering continuous variables and when applied on datasets with categorical features, failed to identify nominal features, which resulted in creating new labels for these attributes.

To solve this problem, Chawla et al., proposed a method called SMOTE-NC [15] (SMOTE-Nominal Continuous), which treated nominal attributes differently than continuous attributes and preserved the original labels of categorical features in the resampled data. However, for multi-label nominal features, SMOTE-NC failed to interpret the difference in association between each label and minority class target. Additionally, the SMOTE-NC method can only function when the dataset has at least one continuous attribute.

In this paper, we propose a novel algorithm, SMOTE-ENC, which, irrespective of the presence of continuous features, will comprehend that different labels of nominal features have a different affinity towards minority classes. We compare this new algorithm with SMOTE-NC and our experiments indicate that this new method performs better than SMOTE-NC for datasets with numerous multi-label nominal features.

2. Materials and Methods

Since the SMOTE method is based on the k-nearest neighbor algorithm, distance calculation between two instances is the most important aspect of generating synthetic samples. In the feature space, deriving the distance between continuous variables is straightforward; however, for categorical variables, calculating the distance between two labels can be complex. In SMOTE-NC, if a label of a categorical attribute differs between an instance and its nearest neighbors, then a fixed-value, i.e., median of standard deviations of continuous features, is added in the distance calculation. Hence, in this method, for a multi-label categorical feature, the distance between any two labels is the same.

Now, let us consider a dataset where the target is to identify a potential buyer of a life insurance plan. Suppose the dataset has a nominal feature, marital status, with three labels—single, married and divorced—and the following is the percentage distribution of people who bought life insurance plan: single → 18%, divorced → 22% and married → 60%. In SMOTE-NC, the distance between married–divorced and divorced–single is considered to be the same. However, based on the data, it seems people who are divorced or single are less interested in buying life insurance than married people. So, considering the similar affinity towards the target, single and divorced labels should be closer to each other than to the married label; i.e., the distance between married–divorced should be more than that between single–divorced. In SMOTE-ENC, labels of a categorical variable are represented by their affinity of association with the minority class target.

We explain our algorithm (Algorithm 1) using a sample training dataset (

Table 1. Sample dataset to explain our algorithm (SMOTE-ENC).

	C1	C2	C3	N	Target
Instance 1 (i1)	100	20	85	a	min
Instance 2 (i2)	200	65	54	b	min
Instance 3 (i3)	166	24	38	a	maj
Instance 4 (i4)	344	67	89	b	maj
Instance 5 (i5)	200	30	75	b	maj

) with three continuous features (C1, C2, C3) and a categorical feature (N). The dataset is imbalanced where we have 2 instances of minority class labelled as min and 3 instances of majority class labelled as maj.

Algorithm 1. SMOTE-ENC algorithm

Input: t = number of minority class samples in training set; n% = amount of oversampling; k = number of nearest neighbors to be considered, s = total number of instances in training set, c = number of continous variables in the dataset, m = median of standard deviation of continuous features when c > 0 ir = t/s for each categorical feature, do   for each “l” in distinct labels, do     e = total number of “l” labeled instances in training set     e′ = $e \ast ir$;     o = number of “l” labeled minority class instances in training set     χ = $\frac{\left(\mathrm{o}-\mathrm{e}\prime \right)}{\mathrm{e}\prime }$;     if c > 0,       l = χ * m;     else,       l = χ;   end end Apply SMOTE (t, n, k) For the synthetic data points, the categorical attribute’s value is decided as the value in the majority of its nearest neighbors. Inverse-encode categorical values to their original labels.

In SMOTE-NC, the distance between i1–i2 is calculated as follows:

The standard deviations of the three continuous features (C1, C2, C3) associated with minority class min, are 70.71, 31.82 and 21.92 respectively, and the median m, of these numbers is 31.82. The distance between the two instances i1 and i2 are calculated as:

$${\mathrm{Distance} }_{\mathrm{SMOTE-NC} (\mathrm{i}2 - \mathrm{i}1)} = \sqrt{{\left(100-200\right)}^2+{\left(20-65\right)}^2+{\left(85-54\right)}^2+{\left(31.82\right)}^2}=118.31$$

In our algorithm (SMOTE-ENC), we do the encoding of nominal labels as follows (alphabetical notations are same as Algorithm 1):

Imbalance ratio (ir) = 2/5 = 0.4.

Total ‘a’ label in our training dataset (e) = 2.

Total ‘a’ label associated with minority class (min) o = 1.

If ‘a’ label had equal affinity towards min and maj class, then the total number of ‘a’ labeled instances in minority samples would be $\mathrm{e}\prime =\mathrm{e} \ast \mathrm{ir}$ = 2 × 0.4 = 0.8.

Encoding of ‘a’ for all target min = $\frac{\left(\mathrm{o}-\mathrm{e}\prime \right)}{\mathrm{e}\prime }\ast m =\frac{\left(1-0.8\right)}{0.8} \times 31.82=7.95$.

Total ‘b’ label in our training dataset (e) = 3.

Total ‘b’ label associated with minority class (min) o = 1.

If ‘b’ label had equal affinity towards min and maj class, then total number of ‘b’ labeled instances in minority samples would be $\mathrm{e}\prime =\mathrm{e} \ast \mathrm{ir}$ = 3 × 0.4 = 1.2.

Encoding of ‘b’ for all target min = $\frac{\left(\mathrm{o}-\mathrm{e}\prime \right)}{\mathrm{e}\prime }\ast m =\frac{\left(1-1.2\right)}{1.2} \times 31.82$ = −5.30.

$${\mathrm{Distance} }_{\mathrm{SMOTE-ENC} (\mathrm{i}2 - \mathrm{i}1) }= \sqrt{{\left(100-200\right)}^2+{\left(20-65\right)}^2+{\left(85-54\right)}^2+{\left(7.95+5.30\right)}^2}=114.72$$

We can see that the distance between two data rows in our sample dataset is reduced when calculated by SMOTE-ENC. Hence, it can be said that, in SMOTE-NC, inter and intra level distance of categorical levels are always the same; whereas in SMOTE-ENC, nominal features are encoded as numeric values and the difference between two such numeric value reflects the amount of change of association with minority classes and hence seems to produce better result than SMOTE-NC.

By applying this algorithm on the nominal features, we can encode each label as a numeric value where a higher value of a label indicates stronger association with a minority class. This encoding is inspired by Pearson’s chi-squared test [25], which can confirm if two categorical variables are related. However, in this method, our objective is not to determine if the nominal features of the samples are related to the target, but rather to quantify the distance between two data points when the label of a categorical variable is changed. Hence, we do not use the chi-squared test, i.e., χ2, but rather use chi (χ) for each label of a nominal feature to measure the amount of association of that label with the minority class target. The magnitude of χ (i.e., chi) is a fraction and when included in the distance calculation between two samples can become insignificant as the difference between two values of a continuous feature is often significantly higher. For this reason, if the dataset has continuous features, χ is multiplied by the median of the standard deviation of the continuous variable to make the value comparable to other continuous attributes. However, if the dataset has only nominal features, chi (χ) is directly used in distance calculation.

We compared our proposed method, SMOTE-ENC, with SMOTE-NC on five different datasets. These datasets have diverse minority to majority ratio, which offers a broad range for testing this algorithm’s efficiency. Among these datasets, forest cover data had 7 classes. We evaluated our model on a pair of classes of the original dataset and discarded the rest of the instances.

The metrics used in our study to evaluate the models’ performance were precision, recall, F-beta score, area under precision recall curve and area under ROC curve. Accuracy was not used as an evaluation metric in this paper, as accuracy is not able to capture model performance for a highly imbalanced dataset. F-beta score is the weighted average of precision (ratio of correctly predicted positive observations to the total predicted positive observations) and recall (ratio correctly predicted positive observations to the total positive observations).

$$F_{\beta }= \left(1+ {\beta }^2\right) \ast \frac{precision\ast recall}{\left({\beta }^2\ast precision\right)+recall}$$

(1)

Choice of β signifies the weightage of precision and recall while 0 < β < 1 assigns more weightage to precision, 1 < β < +∞ assigns more weightage to recall. β = 1 assigns equal weightage to precision and recall and is known as the F1-score. ROC is a probability curve and the area under the ROC curve represents the degree of separability. It identifies how well the model can differentiate between classes as it balances between precision and recall. As the AUC gets higher, the model gives better prediction. This curve is plotted with true positive rate/recall against the false positive rate. However, the ROC plot can be misleading when applied to heavily imbalanced datasets as adding a lot of majority class observations can improve the AUC significantly without any improvement in precision and recall of minority class instances. For imbalanced class problems, the precision–recall curve offers better insight [26]. PRC shows the relationship between precision and recall, and its baseline moves with class distribution; e.g., for a dataset with imbalance ratio 1:10, the baseline moves to 0.09, i.e., 1/(1 + 10).

Stratified cross-validation was performed during oversampling to resolve the overoptimization problem [27]. Random forest was used as the classification algorithm on the resampled data. We created a combined estimator by using the pipeline to be applied on the data, which consisted of the oversampling method and the random forest classifier.

In this study, we used a two-tailed t-test to show that the improved results obtained using our proposed oversampling method and the model using existing oversampling method were statistically significant. The statistical test performed on precision, recall and f1-score results. The p-value of 0.05 has been used as a threshold for the statistical significance.

3. Results

We compared SMOTE-ENC’s performance with SMOTE-NC’s precision, recall, F1-score, and area under PRC and ROC on five different datasets. The performance of each one is discussed below in

Table 2. Comparison between SMOTE-ENC (new method) and SMOTE-NC on minority class data points at threshold 0.5.

Dataset	IR 1	N 2	C 3	Sampling Method	Precision	Recall	F1-Score
Banking telemarketing dataset [28]	9:1	12	5	SMOTE-ENC	31.95	46.84	37.99
				SMOTE-NC	27.33	50.40	35.45
Credit card dataset [29]	12:1	5	14	SMOTE-ENC	65.61	81.75	72.79
				SMOTE-NC	59.78	84.92	70.16
Car dataset [30]	3:1	6	0	SMOTE-ENC	87.91	79.21	83.33
				SMOTE-NC	NA 4	NA4	NA 4
Forest cover dataset (class 2 and 6) [31]	17:1	2	12	SMOTE-ENC	74.76	99.84	85.50
				SMOTE-NC	74.76	99.82	85.49
Rain in Australia dataset [32]	4:1	3	13	SMOTE-ENC	58.45	72.24	64.62
				SMOTE-NC	58.30	72.51	64.63

¹ IR (imbalance ratio) = total number of samples in training set/number of minority class samples. ² N = number of nominal features in the dataset. ³ C = number of continuous features in the dataset. ⁴ NA = SMOTE-NC cannot be applied to this dataset.

.

3.1. Evaluation on Banking Telemarketing Dataset

This is a publicly available dataset to predict the success of telemarketing.

From Table 2, it is observed that, for the bank dataset, SMOTE-ENC achieved a significantly greater improvement in precision and F1-score at threshold 0.5 than SMOTE-NC on minority class instances. We further compared these two methods using ROC and precision–recall curve. Figure 1 demonstrates that SMOTE-ENC was a better sampling method than SMOTE-NC for the banking dataset when identifying minority class instances was the focus of interest. We further looked at the statistical significance of our improved results and identified that the p-value from the two tailed t-test was less than 0.002 for precision, recall and f1-score. This shows that our improved performance was statistically significant. Therefore, it can be said SMOTE-ENC has been able to interpret nominal features more accurately while generating synthetic data points. We further looked at whether SMOTE-ENC had any effect on the change of importance of features (Figure 2).

In the banking dataset, euribor3m, cons.price.idx, cons.conf.idx, age and campaign were continuous features. Figure 2 shows that the top three important features were continuous variables, which remained unchanged, contributing >0.55 importance towards classification. A slight shift in the importance of low-ranked features was observed.

3.2. Evaluation on Credit Card Dataset

This is a publicly available dataset to predict churning customers.

Table 2 demonstrates that SMOTE-ENC yielded significantly better results than SMOTE-NC in terms of precision and F1-score at threshold 0.5 on minority class instances for this dataset, and Figure 3 shows that SMOTE-ENC attained higher ROC and PRC scores than SMOTE-NC. The difference between these two models’ performances was statistically significant as the p-value from the two tailed t-test was less than 0.004 for precision, recall and f1-score. Hence, it can be said that the new SMOTE-ENC method was a better sampling method than SMOTE-NC for this credit card dataset when identifying minority class instances was the point of interest. We further looked at whether SMOTE-ENC had any effect on the change of importance of features (Figure 4).

From Figure 4, it can be observed that none of the five nominal features of this dataset came up in the top 10 feature importance ranking using the random forest classifier. Application of our method, SMOTE-ENC, did not change this ranking showing that our SMOTE-ENC improved performance by interpreting nominal features in a different way; however, it did not diminish the continuous features’ contribution in model prediction.

3.3. Evaluation on Car Dataset

This is a publicly available dataset to evaluate cars.

In SMOTE-NC, the distance between two levels of a categorical feature is the median of standard deviations of all continuous features for the minority class. So, and in order for SMOTE-NC to work, datasets need to have at least one continuous feature.

The car dataset used in this study did not have any continuous features to predict the target outcome. All of its attributes were multi-level categorical features. For this reason, SMOTE-NC could not handle this dataset.

However, the new SMOTE-ENC method could be applied to this dataset because, in our SMOTE-ENC algorithm, inter-level distance of a nominal feature is not dependent on a continuous feature’s presence and hence has been generalized to handle both mixed datasets (i.e., datasets containing nominal and continuous features) and nominal-only datasets.

Table 2 shows that SMOTE-ENC yielded an F1-score of 83.33 at threshold 0.5 on minority class instances for this dataset, and from Figure 5 it can be observed that SMOTE-ENC produced high ROC and PRC scores as well.

3.4. Evaluation on Forest Cover Dataset with Class 2 and 6

This is a publicly available dataset to predict forest cover type.

For the forest cover dataset with class 2 and 6, it was observed from Table 2 that, while precision was consistent for both of the sampling techniques, using SMOTE-ENC increased recall of the minority class observations, which resulted in marginal improvement of the F1-score for SMOTE-ENC at threshold 0.5.

The precision–recall curve (Figure 6) showed that SMOTE-ENC was marginally better than SMOTE-NC for this dataset. However, the area under ROC and PR curve being high implied that the classes were easily separable, which makes the importance of sampling method less prominent. For this reason, we one-hot encoded the categorical features and applied SMOTE to resample the data. In this dataset, we had two categorical features, soil type and wilderness_area. While the former contained 40 labels, the latter had four labels.

From Figure 7, it was observed that one-hot encoding of multi-label features decreased the area under ROC and PR curve. Thus, the importance of interpreting nominal features in a different way than interpreting continuous features in over-sampling method can be ascertained. We further showed that the importance of categorical features (soil type and wilderness_area) was picked up by SMOTE-ENC and SMOTE-NC but not by SMOTE (Figure 8).

3.5. Evaluation on Rain Dataset

This is a publicly available dataset to predict occurrence of rainfall on the next day based on today’s weather attributes.

SMOTE-ENC provided better precision while reducing recall (Table 2). The magnitude of reduction in recall being higher than the magnitude of improvement in precision made the harmonic mean of precision and recall—the F1 score—reduced.

ROC and precision–recall curves (Figure 9) showed that SMOTE-NC yielded marginally better performance than SMOTE-ENC.

In this dataset, there were only three nominal features—WindGustDir, WindDir9am and WindDir3pm—and from Figure 10, it can be observed that none of the three features came up in the top five feature importance ranking using the random forest classifier. Application of our method, SMOTE-ENC, did not change this ranking, showing that our SMOTE-ENC improved performance when there was a reasonable amount of association between the nominal feature and the class outcome.

4. Discussion

SMOTE and its different variant methods, including Borderline-SMOTE, ADASYN, SMOTE-Tomek, etc., only deal with continuous features. However, in the real world, datasets often consist of both continuous and categorical features. One way to use variants of these SMOTE techniques on such datasets is by one-hot encoding categorical features first and then applying the oversampling method to them. However, it generates two major problem: firstly, in cases where the dataset contains several multi-label nominal features, one-hot encoding increases data dimension significantly, which in turn brings the curse of dimensionality [33]. Secondly, being unaware of nominal features, these SMOTE variants produce multiple new labels of these attributes in the synthetically generated resampled data, which do not have any physical significance. SMOTE-NC resolves these problems, but each label is considered to have equal propensity towards the target class, muddling the feature’s contribution towards distance calculation, whereas our developed variation, SMOTE-ENC, differentiates the label of association between labels of a particular nominal feature, capturing the nominal feature’s contribution in distance calculation more precisely. For example, let us suppose there is a dataset which contains two multi-level categorical features (c1 and c2) along with a few continuous features. In the SMOTE-NC algorithm, the inter-level distance of c1 feature will be the same as the inter-level distance of the c2 feature because in SMOTE-NC, distance between any two levels of the nominal attributes is the median of standard deviations of the continuous features of the data. In other words, inter-level distance of the nominal feature is not dependent on that feature, rather on the continuous attributes of the dataset. In contrast, in our version of the SMOTE algorithm, inter-level distance of the c1 feature will be different than that of the c2 feature and the distance will not be dependent on the continuous features, rather on that particular nominal feature’s distribution. Hence, if the inter-level distance of c1 is more than that of c2, c1 will have a higher contribution while calculating distance between two data points.

ROC and precision–recall curves in the Section 3 shows, SMOTE-ENC yielded better results in four out of five datasets. In these cases, when resampled data was applied to the random forest classifier, nominal features turned out to be significant drivers. From this observation, it can be inferred that SMOTE-ENC is likely to yield better results when there is some association between the categorical features and the target class. When the association is poor, SMOTE-ENC is not able to yield better results than SMOTE-NC.

From the Table 2, it can be observed that, even when ROC and precision–recall curves show SMOTE-ENC and SMOTE-NC to yield similar performance, there was a difference between these methods’ performance in terms of precision and recall. For banking, credit card and rain datasets, SMOTE-ENC generated better precision, for forest cover dataset, it produced improved recall. Thus, SMOTE-ENC and SMOTE-NC can be differentiated when either precision or recall had higher priority and $F_{\beta }$ score was a better evaluator than F1-score.

From Figure 1 and Figure 3, it is also evident that SMOTE-ENC outperformed SMOTE-NC method substantially on banking and on credit card datasets. For the rest of the datasets, the new algorithm could not attain significant improvement. It was observed that in these two datasets (Table 2), the proportion of the number of nominal features to total number of features was more than 25%, whereas for rest of the datasets this proportion was less than 20%. From this observation, it can be inferred that, in these two datasets, a change in interpretation of the nominal features could bring a considerable amount of change in interpretation of the whole dataset. This might be the reason why SMOTE-ENC was able to achieve substantial improvement on these datasets.

When a dataset has one or more multi-label nominal features, SMOTE-NC fails to identify if one or few labels are more associated towards minority class instances than others and thus muddles the significance of the feature in distance calculation. In our method, SMOTE-ENC, when the data has multiple nominal features, for each nominal feature the distance between its labels is calculated in isolation, thus the inter-label distance for each nominal feature is different. In contrast, in SMOTE-NC, the inter-label distance between every nominal feature is considered the same.

From Table 2, it can also be observed that SMOTE-NC could not be applied on the car dataset. The car dataset used in this study did not have any continuous feature but only multi-level categorical features and the SMOTE-NC algorithm can only function when there is at least one continuous feature. In the SMOTE-NC algorithm, the distance between two levels of a nominal feature is the median of standard deviations of all continuous features for the minority class. Therefore, if the dataset does not have any continuous features, median of standard deviations of the continuous features cannot be calculated and the algorithm fails to generate synthetic data points to balance the dataset. In contrast, our new SMOTE-ENC algorithm has been generalized to handle both mixed (i.e., datasets with continuous and categorical features) and categorical-only datasets.

5. Conclusions

From the experimental results, it can be inferred that the proposed method outperforms the existing method when the dataset has a substantial number of categorical features. SMOTE-ENC is an extension of SMOTE and can be enhanced further and implemented with other variants of SMOTE, e.g., SVMSMOTE, ADASYN, SMOTEENN, SMOTE Tomek, etc. This algorithm is equipped to handle datasets with only nominal features as well, which is not possible in the existing SMOTE-NC method. SMOTE-ENC implements one way to interpret the relationship between nominal feature and target class by encoding each label as χ. There are various other ways to calculate amount of association between two nominal features and those can be implemented and evaluated on skewed datasets in future. The code is available at https://mkhushi.github.io/ (accessed on 5 January 2021).