A Network Intrusion Detection Method Based on Bagging Ensemble

Abstract

The problems of asymmetry in information features and redundant features in datasets, and the asymmetry of network traffic distribution in the field of network intrusion detection, have been identified as a cause of low accuracy and poor generalization of traditional machine learning detection methods in intrusion detection systems (IDSs). In response, a network intrusion detection method based on the integration of bootstrap aggregating (bagging) is proposed. The extreme random tree (ERT) algorithm was employed to calculate the weights of each feature, determine the feature subsets of different machine learning models, then randomly sample the training samples based on the bootstrap sampling method, and integrated classification and regression trees (CART), support vector machine (SVM), and k-nearest neighbor (KNN) as the base estimators of bagging. A comparison of integration methods revealed that the KNN-Bagging integration model exhibited optimal performance. Subsequently, the Bayesian optimization (BO) algorithm was employed for hyper-parameter tuning of the base estimators’ KNN. Finally, the base estimators were integrated through a hard voting approach. The proposed BO-KNN-Bagging model was evaluated on the NSL-KDD dataset, achieving an accuracy of 82.48%. This result was superior to those obtained by traditional machine learning algorithms and demonstrated enhanced performance compared with other methods.

1. Introduction

The increasing popularity of the Internet and the continuous development of information technology have led to an increase in the prominence of network security issues. This has given rise to a multitude of network attacks and malicious behaviors, which have the potential to pose a significant challenge to the security and stability of network systems [1]. IDSs represent a crucial network security protection measure. By monitoring network traffic and system activities, they can detect and respond to potential security threats in a timely manner. This makes them an invaluable tool in the protection of network security [2]. Network traffic can be categorized into normal and abnormal, and intrusion detection can be regarded as a binary classification problem. The accuracy of IDS can be significantly improved by improving the performance of the classifier [3]. There are three approaches to existing intrusion detection techniques: signature detection, anomaly detection, and statistical detection [4]. Signature detection is one of the most prevalent techniques employed in intrusion detection systems. Its fundamental premise is to identify known attack patterns and malicious behaviors through the utilization of a predefined feature library or rule set. Nevertheless, signature detection is subject to certain limitations in the context of addressing unknown and variant attacks. Anomaly detection represents the opposite approach, whereby potential attacks are identified by constructing a model of normal behavior and detecting activities that deviate significantly from the norm. Although anomaly detection is an effective method for identifying unknown attacks, it is also prone to a higher false alarm rate. Statistical detection is an intrusion detection method (based on statistical principles) that identifies abnormal behavior by analyzing the statistical characteristics of network traffic or system activities. In comparison with signature detection and anomaly detection, statistical detection places greater emphasis on the analysis of data distribution and patterns, which can enhance the accuracy of detection to a certain extent.

In recent years, there has been a proliferation of intrusion detection methods, including machine learning (ML) [5], deep learning (DL) [6], ensemble learning (EL) [7], and others. These methods have been employed in research on intrusion detection, demonstrating enhanced performance. Machine learning models (e.g., decision trees [8], SVM [9], KNN [10]) can be utilized to identify pivotal features within network traffic and system logs, and to detect anomalous activities by learning patterns of normal network traffic and system behavior. Deep learning models (e.g., DNN [11], RNN [12], LSTM [13]) can automatically learn feature representations in data through the hierarchical structure of deep neural networks, and can provide better generalization capabilities when trained on large-scale data. Ensemble learning methods, including bagging [14], boosting [15], and stacking [16], are employed to enhance the model’s robustness and generalizability by integrating diverse approaches to various base estimators.

Although intrusion detection systems are capable of detecting and responding to potential security threats in a timely manner to a certain extent, there are still significant challenges when faced with the real-world, large-scale, high-speed, and complex network environment. One such challenge is the asymmetry of information characteristics and redundancy features of datasets [17,18]. In practical data analysis and model training tasks, it is desirable for the features in the dataset to provide a substantial amount of information that can be leveraged to support model training and prediction. However, the presence of redundant features may result in increased computational complexity and a reduction in the model’s generalization ability. Consequently, it is a significant challenge to accurately and efficiently identify redundant features for filtering. Secondly, there is the asymmetry of network traffic distribution [19]. In a realistic distribution of network traffic data, there is usually a large amount of normal traffic and a small amount of abnormal traffic. Consequently, the model may tend to predict the majority class, which may result in a decrease in model performance and generalization ability. Therefore, another challenge is to improve the stability and generalization ability of the model in the face of unbalanced samples.

Consequently, the conventional network malicious traffic detection techniques currently face significant challenges. In light of the asymmetry of dataset information features and redundant features, and the asymmetry of network traffic distribution in the field of network intrusion detection, there is an urgent need for research on detection methods with a high detection rate in order to enhance cybersecurity. To address the aforementioned issues, this paper employs feature engineering, integrated learning, optimization algorithms, and other methodologies to conduct in-depth research on the effective detection of anomalous traffic. This paper’s contributions are as follows:

• The ERT method was employed to determine the significance of the Gini factors associated with the traffic data features. This involved the removal of irrelevant and superfluous features and the construction of an optimal subset of features.
• The BO algorithm was used to optimize the parameters of the KNN base estimators to improve the performance of the ensemble model.
• A bagging ensemble approach was proposed for intrusion detection. Based on feature selection and parameter optimization, training samples were randomly sampled using the bootstrap method to construct KNN base estimators for hard voting integration. This approach avoided the instability of a single base classifier due to the imbalance of data categories, effectively reducing the variance and improving the model’s generalizability.
• A series of comprehensive experiments were conducted on the intrusion detection dataset NSL-KDD with the objective of comparing the performance of different machine learning models integrated as base estimators. This was performed to validate the effectiveness of the proposed model.

The rest of this paper is organized as follows: Section 2 presents related work. Section 3 explains our proposed model. Section 4 describes the dataset, shows the experimental results and analyzes them. Section 5 summarizes and provides an outlook.

2. Related Work

The existing body of the literature on intrusion detection techniques is primarily concerned with three principal areas of research. One area of investigation is the development of feature engineering techniques for intrusion detection datasets. In their study, Safaldin et al. [20] proposed an enhanced IDS by using the modified binary grey wolf optimizer with support vector machine (GWOSVM-IDS), which improves the accuracy and detection rate of intrusion detection and reduces the processing time by decreasing the rate of false alarms and the number of features generated by the IDS. Kumar et al. [21] proposed a penalized reward-based ant colony optimization (PRACO) feature selection method, which facilitates a more optimal exploration–exploitation trade-off by rewarding useful features and penalizing others. The proposed model achieved an accuracy of 81.682% on the NSL-KDD dataset. Ghosh et al. [22] proposed a new wormwood grouse mating (SGM) algorithm in 2022 and applied it to IDS by reducing the original 41 features to 14, achieving an average accuracy of 81.429%. Ye et al. [23] employed the meta-heuristic algorithm hybrid breeding optimization (HBO) to IDS and proposed an integrated framework for feature selection based on the improved HBO. The framework assigns a subpopulation to each feature space and identifies the optimal subset of features. The objective was to enhance the accuracy of intrusion detection through the integration of the subpopulations. Herrera et al. [24] conducted a comprehensive examination of existing feature selection algorithms to identify their shortcomings and developed novel multimetric feature selection algorithms to reduce the dimensionality of the training dataset by leveraging the qualitative information provided by multiple feature selection metrics, and through experimentation. This study demonstrated the efficacy of the proposed approach. Nazir et al. [25] applied feature selection algorithms to reduce the dimensionality of the data and improve the performance of classifiers by proposing a wrapper-based feature selection method—taboo search–random forest (TS-RF)—and testing it on the UNSW-NB15 dataset. The experimental results demonstrated that TS-RF enhanced classification accuracy while reducing the number of features and the false alarm rate. In 2024, Akhiat et al. [26] proposed an efficient integrated feature selection algorithm for intrusion detection, namely the intrusion detection efficient ensemble feature selection (IDS-EFS). This algorithm enhances the interpretability of the model by decreasing the network data dimensionality, thereby reducing resource requirements and improving generalization. Khammassi et al. [27] tested three different decision tree classifiers. The use of binomial logistic regression and multinomial logistic regression for binary and multiclassified datasets, respectively, has been demonstrated to successfully reduce the feature space and improve classification performance. Yang et al. [28] combined the feature selection technique with the integration method to propose an adaptive ensemble feature selection algorithm for intrusion detection systems (IDS-EFS). This combined approach resulted in the development of an adaptive integration model. Specifically, the neighborhood dependency of neighborhood rough set (NRS) was introduced into the bottle sea sheath algorithm (SSA) to propose a heuristic feature selection algorithm (NRS-SSA). Furthermore, SSA was introduced to optimize the weight matrix when setting the voting weights. The results demonstrated that the model achieved a state-of-the-art level of detection. Mohammad et al. [29] proposed an innovative feature selection algorithm called “the highest wins” (HW), which demonstrated advantages in a variety of evaluation metrics, including recall, precision, and error rate, compared with the well-known chi-square and information gain strategies.

The second area of research is the investigation of sample category imbalance in intrusion detection datasets. Qian et al. [30] identified the issue of sample category imbalance, which can result in the suboptimal detection performance of intrusion detection models. They proposed an improved hybrid sampling (IHS) method based on the chaotic particle swarm optimization (CPSO) algorithm as a sampling algorithm to address this issue, and a deep learning long short-term memory (DLSTM) model with long short-term memory (LSTM) functionality was proposed, which achieved high accuracy in classifying intrusion behaviors. This model outperformed other comparative models in terms of accuracy. Jiang et al. [31] employed a one-sided selection (OSS) method to reduce the influence of noisy samples in the majority class, subsequently augmenting the minority class samples through synthetic minority oversampling (SMOTE). The proposed method enabled the model to fully learn the features of the minority class samples while reducing the training time. In 2022, Jung et al. [32] proposed a hybrid resampling method that added minority classes and removed noisy data by using SMOTE and an edited neural network to generate a more balanced dataset. The proposed method was validated using two publicly available intrusion detection datasets, PKDD2007 and CSIC2012. The results demonstrated a clear advance on previous work. Zhang et al. [33] integrated deep learning methods with statistical techniques to address the challenge of detecting minority class samples. The authors proposed an intrusion detection method, ICVAE-BSM, which employed an improved conditional variational auto-encoder (ICVAE) and a boundary synthesis minority classes oversampling technique (BSM). This method was designed to enhance the efficiency of Internet of Things (IoT) attack detection in the presence of sample imbalance and accuracy issues. In 2024, Liu et al. [34] proposed a multiconstraint migration method with additional auxiliary domains and designed a multiscale and multilevel sample augmentation discriminator to accomplish final IoT intrusion detection under sample distribution imbalance. This approach achieved an average accuracy of 96.398% on four datasets and can be effectively used for intrusion detection in real IoT environments.

The third area of research concerns the improvement and fusion of intrusion detection model structures with the objective of enhancing detection performance. In their study, Esmaeili et al. [35] explored the potential of deep learning-based intrusion detection systems and demonstrated the superiority of LSTM and BiLSTM models over other models in terms of performance. Zaryn et al. [36] conducted a comparative study on the performance and efficiency of IoT anomaly detection models by using the NSL-KDD dataset, and the results indicated that the integrated model XGBoost represents the most accurate and efficient methodology. Lee et al. [37] proposed a two-stage fine-tuning algorithm based on the WGAN-GP model to enhance the recognition accuracy of sparse data probes by fine-tuning the classification algorithm and model parameters. The final experimental results demonstrated that the MLP classifier exhibited a notable improvement in accuracy, rising from 74% to 80% after fine-tuning. This level of performance is significantly superior to that of all other classifiers. Farooq et al. [5] proposed an intrusion detection scheme, IDS-FMLT, which incorporates machine learning techniques to detect intrusions in heterogeneous networks comprising disparate source networks and to safeguard the network from malevolent attacks. The scheme achieved a validation accuracy of 95.18% and a miss detection rate of 4.82%. Sarnovsky et al. [38] proposed a hierarchical intrusion detection system based on a symmetric combination of machine learning methods and knowledge-based approaches, combining several different machine learning models. The system is capable of predicting specific types of attacks and selecting the appropriate model to perform the prediction at a selected level. Alotaibi et al. [39] proposed a fusion of machine learning with an intelligent intrusion detection model that improves accuracy and decision making by combining the predictions of different models using a fuzzy inference system. Elnakib et al. [40] proposed an enhanced anomaly based intrusion detection deep learning multiclass classification model (EIDM), which is able to classify different traffic behaviors, including multiple attack types. The results demonstrated that the method achieved an accuracy of 95% on the CICIDS2017 dataset. In a separate study, Wang et al. [41] integrated deep learning models and proposed two integrated deep learning models, SDAE-ELM and DBN-Softmax. A small batch gradient descent method was employed for network training and optimization, resulting in enhanced classification accuracy and the capacity for real-time response to intrusion behaviors. In their study, Praveena et al. [42] proposed a deep reinforcement learning technique based on the black widow optimization (DRL-BWO) algorithm and used an improved reinforcement learning-based deep belief network (DBN) for intrusion detection, achieving an accuracy of 98.5%.

Our approach is a simultaneous consideration of the three aforementioned aspects of the research. Firstly, the ERT method represents an efficacious solution to the issue of asymmetry between informative and redundant features. The method improves classification performance while reducing the dimensionality of the data, and the highly parallelized training approach is effective when dealing with large-scale data. Secondly, the use of a bagging ensemble method addresses the problem of asymmetry in sample distribution. This method improves the complexity of the model, avoids the instability of individual base estimators due to the imbalance of data categories, and enhances the generalization ability of the fused model. Finally, the parameters of the base estimators are tuned using the BO algorithm, which further improves the performance of the model.

3. Proposed Method

The proposed BO-KNN-Bagging framework is illustrated in Figure 1. It encompasses four principal stages: (1) Data preprocessing. Initially, the original attack categories are converted into binary classification, then the category-based labeled data are converted into numerical classification through label coding, and finally the data are scaled using the min-max normalization technique. (2) Feature selection. The ERT algorithm is employed to evaluate the importance of features and determine the weights of features, thereby identifying the most informative ones. (3) Construction and optimization of base estimators. A total of 5-200 base estimators are constructed, and multiple self-help samples are generated by extracting samples from the original dataset with putback based on the method of bootstrap. Tenfold cross-validation is performed, followed by tuning of the base estimators’ parameters using BO. The model is then trained independently using each of these samples. (4) Model ensemble. In the binary classification problem, the independent prediction results of KNN base estimators are integrated using hard voting, i.e., majority voting, and the final classification result obtained is used as the output of the entire integrated model. The trained BO-KNN-Bagging model is employed for the purpose of detecting attacks.

3.1. Data Preprocessing

To facilitate the training and prediction of machine learning models, we employed label encoding to convert category-based label data into numerical form. Integer encoding maps each feature label to an integer value, which is more space efficient than binary encoding and performs well in many machine learning algorithms.

Concurrently, different features tend to have disparate sizes and ranges, which can result in some features exerting a considerable influence on the model while others have a relatively minor impact. To address this issue, all features are scaled to a uniform range using min-max normalization, also known as deviation normalization. This is a common data preprocessing technique employed to scale numerical data to a specific range, typically within the bounds of [0, 1] or [−1, 1]. The method entails a linear transformation of the original data, whereby the minimum value is mapped to the desired minimum value and the maximum value is mapped to the desired maximum value. The calculation procedure for min-max normalization is presented below.

$$X_{norm}=\frac{X-X_{\min }}{X_{\max }-X_{\min }}$$

(1)

where $X$ represents each sample in the original dataset, $X_{norm}$ is the normalized sample value, $X_{\min }$ and $X_{\max }$ are the minimum value and maximum value in the dataset, respectively. This method is suitable for processing features of different scales, ensuring that they have similar importance within the same range. It also helps to eliminate the dimensional effect between features, thereby improving the performance of the machine learning model. This makes the model more robust and accurate.

3.2. Bootstrap Aggregating

Bagging is an algorithm proposed by Breiman to train multiple base classifiers in parallel and perform ensemble learning based on bootstrapping [43]. As shown in Algorithm 1, bagging combines multiple models to improve the overall prediction performance and reduce the risk of overfitting. Among them, bootstrap means that a new sample matrix $D^{\prime }$ can be obtained by extracting one sample from the $N$ sample matrix $D$ with replacement, and extracting $M\left(M\le N\right)$ times:

$$D^{\prime }=\{\left({\mathrm{x}}_1{,\mathrm{y}}_1\right),\left({\mathrm{x}}_2{,\mathrm{y}}_2\right),\left({\mathrm{x}}_3{,\mathrm{y}}_3\right),\cdots ,\left({\mathrm{x}}_M{,\mathrm{y}}_M\right)\}$$

(2)

where $x$ and $y$ are the feature matrix and label vector of the original samples, respectively. The number of samples is identical to that of the original sample set, and the same samples can be extracted from the new sample set. Although the sample distribution of each sample set differs to some extent, they all retain some of the information in the original training data, thus enabling each base estimator to learn slightly different content, thereby forming the diversity of the bagging ensemble model. Bootstrapping allows for the training of multiple models on different samples, thereby reducing the overall variance of the model and resulting in a more generalizable model. Additionally, by exposing the model to slightly different multisamples, the method ensures that the integrated model is better able to generalize to unseen data. This results in a model that is less sensitive to the idiosyncrasies of a single training dataset. Furthermore, the bootstrap method itself introduces randomness, which helps to deal with overfitting. Models trained on different samples produce different errors, and by combining them, these errors are averaged out, thus improving the model’s generalization performance.

Aggregation represents the integration strategy, which in classification problems is typically accomplished through voting. This involves each model providing a prediction, with the final prediction being the category that receives the greatest number of votes among these models. In regression problems, bagging typically entails taking the average of these models as the final prediction. By combining multiple base estimators, the instability of a single base classifier due to imbalance of data categories can be effectively avoided, and the variance can be effectively reduced. The application of bagging ensemble learning to a fused model enhances its generalizability compared with a single estimator [44,45]. In our study, for the KDDTrain+ training set in NSL-KDD, comprising 125,973 data points, a random sampling with putback was performed, with the same number of data points taken for each evaluator for the subsequent training step.

Furthermore, the probability that an individual is always not picked up in a training dataset containing $N$ samples with putback sampling can be expressed as follows:

$$\underset{N\to \infty }{\lim }{(1-\frac{1}{N})}^N=\frac{1}{e}\approx 0.368$$

(3)

This indicates that approximately 36.8% of the initial samples are not included in the newly constructed samples. Consequently, these samples can be utilized as out-of-bag data (OOB data) for evaluating the generalization capacity of the novel model. The out-of-bag error (OOB error) refers to the discrepancy between the observed values of the OOB data and the predicted values, which is essentially the error of predicting the new samples when the generalization error is considered rather than the training error on the training set. The lower the value of OOB error, the better the prediction performance of the model. In particular, the OOB error reflects the performance of the model on data that have not been seen during training. A lower OOB error indicates that the model has a greater capacity for generalization ability and is more accurate when applied to data that have not been used to inform its development. Consequently, we utilize the OOB error as a supplementary metric for evaluating the model.

Algorithm 1 Bagging

Input: $D$  //The NSL-KDD dataset    $n$ //The number of models    $m$ //The number of instances in the training set Output: $R^{*}$//Prediction results of the ensemble model 1:    Initialize: $E=⌀$ //The initialization set is used to store the prediction results 2:    for $i=1$ to $n$ 3:    for $j=1$ to $m$ 4:    $rnd=\mathrm{random}(0,m)$ //Random sampling 5:    $D_{ij}=D(rnd)$ //Bootstrap 6:    end for 7:    $E_i=\mathrm{training}(D_{ij})$ //Train with optimal parameters 8:    $E=E\cup E_i$ //Result integration 9:    end for 10:  $R^{*}=\mathrm{voting}(E)$ //Aggregating 11:  Return  $R^{*}$

3.3. Extremely Randomized Trees

The ERT algorithm is a machine learning algorithm based on bagging ensemble models, initially proposed by Geurts et al. in 2006 [46]. It typically employs the CART algorithm to construct the base estimators, introducing greater randomness based on random forests. In contrast to the random forest algorithm, ERT randomly selects a subset of features when choosing node splits and randomly selects a threshold within that subset for splitting. This increases the diversity of the model and further reduces the variance of the model.

The execution of traditional decision tree algorithms in a serial manner often results in the suboptimal utilization of computational resources. In contrast, ERT has clear advantages in the feature selection task [47], which employs a highly parallelized training method that effectively utilizes multicore processors and distributed computing resources. This enables it to perform well when dealing with large-scale data and to be robust to noise and outliers in the data. Furthermore, since each tree is trained on a random subset, the degree of overfitting of a single tree is relatively low, and the overall model is more capable of generalization. This enables ERT to maintain model stability and reliability when dealing more effectively with complex data.

The ERT algorithm assesses the significance of features by quantifying the impact of each feature on the model. The importance of a feature is typically gauged by measuring the influence of a feature on the model’s performance when the decision tree is split, with each node split based on a specific feature. The reduction in impurity (typically calculated using the Gini coefficient) before and after splitting based on that feature is determined at the time of splitting. The cumulative value of this reduction is then used as a measure of the importance of that feature. The Gini coefficient is calculated using the following formula:

$$Gini(D)=1-{\displaystyle \sum _{i=1}^C{p}_i^2}$$

(4)

where $Gini(D)$ is the Gini coefficient of the dataset $D$, $C$ represents the total number of categories, and $p_i^2$ is the proportion of samples in the dataset belonging to category I. The Gini coefficient of the dataset is 0, which means that all samples belong to the same category. The Gini coefficient of the dataset is 0, indicating that all samples belong to the same category. A smaller Gini coefficient signifies a lower impurity of the dataset, with a Gini coefficient of 0 indicating that the dataset is pure, i.e., all samples belong to the same category. In our study, the importance of each feature is calculated, and after sorting them in descending order of importance, the selection of relevant features can be made according to the performance of the model.

3.4. K-Nearest Neighbor

KNN is a frequently utilized supervised learning algorithm, typically employed for classification and regression problems. The fundamental concept of KNN is to categorize unknown samples based on the most prevalent category among their nearest neighbors or to predict the mean value of the samples with respect to their nearest neighbors, utilizing the distance metric of the samples in the feature space. The primary steps of the KNN algorithm include determining the number of neighbors, calculating the distances between the unknown samples and all the training samples, selecting the $K$ neighbors with the closest distances, and predicting the classification or regression based on their labels. In the KNN algorithm, Euclidean distance and Manhattan distance are commonly used as the distance index. The Euclidean distance between two n-dimensional vectors $\alpha \left(x_1,x_2,\cdots ,x_n\right)$ and $\beta \left(y_1,y_2,\cdots ,y_n\right)$ in the plane can be expressed as follows:

$$D\left(\alpha ,\beta \right)=\sqrt{{{\displaystyle \sum _{i=1}^n\left(x_i-y_i\right)}}^2}$$

(5)

The Manhattan distance is:

$$D\left(\alpha ,\beta \right)={\displaystyle \sum _{i=1}^n\left|x_i-y_i\right|}$$

(6)

As a traditional machine learning algorithm, KNN is relatively straightforward to comprehend and implement, with commendable prediction outcomes. However, it is susceptible to limitations, particularly in the selection of the number of neighbors and distance metrics, and in its capacity to handle large datasets, which may result in inefficiencies and suboptimal performance in high-dimensional and imbalanced datasets [48].

3.5. Support Vector Machine

SVM is a supervised learning model for classification and regression analysis. It performs well in problems such as pattern recognition, classification, and regression, and has unique advantages, particularly for applications in high-dimensional spaces. The basic idea is to find an optimal separating hyperplane to categorize the data into different classes. This hyperplane should not only maximize the margin between categories, but also avoid misclassification as much as possible. SVM achieves this goal by the following steps.

When linearly divisible, the SVM searches for a linear hyperplane that maximizes the spacing between the two classes of data points, which can be represented as a hyperplane:

$$w\cdot x+b=0$$

(7)

where $w$ is the normal vector, which determines the orientation of the hyperplane, and $b$ is the bias, which determines the distance of the hyperplane from the origin. For the optimization problem: maximize the margin, that is, minimize $‖w‖$, while satisfying the constraints on all data points:

$$y_i(w\cdot x_i+b)\ge 1$$

(8)

When linearly indistinguishable, a slack variable ${\xi }_i$ is introduced, allowing that some data points can be on the wrong side of the line but controlling the total error through a penalty term. For the optimization problem, minimize $\frac{1}{2}{‖w‖}^2+C{\displaystyle \sum _{i=1}^n{\xi }_i}$, where $C$ is the penalty coefficient, and control the trade-off between interval and error classification.

When nonlinearly differentiable, a kernel function is employed to map the data from a low-dimensional space to a high-dimensional space and to identify a linear hyperplane within the high-dimensional space. Commonly utilized kernel functions include the polynomial kernel, the rbf kernel, and the sigmoid kernel. The optimization problem then becomes the maximization of intervals within the high-dimensional space.

SVM is capable of efficiently processing data within a high-dimensional space, while overfitting can be effectively avoided by maximizing the interval. However, this approach is not well suited to large-scale datasets, as the training time and memory consumption may become excessive for very large amounts of data. In our study, the RBF kernel was employed as the kernel function for mapping, and the SVM was assembled using the bagging method to assess the performance of distinct integrated models.

3.6. Classification and Regression Trees

CART is a tree structure based on recursive partitioning that is employed for the purpose of partitioning a dataset and constructing predictive models. The fundamental concept of this method is the recursive partitioning of the dataset into progressively smaller subsets. This process continues until a given sample within a subset is classified into the same category as the subset itself. In the construction of a CART classification tree, the process commences with the root node and entails the selection of a feature and a threshold for the partitioning of the data. This is typically determined through the application of information gain, the Gini coefficient, or the sum of squared errors. The dataset is then divided into two subsets based on the selected feature and threshold. Subsequently, the same segmentation rules are recursively applied to each subset until a stopping condition is met. The stopping condition may be that the number of samples in a node is less than a certain threshold or that the purity of the node reaches a certain threshold (e.g., all samples belong to the same category).

The CART classification tree is capable of handling high-dimensional data and complex classification scenarios, and is relatively immune to outliers. However, it is susceptible to minor variations in the data, which may result in significant alterations to the structure of the generated tree. In our study, the Gini coefficient is employed to segregate the data, and the CART is assembled using the bagging ensemble method to assess the efficacy of distinct integrated models.

3.7. Bayesian Optimization

BO is a global optimization method based on Bayes theorem, which is suitable for the situation when the objective function is difficult to calculate or the computational cost is high. The key idea is to establish a probability model of the objective function $f\left(x\right)$ to guide the search process, and the parameter configuration $X^{*}$ that leads to the optimal value of the objective function is found:

$$X^{*}=\underset{x\in X}{\max }f\left(x\right),X\in R$$

(9)

In order to optimize the objective function, BO typically employs a Gaussian process as an a priori model to represent the unknown objective function. A Gaussian process characterizes the distribution of the data by means of a mean function and a covariance function, which is typically considered an approximation of the objective function. The covariance function is employed to represent uncertainty [49]. In contrast to traditional grid search or random search, BO is capable of identifying optimal solutions in high-dimensional spaces and complex objective functions with greater efficiency [50]. In our study, the optimization objective is to maximize the classification accuracy, and $f\left(x\right)$ is the accuracy of the model under the parameter. Ten-fold cross-validation is used to optimize the number of neighbors K and the type of distance metrics of the KNN.

4. Experiments Results and Discussion

This section delineates the methodology employed in the experiment, and the default parameter settings are presented in

Table 1. Model parameter settings.

Model	Parameter Settings
ERT	‘n_estimators’: 100, ‘criterion’: ‘gini’.
CART	‘criterion’: ‘gini’, ‘splitter’: ‘best’.
SVM	‘kernel’: ‘rbf’.
KNN	‘n_neighbors’: 5, ‘weights’: ‘uniform’, ‘p’: 2.
AdaBoost	‘base_estimator’: ‘DecisionTreeClassifier’, ‘n_estimators’: 50.
MLP	‘hidden_layer_sizes’: 100, ‘activation’: ‘relu’, ‘solver’: ‘adam’, ‘max_iter’: 200.
LSTM	‘units’: 50, ‘activation’: ‘relu’, ‘recurrent_activation’: ‘sigmoid’.
BO	‘cv’: 10, ‘n_iter’: 10, ‘n_neighbors’: (1, 200), ‘p’: (1, 2).

below. Initially, the data and preprocessing are elucidated. Subsequently, the model evaluation metrics are explained Finally, the obtained results are presented, analyzed, and compared with other models.

4.1. Dataset Description

KDD-Cup99, a classic dataset for intrusion detection in computer networks, contains an analysis of simulated U.S. Air Force network traffic with the objective of identifying intrusions in the network [51]. However, it is notable that the dataset suffers from the problem of category imbalance with redundant features. The experiments used the NSL-KDD dataset, which processed and filtered the KDD-Cup99 to make the data distribution more balanced and eliminate some duplicate and unnecessary features, which was more suitable for evaluating the performance of machine learning models on network intrusion detection tasks [52].

The KDDTrain+ and KDDTest+ subsets of the NSL-KDD dataset were employed for training and testing purposes, comprising 125,973 and 22,544 instances, respectively. The original dataset comprised 43 columns, with the first 41 columns representing feature vectors, as illustrated in

Table 2. Feature description.

Item	Feature Name	Item	Feature Name
1	duration	22	is_guest_login
2	protocol_type	23	count
3	service	24	srv_count
4	flag	25	serror_rate
5	src_bytes	26	srv_serror_rate
6	dst_bytes	27	rerror_rate
7	land	28	srv_rerror_rate
8	wrong_fragment	29	same_srv_rate
9	urgent	30	diff_srv_rate
10	hot	31	srv_diff_host_rate
11	num_failed_logins	32	dst_host_count
12	logged_in	33	dst_host_srv_count
13	num_compromised	34	dst_host_same_srv_rate
14	root_shell	35	dst_host_diff_srv_rate
15	su_attempted	36	dst_host_same_src_port_rate
16	num_root	37	dst_host_srv_diff_host_rate
17	num_file_creations	38	dst_host_serror_rate
18	num_shells	39	dst_host_srv_serror_rate
19	num_access_files	40	dst_host_rerror_rate
20	num_outbound_cmds	41	dst_host_srv_rerror_rate
21	is_host_login

. The 42nd column is the attack type label, comprising the four attack subtypes of DoS, Probe, U2R, and R2L with the normal label. The normal label was not considered in this instance, as the specific type of attack was not a focus. Instead, the label of each instance was transformed to normal. The results of the transformation are presented in

Table 3. Statistics of NSL-KDD.

Category	KDDTrain+	KDDTest+
Abnormal	58,630	9711
Normal	67,343	12,833
Total	125,973	22,544

. Column 43 indicates the number of times the initial researchers correctly categorized the sample using 21 machine learning algorithms, with values between 0 and 21.

4.2. Data Preprocessing

It should be noted that, although the NSL-KDD dataset represents an improved version, columns 2, 3, 4, and 42 are not numerical and therefore unsuitable for machine learning model training. The feature “protocol_type” comprises three types (icmp, tcp, udp), while the feature “service” encompasses 70 types. The “flag” contains 11 different types (OTH, S3, RSTOS0, S2, SH, S1, RSTO, RSTR, REJ, S0, SF), and the 42nd column is the attack label, which is divided into normal and abnormal categories. Consequently, further processing is required, as illustrated in

Table 4. Label encoding results.

Feature Name	Types	Results
protocol_type	icmp	0 1 2
	tcp
	udp
service	http, private, domain_u,⋯, tftp_u, http_8001, http_2784	0–69
flag	OTH, S3, RSTOS0, S2, SH, S1, RSTO, RSTR, REJ, S0, SF	0–10
attack	normal	0 1
	abnormal

below. The data are transformed into numerical form using label encoding. Following this, the data are normalized and scaled to the range [0, 1].

4.3. Evaluation Measures

The evaluation of the model’s performance is conducted through the use of accuracy, precision, recall, and f1-score. These metrics are defined in

Table 5. Confusion matrix.

	Predicated Abnormal	Predicated Normal
Actual abnormal	$TP$	$FN$
Actual normal	$FP$	$TN$

, and the aforementioned metrics can be derived by calculating the confusion matrix for cyberattacks [53]. In this context, true positive ($TP$) denotes the number of samples in which abnormal traffic is correctly categorized as such; false negative ($FN$) denotes the number of samples in which abnormal traffic is incorrectly categorized as normal; false positive ($FP$) denotes the number of samples in which normal traffic is incorrectly predicted to be abnormal; true negative ($TN$) indicates the number of samples where the traffic predicted to be normal is actually normal. The formula for evaluating the metrics is shown below.

$$Accuracy=\frac{TP+TN}{TP+FN+FP+TN}$$

(10)

$$Precision=\frac{TP}{TP+FP}$$

(11)

$$Recall=\frac{TP}{TP+FN}$$

(12)

$$F1-score=\frac{2\times Recall\times Precision}{Recall+Precision}$$

(13)

4.4. Feature Selection

The performance of various machine learning models (CART, SVM, KNN, AdaBoost, RF) was initially evaluated with default parameters.

Table 6. Performance with default parameters.

Model	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
CART	78.25	68.91	90.21	78.13
RF	77.44	66.26	97.05	78.75
SVM (rbf)	78.26	66.98	97.70	79.48
KNN	77.57	66.24	97.74	78.97
AdaBoost	73.61	63.30	92.15	75.05

presents the results. It can be observed that the majority of the models exhibit high recall values, indicating that these models are effective in detecting attack traffic. However, both accuracy and precision are lower due to the imbalanced nature of the NSL-KDD dataset, which contains a larger proportion of normal traffic and a smaller proportion of abnormal traffic. The aforementioned imbalance prompts the models to exhibit a proclivity towards predicting the majority class (normal traffic), thereby enhancing the recall value. However, this inclination concurrently results in a decline in accuracy and precision, attributable to the suboptimal performance of the minority class (abnormal traffic), in conjunction with the presence of a considerable number of redundant and noisy features, which may exert a detrimental influence on the training of the models. Rather than applying additional class-balancing techniques, such as undersampling and oversampling, we utilized feature engineering and ensemble methods to enhance the model’s performance and generalizability. Our findings indicated that the SVM model based on the rbf kernel demonstrated the most favorable performance, with an accuracy rate of 78.26% and an f1-score of 79.48%. CART demonstrated the highest precision (68.91%) but the lowest recall (90.21%). The AdaBoost model exhibited poor performance, with all four indicators at a low level. Moreover, although the KNN model exhibited the highest recall at 97.74%, its accuracy, precision, and f1-score were 0.69%, 0.74%, and 0.87% lower, respectively, than the best performing SVM model.

Subsequently, we employed the ERT method for the purpose of feature extraction and selection. Thereafter, we calculated the relative importance of the 41 features. The results of this process are presented in Figure 2. Feature importance is defined as the degree to which each feature contributes to the model prediction. Among the features, “logged_in” exhibited the greatest influence on the model, followed by “same_srv_rate” and “dst_host_srv_serror_rate”. The features “is_host_login”, “num_outbound_cmds”, “num_access_files”, etc., had minimal impact on the model and can be considered redundant. Figure 3 illustrates the impact of 0–41 features selected by ERT on the accuracy. It can be observed that the model’s accuracy did not improve with the increase in the number of features, but fluctuated. This is due to the fact that the dimensionality of the data space increased with the increase in the number of features. In a high-dimensional space, the distance between data points becomes sparser, which impairs the model’s ability to learn and generalize. Additionally, the introduction of redundant features negatively affects the model’s training and prediction capabilities. Consequently, the accuracy of the model will reach its highest value at a specific point, necessitating the identification of optimal feature values that are applicable to a range of machine learning models. The overall accuracy of the CART model was optimal when the number of selected features was greater than 25. When the number of selected features was between 10 and 20, the KNN model had the highest accuracy, reaching a maximum of 81.95%, when the number was 17. This value was higher than the maximum accuracy of the other four machine learning models. When the number of selected features was between 0 and 10, the KNN model exhibited the most significant accuracy improvement.

The features were ordered according to their importance, with the most significant features listed first. The optimal number of features for different machine learning models was then selected in order of serial number. The specific features selected are presented in

Table 7. Selected feature number.

Model	Number
CART	11, 28, 38, 32, 25, 3, 1, 37, 33, 35, 24, 2, 26, 31, 22, 39, 40,27, 36, 4, 7
RF	11, 28, 38, 32, 25, 3, 1, 37, 33, 35, 24, 2, 26, 31, 22, 39, 40,27, 36, 4, 7, 23, 29
SVM (rbf)	11, 28, 38, 32, 25, 3, 1, 37, 33, 35, 24, 2, 26, 31, 22, 39
KNN	11, 28, 38, 32, 25, 3, 1, 37, 33, 35, 24, 2, 26, 31, 22, 39, 40
AdaBoost	11, 28, 38, 32, 25, 3, 1, 37, 33, 35, 24, 2, 26, 31, 22, 39

, and

Table 8. Best performance for feature selection.

Model	Selected Top n Features	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
CART	21	81.93	71.49	96.56	82.15
RF	23	79.17	68.19	96.79	80.01
SVM (rbf)	16	79.68	68.77	96.79	80.41
KNN	17	81.95	71.48	96.67	82.19
AdaBoost	16	78.51	68.79	91.71	78.62

illustrates the performance of the models after the features were selected. It can be observed that the features selected for different models were largely disparate, which is attributable to the differing learning abilities and complexities of the various classifiers. For instance, decision tree models (including CART and RF) tend to select features that can be most efficiently segmented, which may be directly related to the target variable or ability to achieve high information gain with a small number of segmentation steps. In contrast, SVM employs kernel functions to map the data, potentially leading to the selection of features that are most relevant to the decision boundary. Consequently, different models may select distinct features as the most crucial ones due to the distinctive characteristics of their learning algorithms. Following the application of the ERT feature selection method, there was a notable enhancement in the accuracy, precision, and f1-score of all models (in comparison with Table 6). Specifically, the CART model had the most significant improvement in recall, which increased by 6.35%. AdaBoost had the most improvement in the accuracy, precision, and recall, which went up by 4.9%, 5.49%, and 3.57%, respectively, followed by KNN (4.38%, 5.24%, and 3.22%). The RF model’s performance also slightly improved to reach an accuracy of 79.17%, precision of 68.19%, and recall of 80.01%. The SVM model had the least performance improvement, with 1.42%, 1.78%, and 0.93% in accuracy, precision, and recall, respectively. The KNN model performed the best among the five models, achieving the highest accuracy (81.95%) and f1-score (82.19%).

4.5. Performance of Ensemble Models

To further improve the model performance, we used different machine learning models for integrated training, and explored the relationship between different number of estimators and accuracy, increasing the number of estimators in intervals of 5 from 5 to 200, i.e., the sample size of training increased from 5 times to 200 times; the results are shown in Figure 4. CART-Bagging, RF, Adaboost model accuracy fluctuates, KNN-Bagging, and SVM-Bagging are more stable, while KNN-Bagging model has higher accuracy. When increasing the number of estimators, with the increase in training samples, the initial number of some base estimators usually improves the accuracy significantly, this is because by integrating multiple models, the variance of a single model can be effectively reduced, and each base estimator can focus on a different aspect of the data during the training process, which makes the integrated model more robust. As the number of estimators continues to increase, the improvement in accuracy slows down, this is because as the number of estimators increases, the variance of the model on the training set has stabilized or decreased, and the contribution of additional estimators to reducing the variance becomes limited, a phenomenon often referred to as diminishing returns. After adding a certain number of estimators, the overall accuracy of the model stabilizes and reaches a convergence value. At this point, adding more estimators has little or no significant effect on the accuracy of the model. This is because the integration already benefits from sufficient diversity and generalization capabilities, and adding more estimators does not significantly improve the model’s performance. Changes in the number of estimators reflect changes in the data samples, i.e., as the number of training samples increases or decreases, the overall model performance changes. Specifically, when the number of training samples is small, increasing the number of estimators significantly improves the accuracy of the model because more models can help capture different features and patterns in the data and reduce the variance of the model, thereby improving generalization. As the number of samples increases, the generalization ability of the model gradually increases, but the improvement in model performance from increasing the number of estimators gradually decreases. This is because, although increasing the estimators can continue to reduce the variance, further improvement becomes more limited as the integration is already relatively stable. When the training samples are very large, the model may already be sufficient to fit the complexity of the data, at which point increasing the number of estimators will have a very limited improvement in model accuracy because the model is already sufficiently generalized, and further increases in the number of estimators will not significantly reduce the variance of the model. In practical applications, it is necessary to balance the model performance and the use of computational resources in order to identify a number of estimators that can significantly improve accuracy without consuming excessive resources.

In summary, we chose the optimal number of estimators for five integrated models, namely CART-Bagging, SVM-Bagging, KNN-Bagging, AdaBoost, and RF. The performance of these models was compared, and the results are shown in

Table 9. Performance comparison of ensemble models.

Model	Number of Estimators	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
CART-Bagging	30	80.07	69.74	94.93	80.41
SVM-Bagging	30	79.84	68.96	96.77	80.53
KNN-Bagging	55	82.00	71.48	96.62	82.22
AdaBoost	20	78.54	67.77	95.71	79.35
RF	5	79.31	68.38	96.66	80.10

. The number of estimators chosen for the above five integrated models were 30, 30, 55, 20, and 5, respectively. When bagging ensemble was used, KNN was chosen as the evaluator with the best performance, achieving an f1-score of 82.22%, outperforming CART (80.41%) and SVM (80.53%). The AdaBoost model performed poorly, with the lowest accuracy (78.54%), precision (67.77%), and f1-score (79.35), and the CART-Bagging model had the lowest recall (94.93%) and highest accuracy (80.07%). Among all the integrated models, the KNN-Bagging model had the best performance, although accuracy was slightly lower than the CART-Bagging model (0.07%), achieving the highest precision (71.48%) and f1-score (82.22%), which was an improvement of 5.24% and 3.25%, respectively, with respect to the original KNN model, so we further tuned the parameters of KNN using BO.

4.6. Performance of Ensemble Models after Parameter Optimization

Finally, we used BO to tune the number of neighbors K and the type of distance metrics (Euclidean and Manhattan distances) of the KNN. The range of BO parameters can be seen in Table 1 above, and the results show that the model performed best when the value of K was taken to be 2 and Euclidean distances were used. The accuracy, precision, and f1-score of the BO-KNN-Bagging model were improved by 0.48%, 0.74%, and 0.36%, respectively, relative to the untuned KNN-Bagging model.

Table 10. Performance of ensemble models after parameter optimization.

Number of Estimators	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
5	81.98	71.62	96.35	82.17
50	82.46	72.21	96.35	82.55
100	82.48	72.22	96.40	82.58
150	82.46	72.20	96.40	82.56
200	82.47	72.21	96.40	82.57

investigates the effect of different number of estimators (5–200) on the accuracy of the BO-KNN-Bagging model, and the results show that when the number of estimators was small, the model performance was poor, achieving only 81.98% accuracy, which was even inferior to the accuracy of the KNN-Bagging model without parameter tuning (82%). However, as the number of estimators increased, the accuracy stabilized and reached its highest value (82.48%) when the number of estimators was 100, and, as the number of estimators increased further, the accuracy fluctuated in a smaller range, which confirms the argument in Section 4.5. The performance comparison results of the proposed BO-KNN-Bagging with other machine learning models are shown in Figure 5 and Figure 6. It can be observed that the optimized models all achieved the highest values of accuracy (82.48%), precision (72.22%), and f1-score (82.58%).

To further explore the generalization performance of the integrated model, we evaluated it using OOB error, which reflects the performance of the model on unseen data, and a lower error implies that the model has a higher accuracy on unseen data and a better generalization ability. The results are shown in

Table 11. Comparison of generalization performance of ensemble models.

Model	OOB Score	OOB Error
CART-Bagging	0.9875	0.0125
SVM-Bagging	0.9825	0.0175
KNN-Bagging	0.9940	0.0060
BO-KNN-Bagging	0.9947	0.0053

, where the OOB score is the score of the model on OOB data, i.e., the prediction accuracy of the model on OOB data. It can be found that the proposed BO-KNN-Bagging model had the highest OOB score (0.9947) and the lowest OOB error (0.0053). The error was reduced by 0.0007 for the untuned KNN-Bagging model, which indicates that the proposed model achieved the highest accuracy with the lowest error on unknown data with good generalization performance.

Table 12. Performance comparison of different models.

Model	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
CART	78.25	68.91	90.21	78.13
SVM (rbf)	78.26	66.98	97.70	79.48
Adaboost	73.61	63.30	92.15	75.05
MLP	77.84	66.65	97.17	79.07
LSTM	75.10	63.87	97.17	77.08
RF	77.44	66.26	97.05	78.75
KNN	77.57	66.24	97.74	78.97
ERT-KNN	81.95	71.48	96.67	82.19
KNN-Bagging	82.00	71.48	96.62	82.22
BO-KNN-Bagging	82.48	72.22	96.40	82.58

demonstrates the comparison of the proposed model with machine learning models and existing methods. Our model achieved 82.48% accuracy, 72.22% precision, and 82.58% f1-score, which was an improvement over the traditional KNN model by 4.91%, 5.98%, and 3.61%, respectively, and maintained a relatively high recall (96.4%). The accuracy also improved (4.64% and 7.38%) for the classical deep learning models (MLP and LSTM), which provides a significant advantage in the field of network intrusion detection.

Table 13. Performance comparison of different methods.

Method	Year	Feature Selection	Accuracy (%)
SVM [20]	2021	YES	79.00
LSTM [35]	2022	YES	82.30
PRACO [21]	2022	YES	81.68
SGM [22]	2022	YES	81.73
XGBoost [36]	2023	NO	80.30
MLP [37]	2023	NO	80.00
BO-KNN-Bagging	2024	YES	82.48

shows the comparison of the accuracy of our model with the existing methods in the last five years, and our model had the highest accuracy, with an improvement of 3.48% and 2.18% compared with the SVM model studied by Safaldin et al. [20] and the integrated model XGBoost [36] studied by Zaryn et al. Compared with models that have been optimized for algorithmic feature selection, namely PRACO [21] and SGM [22], also demonstrated improved performance. Similarly, for some deep learning models, namely LSTM [35] and MLP [37], there was a 0.18% and 0.48% improvement in accuracy, respectively.

5. Conclusions and Future Work

Aiming at the problem of asymmetry of information features and redundant features of datasets and asymmetry of network traffic distribution in the field of network intrusion detection, which leads to low accuracy and poor generalization of traditional machine learning detection methods in IDS, a network intrusion detection method based on bagging ensemble is proposed. Using ERT for feature selection, we explore the model performance of different machine learning models as bagging basis ensembles, and find that the KNN-Bagging ensemble model has the best performance, and then we use BO for parameter tuning of the basis ensembles KNN, and finally the basis ensembles are integrated. The results show that the BO-KNN-Bagging model achieves an accuracy of 82.48%, which is higher than traditional machine learning algorithms, and also has better performance compared with other methods. Nevertheless, there are still some limitations. The effectiveness of bagging depends on the variability among the base estimators, and although we performed random sampling to ensure diversity in the training data, the base estimators use the same model and parameters, and the performance gains of the integrated model may be limited. Also, bagging requires the training of multiple base estimators and therefore has a high computational overhead, especially for complex learning algorithms or large-scale datasets.

The next step is to construct more complex artificial neural network (ANN) models as base estimators for integrating different methods (e.g., boosting, stacking), and also to consider using oversampling techniques to make the model better learn the features of a few categories, so as to further improve the model’s generalization performance and accuracy.