Enhancing the Sustainability of Deep-Learning-Based Network Intrusion Detection Classifiers against Adversarial Attacks

Abstract

An intrusion detection system (IDS) is an effective tool for securing networks and a dependable technique for improving a user’s internet security. It informs the administration whenever strange conduct occurs. An IDS fundamentally depends on the classification of network packets as benign or attack. Moreover, IDSs can achieve better results when built with machine learning (ML)/deep learning (DL) techniques, such as convolutional neural networks (CNNs). However, there is a limitation when building a reliable IDS using ML/DL techniques, which is their vulnerability to adversarial attacks. Such attacks are crafted by attackers to compromise the ML/DL models, which affects their accuracy. Thus, this paper describes the construction of a sustainable IDS based on the CNN technique, and it presents a method for defense against adversarial attacks that enhances the IDS’s accuracy and ensures it is more reliable in performing classification. To achieve this goal, first, two IDS models with a convolutional neural network (CNN) were built to enhance the IDS accuracy. Second, seven adversarial attack scenarios were designed against the aforementioned CNN-based IDS models to test their reliability and efficiency. The experimental results show that the CNN-based IDS models achieved significant increases in the intrusion detection system accuracy of 97.51% and 95.43% compared with the scores before the adversarial scenarios were applied. Furthermore, it was revealed that the adversarial attacks caused the models’ accuracy to significantly decrease from one attack scenario to another. The Auto-PGD and BIM attacks had the strongest effect against the CNN-based IDS models, with accuracy drops of 2.92% and 3.46%, respectively. Third, this research applied the adversarial perturbation elimination with generative adversarial nets (APE_GAN++) defense method to enhance the accuracy of the CNN-based IDS models after they were affected by adversarial attacks, which was shown to increase after the adversarial attacks in an intelligible way, with accuracy scores ranging between 78.12% and 89.40%.

1. Introduction

Due to the extensive use of technology in many domains, including medical, education, and entertainment, among others, cybersecurity has become a major concern. Users connect to the network and spend the majority of their time using digital devices for various tasks. Therefore, every single packet that enters their phones comes from the network, and vice versa. These devices may be compromised by the hacking of the network, which has an impact on the user’s devices by obtaining private information, such as credit card numbers, Apple Pay passwords, and PayPal account information, among others.

In light of this, IDSs are used by practically all network beneficiaries to protect their institutions from cyberattacks, such as SQL injection, CSS attacks, and DoS attacks. An IDS distinguishes between malicious and benign network packets by determining their differences. IDSs are enhanced and become more accurate and dependable when built on ML/DL. Because ML/DL is essential for avoiding intrusions and malicious behavior, it is extensively employed [1]. As in Ref. [2], the researchers built an NIDS with a CNN to enhance the accuracy. The model was evaluated with well-known datasets and achieved 99.56% accuracy.

However, utilizing ML/DL has a weakness, which is adversarial attacks. The attacks designed by the attackers against the ML/DL models are known as adversarial attacks. The ML/DL model will perform poorly during training, testing, or both when faced with such adversarial attacks. Moreover, in Ref. [3], the authors crafted adversarial attacks against a DL-based IDS and concluded that even if the DL-based IDS had high accuracy in detection, it was vulnerable to misclassification due to adversarial attacks. As a result, various protective methods are required, as DL/ML can also be deceived. Communication across an open network also makes the system vulnerable, giving adversaries a wide attack surface [4].

Overall, the valuable service that ML/DL can provide to the IDS is not diminished by this failure, but it does call for greater security measures. Thus, in this situation, IDSs are vulnerable to not spotting cyberattacks, posing a hazard to the network and compromising user data, or something more significant, depending on the network it is protecting. Therefore, this challenge drives researchers to improve the resilience of ML/DL algorithms.

The interest behind this research is to develop a sustainable IDS as an application that has a significant effect on the network domain as a protector. Hence, we apply a protection technique for the IDS, which is used to protect the network. Without the protection technique (defense strategy), the IDS will be deceived by adversarial attacks, and the network beneficiaries will be susceptible to cyberattacks. Our goal is to eliminate adversarial attacks before they affect the IDS model by using the defense strategy.

In addition, this research begins with a deeper investigation into the related studies in which the IDS was built with ML and DL. It then tries to complete and address some of the identified gaps. Figure 1 below presents the concept of the work proposed in this paper to accomplish the main research goal, which is enhancing CNN-based IDS classification accuracy in the case of adversarial attacks.

To accomplish the research goal, the proposed work consisted of three parts, shown in Figure 1. The first part was to build two IDS models based on a CNN. Second was the crafting of adversarial attacks that affect the CNN-based IDS models’ classification accuracy. Finally, the defense strategy to enhance the IDS accuracy was implemented. The defense strategy used in this research is based on using a GAN as an extra network for the CNN-based IDS models. There are numerous studies that use GANs as a defender against adversarial attacks and demonstrate its effectiveness, such as [5,6,7,8]. Some of these examples rely on adversarial training, while others only concentrate on detecting adversarial attacks. In contrast, the others seem unlikely to fit all domains and eliminate all types of attacks.

Thus, this research used one of the GAN-based models, which is the APE_GAN++ [9]. The APE_GAN++ model has demonstrated its effectiveness against adversarial attacks in the computer vision domain, and the authors mention that it could eliminate adversarial attacks with a high success rate. Additionally, in [9], the authors mentioned that this model could fit all domains and eliminates all types of adversarial attacks.

Furthermore, this research contributes to proving two important concepts regarding the APE_GAN++ defense, which are the capability of the APE_GAN++ to eliminate all types of adversarial attacks, and the APE_GAN++ fits all domains by applying it with an IDS, in which an intelligible increase in accuracy is achieved. This research validates these concepts through the following achievements:

• Building two CNN-based IDS models as a network classifier (IDS) with the CICIDS2017 [10] dataset;
• Generating seven well-known adversarial attacks against the CNN-based IDS models, which are FGSM, BIM, JSMA, DeepFool, C&W, PGD, and Auto-PGD;
• Building the APE_GAN++ defense to enhance the classification accuracy of the affected models. The defense strategy applied in this research has never been implemented in the IDS domain;
• Evaluating the CNN-based IDS models before and after attacks, as well as after applying the APE_GAN++ defense;
• In the literature, the set adversarial parameters are inaccurate; in this research, more accurate parameters were used for adversarial perturbations, especially for FGSM attacks.

This paper is organized as follows: An introduction to adversarial machine learning and IDS is given in Section 1. Section 2 presents the related works and the background of the research topic. Section 3 clarifies the research methodology for implementing the main idea of this research. Section 4 goes through the steps for implementing the research plan. Section 5 presents and analyzes the results of implementing the APE_GAN++ in defense and enhancing the IDS accuracy. Section 6 discusses the research limitations and clarifies future work. Section 7 concludes this paper.

2. Background and Related Works

This section presents the most relevant studies leading to the research problem. First, it begins with the related works for building CNN-based IDSs, and then the studies on crafting adversarial attacks against these models. Second, it mentions some of the defense methods against adversarial attacks in the IDS domain. Finally, it introduces the problem visualization and solution concept.

2.1. CNN-Based IDS Models

This section presents the related studies in which the IDS was built with a CNN. The authors in [11] used an IDS based on the CNN model, which classifies the network traffic and evaluates the model using the CICIDS2018 [10] and NSL-KDD [12] datasets. The study focused on crafting a novel attack that could affect the IDS performance. In Refs. [2,3,4,5,6,7,8,9,10,11,12,13], the authors also built an IDS with a CNN to classify the network traffic, and the CNN was simply evaluated as a classifier for the network security domain without any adversarial scenarios.

Additionally, the authors in [14] suggested direct packet preprocessing and utilized a CNN-based NIDS, which performed well when evaluated using the NSL-KDD dataset. The authors in [15] describe a method for identifying intrusions into agricultural IoT networks using a CNN. This model was evaluated by using the NSL-KDD dataset and achieved an accuracy rate of 99%.

2.2. Adversarial Attacks against IDS

In Refs. [16,17,18], the researchers crafted white-box attacks against an ML/DL-based IDS. After the adversarial attacks, the ML/DL model accuracies dropped to 22.52% and 29.87%, respectively. For some models, the adversarial attacks had low affection, such as MLP with an 83.27% classification rate, followed by the BAG at 80.20% and the LDA at 79.68%. As a result, adversarial attacks affect the models with the scores varying from one model to the next. In Ref. [19] as well, the authors crafted white-box attacks against IDS-based ML in the IoT domain. This research was evaluated using the Bot-IoT dataset.

In Refs. [20,21,22,23,24,25,26], the researchers developed black-box attacks against ML/DL-based IDS to affect the IDS performance. These studies all differed in the purpose of the attacks. More specifically, the researchers in [23] crafted adversarial attacks to improve the IDS performance by training the model using novel attacks. Other studies have focused on demonstrating that adversarial attacks can lower IDS detection rates to near zero. In contrast, all these studies used the GAN to develop black-box attacks against the IDS.

2.3. Defense Method against Adversarial Attacks

In order to secure the ML/DL algorithms against attackers, this subsection describes the most cutting-edge protection measures.

Adversarial training is one of the defense mechanisms against adversarial attacks. Thus, a DNN’s regularity and robustness are primarily increased by the adversarial training [27]. In Refs. [5,28,29,30,31], the researchers used adversarial training to detect adversarial attacks against ML/DL models. In contrast, the zero-knowledge adversarial training defense (ZK-GanDef) method achieved an increase in accuracy of 49.17%. In addition, adversarial feature reduction (AFR) has decreased some of the adversarial attacks but needs further improvements. The adversarial perturbation elimination APE_GAN, which is based on the GAN model, has many applications, as it can work without knowledge of the model on which it is based. As well, the suggested defense used in the diabetic retinopathy recognition domain is reliable, with 99% accuracy.

There is another concept regarding defense mechanisms, which is preprocessing. In this regard, the researchers in [32,33,34] present images modified using total variance reduction, and image quilting was found to be helpful in removing adversarial perturbations from pictures, including when some noise was added to the pictures before entering them into the classifier. In Refs. [35,36,37,38], the researchers developed defense methods that also depend on the preprocessing concept. Moreover, the matrix estimation (ME-Net) achieved good results and enhanced the DNN against adversarial attacks. The deep image prior-driven defense (DIPDefend) can intelligibly remove the adversarial content from images and fits various domains. In addition, the deep image restoration model produced excellent experimental results compared with other techniques. There is another defense method against adversarial attacks named Omni, based on hyperparameter optimization, such as in [39]. The authors used an ML-based IDS and evaluated the model using five benchmark datasets.

Some of the defense methods depend on adding an extra network, such as utilizing specific external models [40]. A mechanism for guarding against adversarial attacks using universal perturbations was devised by the researchers in [41,42]. The fundamental idea of this approach is to integrate the first model with a second trained network. A Def-IDS and adversarial sample detector (ASD) are presented by the authors in [6,7]. These approaches could improve the classification by making the models more accurate at spotting adversarial attacks.

In Ref. [43], the authors present dropout as a defense method that depends on disrupting the model’s architecture [44]. The adversarial network detector presented in [42], in contrast, relies on transferability, which is not ideal for detectors. In Ref. [8], the authors developed a defense method that depends on the GAN as a defender for the IDS. As a result, the IDS’s efficiency and precision increased. Furthermore, the APE_GAN++ that was developed by [9] achieved a better performance than other defenses, including APE_GAN.

These two approaches differ in the domain that they are applied to. Thus, the study in [8] implemented an instruction detection system based on ML models and used the GAN as an extra network against one type of white-box attack, which is the C&W. APE_GAN++ was implemented in a computer vision domain and evaluated the defense model against two CNN models using two datasets, MNIST and CIFAR-10.

To sum up, the previous studies crafted adversarial attacks against the IDS based on ML and DL models. One study of the related works built the IDS with a CNN using the CSE-CIC-IDS2018 dataset, resulting in high precision. In order to enhance IDS classification, this research suggests building the IDS with the CNN model using the CICIDS2017 dataset. In the domain of computer vision, the researchers in [9] applied the CNN model using the MNIST and CIFAR10 datasets. The adversarial attacks developed in this research against the CNN model included FGSM, JSMA, BIM, DeepFool, CW, PGD, and Auto-PGD. Furthermore, presented in the defense strategies subsection is more than one method that used the GAN technique to address adversarial attacks. Hence, for this research, the APE_GAN++ approach was chosen, as it fits all domains and can address new types of attacks. As a result, the primary contribution of this research is the implementation of the recommended defense APE_GAN++, which is originally based on the GAN, to counter adversarial attacks against IDSs.

2.4. Problem Visualization and Solution Concept

A problem must be recognized before any solution can be considered. Thus, this subsection demonstrates the research problem and solution concept. In this research, the main problem is related to the network security domain. As previously presented, network security is reliant on the ML and DL models, such as the IDS, recognizing the patterns of network packets.

The IDS is responsible for discriminating the network packets as either “normal” or “attack”. The users rely on it to classify the network packets. The adversarial attacks crafted by the attackers against the CNN-based IDS result in the misclassification of these packets and thus lead the system to becoming prone to cyberattacks. Figure 2 depicts the problem.

As we already concluded, adversarial attacks can affect ML and DL models. For this reason, the APE_GAN++ model is recommended for addressing this problem and eliminating the adversaries before using the ML/DL model. Furthermore, the APE_GAN++ model has been applied in the computer vision domain, and it has achieved impressive results. This research uses the APE_GAN++ to eliminate adversarial attacks before they enter the CNN-based IDS. First, this research built two CNN models based on the IDS with the CICIDS2017 dataset. Second, adversarial attacks were crafted against the CNN-based IDS models. Third, the APE_GAN++ was used as a defense strategy against the two CNN-based IDS models. Finally, the CNN-based IDS models were evaluated before and after the attacks, as well as after applying the APE_GAN++ defense.

Table 1. Problem and solution concept.

Problem	Solution
Crafting adversarial attacks against ML/DL models.	Protecting the ML and DL models against adversarial attacks by applying the APE_GAN++ defense.
Adversarial attacks affect the CNN-based IDS model, which make the model misclassify network packets.
The model performance will be unstable and classify benign packets as attacks, and vice versa.

summarizes the problem and solution concept of this research.

3. Proposed Research Framework

This section presents the research methodology and framework to implement the proposed solution. The evaluation metrics to evaluate this solution are also presented. Finally, the experiment settings and simulation environment are also introduced.

Figure 3 below demonstrates the research framework, which consists of four phases, and every phase is the input to the other phases. Phase One is the building of the CNN models for the IDS to improve its accuracy. Phase Two involves crafting adversarial attacks against the CNN-based IDS, which decrease the model’s accuracy. Phase Three involves applying the defense method (APE_GAN++) to improve the CNN-based IDS accuracy. Finally, in Phase Four, the CNN-based IDS is evaluated before and after adversarial attacks, as well as after defense.

1. 

Phase One: Building CNN-based IDS models

The researchers in [9] used two datasets for two CNN models in the computer vision domain, which are the CIFAR-10 and MNIST datasets. However, in this proposed research, the CICIDS2017 [10] dataset is used to evaluate the CNN for the IDS in the network security domain.

The first model used in this research was the CIFAR_CNN model. This model has four convolutional layers (Conv2d) to extract features from the images. In addition, three fully connected (FC) layers are used to establish the class to which the image belongs, and the output layer outputs the dataset classes, of which there are 11. This model uses the ReLu and softmax functions as an activation function and two-dimensional max-pooling layers for each Conv2d layer to reduce the size of the features.

The MNIST_CNN model used in this research has only two convolutional layers in addition to the FC layers. It also uses the same activation functions that were used for the CIFAR_CNN model, as well as two max-pooling layers, one of which is used with dropout2d. After bundling the models, the CICIDS2017 was used to evaluate their performance in the network security domain. Furthermore, this research used the CICIDS2017 dataset for the CNN-based IDS models, as it has the highest number of records for each class, which is promising for achieving more accurate results.

2. 

Phase Two: Crafting Adversarial attacks against CNN-based IDS

In this phase, seven different attacks, which were FGSM, JSMA, BIM, DeepFool, C&W, PGD, and Auto-PGD, were used against the CICIDS2017 dataset to decrease the CNN-based IDS models’ accuracy.

3. 

Phase Three: Applying the APE_GAN++

The aim of this phase is to defend against the adversaries crafted in Phase Two by eliminating them before feeding into the CNN-based IDS models. First of all, this defense uses the GAN strategy, which consists of two convolutional neural networks, a generator, and a discriminator. As presented in Section 2, a generator is used to generate data from datasets; in this scenario, it was used to generate adversarial attacks. It also consists of four convolutional layers, (Conv2d) and (ConvTranspose2d), to create features. Furthermore, it uses the ReLu function and the Tanh as an activation function.

The discriminator is used to distinguish between benign data and adversaries and eliminate the adversarial perturbations before they enter the CNN-based IDS. This model is built with three convolutional layers (Conv2d) and one fully connected layer (FC). For the activation functions, it uses ReLu and Sigmoid functions. As a result, the discriminator takes real samples from the CNN-based IDS models and affected samples and then distinguishes which are real and eliminates the affected ones (fakes) to deliver more accurate classification results.

4. 

Phase Four: Evaluating the CNN-based IDS models

In this phase, four evaluation metrics were used to evaluate the CNN-based IDS models before and after attacks, as well as after APE_GAN++ defense with the CICIDS2017. These are accuracy, precision, the F1 score, and the recall score. The details of these metrics are presented in Section 3.1.

3.1. Evaluation Metrics

There were two aspects for evaluation in this research, which were as follows:

• Evaluating the APE_GAN++, which is a GAN model that is used as an eliminator of adversarial attacks. To evaluate the GAN model, we used the following loss function:

$${\mathrm{L}}_{\mathrm{APE}-\mathrm{GAN}++}={\mathrm{L}}_{\mathrm{WGAN}-\mathrm{GP}}+{\mathsf{\lambda }}_1{\mathrm{L}}_{\mathrm{MSE}}+{\mathsf{\lambda }}_2{\mathrm{L}}_{\mathrm{CLC}}$$

(1)

$${\mathrm{L}}_{\mathrm{CLC}}=\mathrm{CrossEntropy}\left({\mathrm{Y}}_{{\mathrm{Real}}_{\mathrm{Benign}}}-{\mathrm{Y}}_{{\mathrm{Fake}}_{\mathrm{Benign}}}\right)$$

(2)

$${\mathrm{L}}_{\mathrm{MSE}}={\left|\left|{\mathrm{X}}_{{\mathrm{Real}}_{\mathrm{Benign}}}-{\mathrm{X}}_{{\mathrm{Fake}}_{\mathrm{Benign}}}\right|\right|}^2$$

(3)

In Ref. [9], the Wasserstein GAN with gradient penalty (WGAN_GP) was used instead of the original GAN’s loss function, as it provides more stable training. The ${\mathrm{L}}_{\mathrm{APE}-\mathrm{GAN}++}$ in the first equation refers to the loss function of the whole model, which is the sum of the WGAN_GP loss, minimum square error loss, and classification loss from the CNN-based IDS models. The difference between actual network classes and fake classes is used to compute the minimal square error loss (${\mathrm{L}}_{\mathrm{MSE}}$). The classification loss (${\mathrm{L}}_{\mathrm{CLC}}$) is the classification error of the CNN-based IDS models when classifying the network packets after adversarial attacks. The values of λ₁ and λ₂ are set to 0.5;

2.

Evaluating the CNN-based IDS model as a classifier for network packets before and after attacks, as well as after defense. The evaluation metrics used in this research for the CNN-based IDS models are illustrated in

Table 2. Evaluation metrics.

Evaluation Metrics	Definition	Equation
Precision	Identifies the number of accurate classifications recognized as incursions across all results.	$\mathrm{P}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$
Recall	Shows the fraction of accurately recognized intrusions across all real intrusions.	$\mathrm{R}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$
F1 score	A measure of the effectiveness of precision and recall.	$\mathrm{F}1=\frac{2.\mathrm{P}.\mathrm{R}}{\mathrm{P}+\mathrm{R}}$
Accuracy	Identifies the proportion of properly categorized regular traffic and incursion samples.	$\mathrm{Acc}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$

below.

3.2. Experiment Settings and Simulation Environment

This research evaluated the suggested models by using the CICIDS2017 dataset. In the CICIDS2017 dataset, there are eight files in all, both malicious and benign network packets. These data from the network traffic were collected from the Canadian Institute of Cybersecurity. As mentioned in Section 2, the dataset consists of 83 features and 15 classes of network packets, which are Normal; DoS Hulk; PortScan; DDoS; DoS GoldenEye; FTP-Patator; SSH-Patator; DoS slow loris; DoS slowhttptest; Bot; Web Attack-Brute Force; Web Attack-XSS, Infiltration; Web Attack-SQL Injection; Heartbleed.

In this paper, all models were built using Pytorch as the deep learning framework to demonstrate the effectiveness of the suggested models. The settings for this experiment were Google Colab Pro+, macOS Ventura 13.0.1 operating system, Apple M1 Chip, and 8 GB RAM. In addition, the CNN models used in this research were trained with 10 epochs, 0.01 for the learning rate (lr), 0.1 for gamma, and 128 for the batch_size. The two models used the stochastic gradient descent (SGD) as an optimizer with the model parameters, and the momentum was set up to 0.9, as this optimizer produces results more quickly and uses significantly less memory [45]. The parameters for the APE_GAN++ are 0.0002 for the lr and 1 for the input_dim 128 for the batch_size, and it is trained with 10 epochs for both the generator and discriminator. The generator and discriminator were trained with the Adam optimizer.

4. Implementation

This section provides the details of the experimental part, which began with cleaning the dataset and preparing it to be fed into the CNN models. In addition, it clarifies the details of building the CNN models for the IDS. It then presents the well-known white-box attacks that were used to affect the CNN-based models. Finally, the implementation process of the APE_GAN++ model is clarified.

4.1. Implementation Steps

This section goes through the steps that were followed to implement the research solution, which are demonstrated in Figure 4.

4.1.1. Cleaning the CICIDS2017 Dataset

In this research, the CICIDS2017 dataset was used for benchmarking the proposed models. First of all, after downloading the dataset with all eight files, Google Colab Pro+ was used to process and prepare the dataset. There are three steps for cleaning the dataset, which are as follows:

• Reading the files using the (pandas) library, removing the null values, and replacing the infinity values;
• Changing the unrecognized characters;
• Gathering all these files into one table, which will contain (2,830,743, 79) for rows and columns, as well as 15 labels.

Table 3. CICIDS2017 dataset file contents.

Data Files	Number of Benign Samples	Number of Attack Samples
File One	97,718	DDoS = 128,027
File Two	127,537	PortScan = 158,930
Files Three and Four	189,067	Bot = 1966
File Five	288,566	Infiltration = 36
File Six	168,186	Web Attack-Brute Force = 150 Web Attack-XSS = 652 Web Attack-SQL Injection = 21
File Seven	432,074	FTP-Patator = 7938 SSH-Patator attack = 5897
File Eight	440,031	DoS Hulk = 231,073 DoS GoldenEye = 10,293 DoS slowloris = 5796 DoS Slowhttptest = 5499 Heartbleed = 11

below demonstrates the content of each file in the CICIDS2017 dataset.

As presented in Table 3, the first file includes 128,027 DDoS attacks and 97,718 benign samples. The second file includes 158,930 PortScan and 127,537 benign samples. The third file includes 189,067 benign and 1966 Bot samples. Moreover, the benign and Bot samples in the fourth file are identical to those in the third file. In the fifth file, there is an obvious difference in the sampling numbers; hence, there are 288,566 benign samples and only 36 samples for the Infiltration attack. The sixth file is encoded in latin1 format, so it was converted to utf-8 in order to make it similar to other files.

As previously declared, the CICIDS2017 dataset includes 2,830,743 rows and 79 columns, and more resources are consumed to handle all these rows. For this reason, in this experimental research, only 50% of the data were used to evaluate the models, with the inclusion of 1,415,372 rows and 79 columns. Furthermore, some of the dataset classes had fewer instances, such as Web Attack-SQL Injection, Heartbleed, Infiltration, Web Attack-Brute Force, and Web Attack-XSS. For this reason, this research gathered all these classes into one class named Web Attack. As a result, the dataset now contains 11 classes, which improved the results in the testing phase.

As mentioned in Section 2, the CNN model is more effective with pictures. Thus, the CNN model needs some parameters, such as the height and width of the image. At this point, the CICIDS2017 dataset has 79 columns, so we need a height and weight that are suitable for this dimension. It was concluded in this research to use 9 × 9 as the height and width of the image. Thus, the resulting image is equal to 81 pixels, so it is increased with zeros for the original, which is 79 pixels, as illustrated in Figure 5. Furthermore, this research used 1,372,910 data items for training and 42,462 for testing from the dataset.

4.1.2. Building the CNN-Based IDS Models

This research used the same CNN models applied in [9] because it was established that these models had effective results in classification. Therefore, this research used these models, but in the network security domain rather than the computer vision domain. This research also used the CICIDS2017 dataset, which is one of the IDS datasets in the network traffic classification task. As presented in Section 3, these two models include various numbers of convolutional layers, fully connected layers, as well as activation functions. Figure 6 demonstrates these two models in detail. The CNN models take the network packets (status) as input and then outputs the learned features. As a result, the CNN classifiers then utilize the features to determine the likelihood of various classes. The following subsection clarifies the models’ architecture layers.

CNN Models’ Architecture

As Figure 7 shows, the CIFAR_CNN architecture consists of four convolutional layers and three fully connected layers. The Conv2D layer, which is a two-dimensional convolutional layer, takes an arbitrary input from the dataset with a smaller number for the kernel (filter), set here to (3,3). In other words, the convolution procedure moves the kernel across the input while computing how similar the kernel is to the specific input data [45] in addition to the stride, which is a number for the filter movement. In this architecture, it is set to (1,1) based on the others in [9]. Padding in the Conv2D unit is used to prevent information loss at the edges. The padding is set to (1,1) in this architecture for the first convolutional unit, and (2,2) for the other units. In addition, the convolutional unit used batch normalization.

Batch normalization is a method for normalizing the inputs to a layer for each minibatch, which aids in the training of deep neural networks. The number of training epochs needed to train deep networks is significantly decreased by standardizing the inputs, which helps to stabilize the learning process [45]. The last part of the Conv2D unit is the activation function ReLu. To sum up, this CNN model had four convolutional units that consisted of a Conv2D layer, batch normalization, activation, and max-pooling. The second component in this architecture is the fully connected layer (linear). This layer takes the input nodes from the convolutional unit. Using the forward function, these convolutional units are successively connected to the fully connected layers. The final layer will contain 11 output nodes because this problem involves many classes, such as categorizing network traffic as Normal, Web Attack, DoS, etc.

The MNIST_CNN model is similar to the CIFAR_CNN model but uses a smaller number of layers. This model includes two convolutional units and two connected layers. In addition, this model uses dropout with the convolutional unit. The main principle of dropout is to randomly remove units and their connections while training the CNN.

4.1.3. Crafting Adversarial Attacks

This research used the well-known white-box attacks, which are FGSM, BIM, JSMA, DeepFool, C&W, and PGD. As already declared in Section 2, these attacks could cause the IDS to misclassify the network packets, as they evade the CNN-based IDS model, which cannot detect them. First of all, the FGSM attack was used to generate adversaries in the training phase for the models with 0.10 epsilon for the CIFAR_CNN model and 0.15 for the MNIST_CNN model as an impact factor for the attack, which is called the perturbations. Furthermore, the other attacks that were used in this research were applied in the testing phase with the art.attacks.evasion library to evaluate the affection of these attacks against the CNN models.

Parameter Settings

The FGSM attack was used to train the CNN-based IDS-1 (CIFAR_CNN) model with 0.10 epsilon and 0.15 for the CNN-based IDS-2 (MNIST_CNN) model. In addition, the APE_GAN++ model was trained with 0.17 epsilon for the CNN-based IDS-1 (CIFAR_CNN) model and 0.30 for CNN-based IDS-2 (MNIST_CNN). In contrast, the epsilon was set in the testing phase at 0.17 for the APE_GAN++ model with the CIFAR_CNN model and 0.30 for the MNIST_CNN model.

As declared in the previous section, this research used seven different attacks, which were FGSM, BIM, JSMA, DeepFool, C&W, and Auto-PGD.

Table 4. The parameters of the adversarial attacks.

PGD	Max_iter = 10 Batch_size = 32 eps_step = 0.1
Auto-PGD	Max_iter = 10 Batch_size = 32 eps_step = 0.01
FGSM	epsilon = 0.15–0.17
BIM	epsilon = 0.15 eps_step = 0.05
JSMA	Theta = 0.01 Gamma = 0.8
DeepFool	epsilon = 0.17 batch_size = 32 max_iter = 10
C&W	Confidence = 0.1 Binary_search_steps = 10 Max_iter = 10

below demonstrates the parameters for each attack.

4.2. Building the APE_GAN++ Model

As already stated in earlier sections, the primary contribution of this research is the implementation of the recommended defense, APE_GAN++, which is originally based on the GAN to counter adversarial attacks against the IDS. The GAN model consists of two trained models: the generator and the discriminator. In this situation, the generator is responsible for generating adversarial attacks against the training model, which is a CNN-based IDS that suffered an FGSM attack in the training phase. The generator learns the adversaries’ features and does its best to generate the adversaries’ attributes. On the contrary, the discriminator is responsible for distinguishing and then rejecting adversarial attacks. Figure 8 demonstrates the generator and discriminator architecture [9].

In the domain of computer vision, the researchers in [9] applied the CNN model to the MNIST and CIFAR-10 datasets. In this research here, the IDS was built with the CNN model using the CICIDS2017 dataset. The adversarial attacks developed in this research against the CNN model included FGSM, JSMA, BIM, DeepFool, and C&W, in addition to PGD and Auto-PGD. Figure 9 provides an illustration of the proposed model.

As shown in Figure 9, the generator is responsible for generating the adversarial examples. First of all, the generator takes the adversarial attacks with random noise as input. It then generates adversarial attacks against the CNN-based IDS models. The CNN-based IDS models that were trained with the CICIDS2017 dataset are affected by the adversarial attacks that are launched against them by the generator. Consequently, the classification results of the CNN-based IDS models are affected by the adversarial samples, which result in inaccurate labels.

The last part of this model is the discriminator, which takes the adversarial attacks from the generator and the real data as input, and then distinguishes them as fake or real. As is known, the generator and discriminator have an iteration process, and in this scenario, these two parts are connected to the CNN-based IDS, and the discriminator will reject the adversarial attacks that come from the generator before entering the CNN-based IDS model. In other words, the generator generates adversarial attacks to pass them to the discriminator, which filters them, rejects the adversarial attacks and “eliminates them”, and takes the real dataset labels, as the discriminator is trained to discriminate the adversaries’ attributes from the CNN models that were attacked by one of these adversaries. Finally, this operation depends on the discriminator to distinguish between these attacks and reject them before they enter the CNN models.

As a result, the discriminator tries its best to reject all fake samples (affected by the adversaries). After using the discriminator, the CNN-based IDS is more reliable and produces more accurate classifications.

4.3. Experiment Settings and Simulation Environment

This research evaluated the suggested models by using the CICIDS2017 dataset. In the CICIDS2017 dataset, there are eight files in all, with both malicious and benign network packets. These data from the network traffic were collected from the Canadian Institute of Cybersecurity. As mentioned in Section 2, the dataset consists of 83 features and 15 classes of network packets, which are Normal; DoS Hulk; PortScan; DDoS; DoS GoldenEye; FTP-Patator; SSH-Patator; DoS slow loris; DoS slowhttptest; Bot; Web Attack-Brute Force; Web Attack-XSS, Infiltration; Web Attack-SQL Injection; and Heartbleed.

In this paper, all the models were built using Pytorch as the deep learning framework to demonstrate the effectiveness of the models that were suggested. The settings for this experiment were Google Colab Pro+, macOS Ventura 13.0.1 operating system, Apple M1 Chip, 8 GB RAM. Moreover, the CNN models used in this research were trained with 10 epochs, 0.01 for the learning rate (lr), 0.1 for gamma, and 128 for the batch_size. These two models used the stochastic gradient descent (SGD) as an optimizer with the model’s parameters, and the momentum was set up to 0.9, as this optimizer produces results more quickly and uses significantly less memory [45]. The parameters for the APE_GAN++ are 0.0002 for the lr, 1 for the input_dim 128 for the batch_size, and it is trained with 10 epochs for both the generator and discriminator. The generator and discriminator were trained with the Adam optimizer.

5. Results and Discussion

The major findings from applying the models proposed in the framework are presented in this section. First of all, the results for the evaluation metrics of the CNN-based IDS models are presented. Then, the accuracy results of applying the adversarial attacks against the CNN-based IDS models are reported. Furthermore, the findings of applying the APE_GAN++ defense method for eliminating the effects of adversarial attacks on CNN-based IDS models are also presented. Finally, a comparison with existing research studies is also reported.

5.1. Major Findings

The evaluation metrics accuracy, precision, recall, and the F1 score were used to evaluate the proposed models in this research. Then, the proposed CNN-IDS was evaluated in addition to the APE_GAN++ defense model before and after introducing the adversarial samples. Presented in this section are the major findings from applying the proposed models, which consist of three parts: the classification results of the CNN-based IDS models, the effects of adversarial attacks against the CNN-based IDS models, and the performance of the target models after applying the APE_GAN++ defense.

5.1.1. Classifying the Network Traffic Using CNN-Based IDS Models

The experimental findings of building the IDS using the CNN models are presented in this subsection. In [9], the researchers used two CNN models, which were CIFAR_CNN and MNIST_CNN, as classifiers with two different datasets in the computer vision domain. In this research, the same models were applied and evaluated using the CICIDS2017 dataset.

Table 5. Evaluation of the CNN-IDS models.

Accuracy	CNN-Based IDS-1	CNN-Based IDS-2
	97.51	95.43
Precision	97.54	95.39
Recall	97.51	95.57
F1 Score	97.44	95.26
Loss	0.04	0.14

reports the accuracy, precision, recall, and F1 score of the models.

As shown in Table 4, the classification using CNN models resulted in high accuracy. The CNN-based IDS-1 (CIFAR CNN) model achieved more accurate classification results than the CNN-based IDS-2 (MNIST CNN) model because it has more Conv2D layers and fully connected layers. Moreover, the CNN-based IDS-1 classification accuracy was approximately 97.51%, which is thought to be high accuracy. Moreover, the CNN-based IDS-2 obtained a high accuracy of around 95.43%. Overall, the CNN-based models classified network packets as benign or attacks with high accuracy (an average of 96.47%). Figure 10 shows the accuracy, training, and testing losses of the CNN-based IDS-1 model when trained with 10 epochs. Similarly, Figure 11 illustrates the evaluation of the CNN-based IDS-2 model with the same parameters.

As the previous figures show, the CNN-based IDS-1 model had unstable accuracy scores for all epochs in testing. The training accuracy, however, showed comparable results. The CNN-based IDS-2 achieved almost the same results with training and testing across all ten epochs. Overall, in testing and training, these models achieved accuracy scores that exceeded 90%.

5.1.2. Adversarial Attacks against CNN-Based IDS Models

As mentioned in Section 4, this research used seven adversarial attacks, FGSM, JSMA, BIM, DeepFool, C&W, PGD, and Auto-PGD, against the CNN-based IDS models. The adversarial attacks created against the CNN-based IDS models are presented in

Table 6. The classification accuracy of classification models after adversarial attacks.

Adversarial Attack	CNN-Based IDS-1				CNN-Based IDS-2
	Accuracy	Precision	Recall	F1 Score	Accuracy	Precision	Recall	F1 Score
PGD	22.21	65.74	22.52	33.55	11.32	61.51	13.22	21.33
Auto-PGD	3.46	29.42	9.32	14.15	16.06	46.81	18.11	20.49
FGSM	79.54	63.74	77.31	69.87	35.69	58.81	38.28	45.28
BIM	35.10	70.77	42.85	53.38	2.92	17.00	2.54	4.42
JSMA	10.69	49.51	6.79	11.95	55.32	73.45	46.87	54.85
DeepFool	77.00	77.76	85.24	81.33	17.62	47.36	13.79	21.20
C&W	71.24	66.85	67.54	67.19	48.92	78.91	55.08	62.58

.

Table 6 above demonstrates how adversarial attacks can negatively impact the accuracy, precision, recall, and F1 score of the CNN-based IDS models. In addition, the accuracy of the CNN-based IDS-1 model dropped to 79% after the FGSM attack, an 18% decrease. As a result, there was a drop in the accuracy of the network traffic classification. Consequently, the accuracy in the CNN-based IDS-2 model dropped from 60% to 35% after crafting the FGSM attacks.

In fact, all the adversarial attacks affected the models with significant drops in accuracy. The BIM attack had the greatest effect against the CNN-based IDS-2, causing the model accuracy to drop around 3% from 95%. In this case, the CNN-based IDS-2 model makes a random guess between the network traffic (labels). The Auto-PGD attack caused the accuracy of the CNN-based IDS-1 model to drop about 3.46% from 97%. As a result of such drops in the classification accuracy, users cannot rely on the IDS as a defense method against cyberattacks.

5.1.3. APE_GAN++

This subsection presents the evaluation results of the defense model, which is the APE_GAN++ model. The discriminator and the generator were the two main components of the GAN model that were evaluated based on a loss function. Thus, a low value for the loss function of the generator means it will more effectively generate the instances (similar to the original). However, if the discriminator loss decreases, then it will be able to discern the affected instances (fakes) more precisely.

The generator and discriminator losses of the APE_GAN++ were computed using the WGAN-GP loss function, as described in Section 3 of this research.

Table 7. The generator and discriminator losses.

Epochs	CNN-Based IDS-1		CNN-Based IDS-2
	Generator Loss	Discriminator Loss	Generator Loss	Discriminator Loss
0	−0.393417	−0.092697	−0.114110	−0.232262
1	−0.073689	−0.436629	−0.400701	−0.129880
2	−0.053850	−0.464141	−0.426481	−0.140810
3	−0.043958	−0.472849	−0.691791	−0.060987
4	−0.034766	−0.478680	−0.740239	−0.056947
5	−0.031817	−0.477417	−0.562987	−0.060655
6	−0.027683	−0.482876	0.101951	−0.109693
7	−0.025440	−0.488126	−0.034363	−0.329781
8	−0.025312	−0.485837	−0.433175	−0.175218
9	−0.023589	−0.489978	−0.462576	−0.070152

presents the generator and discriminator losses for the two CNN-based IDS models.

The generator and discriminator losses were extremely low when trained with the CNN-based IDS models, as shown in Table 6. The APE_GAN++ model achieved an excellent outcome in generating and distinguishing the adversarial samples as a result. Nearly all the losses of the generator and the discriminator for the CNN-based IDS models were below zero. To evaluate the APE_GAN++, this research applied this approach against CNN-based IDS models that were affected by adversarial attacks.

Table 8. Evaluation of the classification models after APE_GAN++ defense.

Adversarial Attack	CNN-Based IDS-1				CNN-Based IDS-2
	Accuracy	Precision	Recall	F1 Score	Accuracy	Precision	Recall	F1 Score
PGD	81.88	85.61	90.16	87.72	85.20	90.06	92.06	90.36
Auto-PGD	83.74	84.60	86.88	83.72	78.12	73.35	78.99	75.71
FGSM	89.40	90.08	95.83	92.71	80.41	80.76	85.59	82.79
BIM	86.47	89.23	94.11	90.27	80.51	90.81	88.09	86.25
JSMA	85.00	84.22	89.07	86.05	84.86	85.27	85.82	82.26
DeepFool	82.56	90.05	89.34	85.91	86.81	87.08	87.40	87.22
C&W	83.54	73.91	77.77	71.51	87.15	83.56	84.25	82.35

shows the CNN-based IDS models’ accuracy, precision, recall, and F1 score after applying the APE_GAN++ defense.

In fact, the APE_GAN++ defense enhanced the accuracy of the CNN-based IDS models in meeting the research goal. As shown in Table 8, almost all the values in the results for each accuracy, precision, recall, and F1 score increased and exceeded 80% after applying the APE_GAN++ defense. Furthermore, the APE_GAN++ yielded excellent results against the other white-box attacks crafted in this research. The APE_GAN++ defense clearly performs quite well with the IDS in the network security domain. Figure 12 and Figure 13 demonstrate the accuracy of the CNN-based IDS models after various adversarial attacks, as well as after APE_GAN++ defense.

As seen in the preceding, the CNN-based IDS-2 was more significantly impacted than the CNN-based IDS-1 by the PGD and BIM attacks, although both models were clearly affected. Hence, all adversarial attacks had impacts on the models, but with varying degrees of accuracy. The APE_GAN++ model is superior to other defense methods, as it can address all types of attacks and fit all domains [9]. To demonstrate this here, the APE_GAN++ defense was applied to enhance the IDS accuracy, which was affected by the adversarial attacks. As shown in Figure 12 and Figure 13 the APE_GAN++ significantly enhanced the model accuracy. The accuracy of the classification models was increased to between 78 and 89 percent, which is promising for indicating excellent classification. The APE_GAN++ works effectively with the C&W attack, which was crafted against the CNN-based IDS-2 model. Similarly, the APE_GAN++ achieved good results against the FGSM attack that was crafted against the CNN-based IDS-1 model.

Under the APE_GAN++ defense against the Auto-PGD, BIM, JSMA, DeepFool, and C&W attacks, the CNN-based IDS-1 achieved reasonably comparable accuracy results. On the contrary, the PGD had the lowest accuracy score for the CNN-based IDS-1 model at roughly 81%. Retrieving the CNN-based IDS-2 model after the Auto-PGD attack had the lowest accuracy at roughly 78.12% among all other attacks. Overall, the APE_GAN++ can defend against adversarial attacks, which is encouraging, as it can distinguish and then effectively eliminate the adversarial perturbation. Thus, the accuracy for almost all the models reached over 80%, which is considered to be reasonable accuracy for classification models.

5.2. Comparison with Related Works

This subsection discusses related studies. Furthermore, it consists of two parts: a comparison with the related studies that used CNN models for the IDS, and the related studies that used the APE_GAN++ model for defense.

5.2.1. Comparison with Existing CNN Models

This subsection compares the proposed models with the related works. First of all, a CNN was used in this research as a classifier for the network traffic IDS. In [9], the authors used two CNN models, which were also used in this research, although with a different dataset (CICIDS2017). Thus,

Table 9. Comparison between the classification models’ accuracy.

This Research	CNN Models	Accuracy
	CNN-based IDS-1 (CIFAR_CNN)	97.96
	CNN-based IDS-2 (MNIST_CNN)	95.64
Base Line Research [9]	CIFAR_CNN	83.98
	MNIST_CNN	98.51

presents a comparison between the CNN models in different domains.

All these results are of the model accuracy in classification before adversarial attack. As presented above, the CIFAR_CNN (CNN-based IDS-1) model works more effectively with the CICIDS2017 dataset, which means it is more suitable in the network security environment. In contrast, the MNIST_CNN (CNN-based IDS-2) achieved high accuracy with the MNIST dataset rather than the CICIDS2017 dataset used in this research. In general, all these models achieved high accuracy in classification, which suggests that the CNN can potentially produce remarkable results in domains other than simply image classification. Furthermore, the CICIDS2017 dataset has more samples than the CIFAR-10 and MNIST datasets; hence, it performs well even with 10 training epochs.

5.2.2. Comparison with the Existing APE_GAN++ Model

As stated in the previous section, this research used a defense approach based on the GAN. This approach was used with an image classification domain and achieved impressive results. In Ref. [9], the authors built two CNN models with two different datasets and then crafted five well-known white-box attacks against these models. In addition, they applied their suggested defense strategy to eliminate these adversaries and enhance the model accuracy, which decreased after the attacks. Thus,

Table 10. General comparison between the APE_GAN++ study and this research.

Study	Number of Adversarial Attacks	Attack Method	Dataset	Evaluation Metrics	Number of Epochs	Domain
[9]	5	FGSM BIM JSMA DeepFool C&W	MNIST CIFAR-10	Accuracy	100	Computer Vision
Proposed work	7	- Auto-PGD - PGD - FGSM - BIM - JSMA - DeepFool - C&W	CICIDS2017	- Accuracy - Precision - Recall - F1 score	10	Network Security

presents a general comparison between the APE_GAN++ study and this research.

The study in [9] used two CNN models for classifying the images. In addition, these models were trained with 100 epochs, while this research trained the CNN-based IDS models with 10 epochs. In the training process for the CNN-based IDS models, good results were achieved even with only 10 epochs, as the model was trained using the CICIDS2017 dataset, which includes more data samples than MNIST and CIFAR-10. Furthermore, the study in [9] crafted five white-box attacks against the CNN models and notes that the APE_GAN++ can defeat any type of attack. For this reason, this research crafted all these attacks against CNN-based IDS models. In addition, it crafted two well-known white-box attacks, which were PGD and Auto-PGD, against the CNN-based IDS models to evaluate the APE_GAN++ defense.

Table 11. Comparison of accuracy between classification models.

CIFAR-CNN [9] and MNIST-CNN [9] Models’ Accuracy	CNN-Based IDS-1 and CNN-Based IDS-2 Models’ Accuracy
The accuracy of the CIFAR-CNN model varied from 74.20% to 82.03% after applying the APE_GAN++ model for all 5 attacks. In contrast, the accuracy of the MNIST_CNN varied from 76.13% to 98.83%.	The accuracy of the CNN-based IDS-1 model varied from 82.26% to 89.40% after applying the APE_GAN++ model for all 5 attacks. In comparison, the accuracy varied from 80.41% to 87.15% for the CNN-based IDS-2.

presents the experimental results for applying the APE_GAN++ to the CNN models in [9] and in this research.

As shown in Table 11, the accuracy of the CNN models in classification differs from one model to another. The APE_GAN++ produces excellent results with the MNIST-CNN model, but it is not reasonable that the model accuracy is higher than before being affected by the attack, which is 98.51%.

In other words, the accuracy of the MNIST-CNN model before adversarial attacks was 98.51%, and after crafting adversarial attacks, it dropped to 1.00%, but increased to 98.83% after the APE_GAN++ defense, which is more than the original accuracy. Thus, from a research perspective, the model accuracy after applying the defense must be greater than after it is affected by the attack but not exceed the model accuracy before the attack.

Nevertheless, it did not perform as well with the CIFAR-CNN model as it did with the MNIST-CNN model. In this research, the CNN-based IDS-1 model had good results compared with the CIFAR-CNN model of the same architecture. Comparative to the MNIST-CNN model, the CNN-based IDS-2 obtained lower accuracy. In comparison to previous protection strategies, the APE_GAN++ generally improves the model accuracy after adversarial attacks [9].

6. Limitations and Future Work

In this research, a defense approach was implemented to enhance the accuracy of the IDS and make it more resilient against adversarial attacks. The following are limitations that could be improved in future work.

6.1. Limitations

• Processing the whole dataset and building the models takes more time and demands more resources;
• The IDS datasets had a lack of balance, with significant bias for normal instances; for example, the CICIDS2017 dataset used to evaluate this research had more normal instances than attack instances.

6.2. Future Work

• The APE_GAN++ defense fits all the ML/DL models in other fields. Thus, using APE_GAN++ in other domains could be helpful to increase the accuracy of ML/DL models and render them immune to all types of white-box attacks;
• Evaluation of the APE_GAN++ defense against other types of attacks, such as black-box attacks, is needed;
• Collecting a new dataset for the IDS, as there is a need for a balanced dataset in the IDS domain;
• In the model architecture stage, the results could be enhanced by changing some of the activation functions and number of epochs, and by training the models with more data samples.

7. Conclusions

In cybersecurity, the use of ML/DL algorithms is attracting much attention, especially in intrusion detection systems (IDSs). Moreover, adversarial attacks can have a significant impact on DL algorithms. Hence, DL-based IDSs are vulnerable to adversarial attacks that spark security concerns. In such a situation, the IDS is unreliable in defense, posing a severe threat to networks. In order to enhance the sustainability of the IDS and increase its resilience, the APE_GAN++ defense method was implemented in this research.

First, in this research, two CNN models based on an IDS were built using the CICIDS2017 dataset. Second, seven white-box attacks were crafted against these models. Lastly, the APE_GAN++ was evaluated by implementing it with the affected models (the CNN-based IDS models). The model accuracy after these attacks exhibited an intelligible drop. Some of these attacks affected the models and caused the accuracy to drop to 2.92%, which means that the models randomly classified the network traffic.

APE_GAN++ increased the model’s accuracy up to 80% against almost all of these attacks. For evaluating the APE_GAN++ framework, this research compared the results of the baseline research and this research. The APE_GAN++ defense shows a clearly encouraging result, and it is also a promising approach for the future.