1. Introduction
The last few decades have shown significant growth in the interconnectedness of almost all parts of life. The proliferation of ever-faster wired and wireless network connections combined with advances in energy-efficient and powerful mobile processors has led to a massive deployment of ubiquitous Internet of Things (IoT) devices.
The COVID-19 pandemic in 2019, with its restrictive social distancing measures, accelerated the push towards a much more digitalized society [
1]. In weeks, the whole economic sector and most of the education sector were forced into predominantly digital interactions. Mobile healthcare services were also swiftly put in place. The long-lasting impacts of these transformations are hard to predict, but digitalization will most probably continue to occur.
The transition from a pre-digitalized society towards a mostly digital one yields tremendous opportunities for improved and more economical services. However, the dependence on digitally connected devices and services renders us susceptible to attacks on the underlying network and its connected appliances.
Attackers of the systems above can be classified into three attack scenarios [
2]. Based on the attacker’s motivation, resources, and the complex nature of the attack, it is possible to distinguish between cybercrime, cyber-terrorism, and cyber-warfare. Cybercrime is often motivated by financial incentives. Cybercriminals often indulge in ransomware or phishing attacks. Cyber-terrorism aims to spread terror to push some form of ideological agenda. Cyber-warfare is funded or executed by nation-states and often shows high sophistication.
The presence of cyberattacks makes it necessary to implement mechanisms to protect digital infrastructures and ensure the secure and continuous operation of the corresponding services. Hardening digital services to protect them from malicious activities can take many forms, such as intrusion detection systems (IDSs) that analyze network traffic to detect suspicious activities or intrusion prevention systems (IPSs) that can act upon the detection of a potential attack.
Traditional intrusion detection systems use specially crafted patterns to match certain network traffic characteristics to detect malicious activities. Such systems produce deterministic and dependable results, but the maintenance and creation of the patterns take lots of time and effort. Furthermore, a growing number of detection patterns increases the workload of the underlying network infrastructure and can reduce throughput. In [
3], machine learning algorithms can learn complex patterns reliably while detecting and classifying various attack types.
Another advantage of machine learning algorithms is their inherent capability to generalize and thus correctly classify an input, even if it differs from the initially provided training data.
Since about 2012 [
4], there has been a renewed interest in machine learning. This renewal in interest was mainly driven by the development of new training methods like stochastic gradient descent and by advances in graphic processing units (GPUs) capable of dealing with computationally complex tasks like the training of deep neural networks [
5]. Advances in graphical object detection and natural language processing driven by ever deeper neural networks led to the now conventional wisdom that good results in machine learning is only possible with deep and complex neural architectures. However, these deep neural networks pose a sizeable challenge when using them in edge computing devices or with high-throughput applications, such as switches in a network infrastructure. Both applications (edge devices and switching infrastructure) came with their own set of specific challenges. Modern switching infrastructure has to cope with high throughput and short latencies. Limiting the computation spent for each packet is necessary to satisfy those requirements.
Embedded systems are often subject to tight energy budgets and possess limited processing power. An example of embedded systems that can profit from machine learning-powered attack detection algorithms is distributed autonomous systems like satellite constellations as described in [
6]. Detecting attacks on such connected edge devices can reduce reaction times and thus help to improve network security. However, using deep neural networks on edge platforms is very challenging, and applications are often transferred to some form of cloud server. Communication with the necessary cloud services generally consumes a considerable amount of energy and imposes additional processing latencies and financial costs for the communication services. In addition, securing the interaction with the cloud provider is necessary, which further increases the processing and communication burden on the edge device. In a high-throughput environment, calculations must be performed quickly. Deep neural networks may perform millions of computations for a single inference step, which would take a long time to complete on low-power edge devices. By contrast, using more powerful processors is costlier and generally incompatible with edge devices’ power and energy budget.
In this paper, we show that a deep neural network may not be necessary to treat the problem of network flow analysis for attack detection. The reported results show that a shallow neural network can perform equally well as competing deeper neural architectures. Using a shallow network reduces the overall processing cost and energy consumption. An additional advantage of a shallow neural network is its low computational latency, which can be further improved by parallelization.
Our main objective in this article is to analyze the traffic of a computer network to detect malicious activities. We use flow statistics that characterize the network traffic to achieve this goal. Typical characteristics of network flow entries are the mean packet size, the mean time between two packets, the total number of packets exchanged, or the number of packets per second. The advantage of using network flow characteristics is that they do not contain packet payloads, which are often encrypted and can combine multiple elementary packets into a single set of characteristics, further limiting the amount of data to analyze.
Typical attacks that can be detected are, for instance, denial-of-service attacks, where the attacker tries to bind a maximum of server resources to prevent the victim from servicing other clients [
7]. Another type of attack is a port scan. In this attack, the attacker tries to find open ports on the victim’s server. When the attackers find such an open port, they check in their database whether known vulnerabilities are associated with them. When attackers successfully identify a vulnerable system, they try to exploit the vulnerability to compromise the corresponding system. Yet another type of attack consists of guessing a username and password for a standard service like FTP, SSH, or any other type of restricted access.
Our objective is to find the best fully connected neural network to detect and classify such types of attacks on edge devices.
The main contributions of the present paper are as follows:
Proposing a shallow neural network capable of detecting and classifying network attacks for various network applications.
Proposing an improvement of the attack-sharing loss function proposed by Dong et al. [
8].
Proposing a straightforward architecture search strategy for a low-complexity neural network.
Showing the applicability of our approach with different intrusion detection training datasets representing general Internet use as well as Internet of Things and industrial applications.
Section 2 presents the datasets used for this article. This section contains a detailed description of the CIC-IDS2017 and its derived Lycos dataset, and a brief overview of additional datasets used for extended testing of our proposed solution. In
Section 3, we explore the problem of imbalanced learning, which is associated with all significant intrusion detection training datasets. Based on the observed problem of imbalanced learning, we will provide an overview of possible mitigation strategies and present our proposed improvement on the attack-sharing loss function, proposed by Dong et al. [
8].
Section 6 introduces the different metrics used to determine a neural network’s performance. In
Section 7, we present the performances achieved with our proposed solution. We will discuss our results in
Section 8. Finally,
Section 9 summarizes the paper’s main contributions and concludes this work.
5. Implementation
We have defined a strategy to find an efficient MLP that can detect network attacks on the edge; in the next step, we implemented a data preparation pipeline to manage out-of-range dataset values such as infinite or Not a Number. We chose to replace infinite values with the corresponding column’s maximum value. Missing values, indicated by the Not a Number (NaN) value, were replaced with the column’s median value by applying the Scikit-learn SimpleImputer class. Further, the data were normalized by subtracting the mean value and dividing by the corresponding standard deviation. For this normalization step, we used the Scikit-learn StandardScaler class.
To improve confidence in the generalization capacity of the model, we used the Scikit-learn StratifiedKFold class for a 5-fold cross-validation. We further split the holdout set into two equal parts using the Sciekit-learn train_test_split function with the stratify option activated. The advantage of the stratified shuffle split is that the train, test, and validation datasets contain the initial proportion of samples for each class label. For the Lycos-IDS2017 dataset with all attack classes, this approach however led to the problem that for some class labels, the holdout set did not contain enough samples for the split. We therefore applied in this case a standard stratified shuffle split to obtain an 80% training, 10% validation, and 10% test dataset.
The MLP is implemented in Pytorch. We use the rectified linear unit (ReLU) as activation function for the artificial neurons in the single hidden layer. The advantage of the rectified linear unit (Equation (
5)) is its very low computational complexity. The ReLU activation function can be implemented by combining a comparison and a value assignment operation.
We use the Softmax activation function for the output nodes, shown in Equation (
6).
The training was performed with the standard stochastic gradient algorithm. The values of the weights and biases were initialized with the HeNormal initializer [
23]. As an optimizer, we chose the Nadam optimizer from the Pytorch library, which had a learning rate of 0.001.
We used a custom early stopping to prevent overfitting that monitored the validation loss and F1 score.
6. Performance/Complexity Measures
We use the confusion matrix to judge each neural network’s performance. A confusion matrix shows the results of multiple inferences typically made with the test dataset at the end of the training to evaluate the model’s performance.
Figure 4 shows a schematic representation of a confusion matrix for a binary classification problem. Every cell of such a matrix contains the number of samples classified according to the corresponding class. The columns of a confusion matrix represent the true value given in the test dataset. The rows represent the model’s prediction. The binary case has two correct classifications: true negatives (
) and true positives (
). These values are located on the diagonal of the confusion matrix. They represent cases where the model could correctly classify the input data.
The number of negative samples wrongly classified as positive is called the number of false positives (). The number of positive values wrongly classified as negative is called the number of false negatives ().
Several performance metrics exist based on the values shown in a confusion matrix. The recall metric represents a ratio of the true positive values and the total number of positive class samples, as shown in Equation (
7). The recall metric can be interpreted as the capacity of the network to predict the positive class.
The precision metric, as shown in Equation (
8), is the ratio of the number of true positive predictions relative to the total number of samples classified as positive. The precision metric can be interpreted as the quality or pureness of the prediction. A precision value close to one means that only a few false positive predictions exist.
It is possible to derive a combined measure by applying the harmonic mean. This type of performance metric is called F1-Score, shown in Equation (
9). The F1-Score always lies between the two performance metrics, precision and recall. It gives a good overall measure of the neural network’s performance.
For neural networks with more than two classes, it is also possible to calculate the performance using the abovementioned metrics. To obtain a global measure for precision, recall, and the F1-Score, it is necessary to calculate the individual metrics for each class first. The number of false positives for the
i-th class is, in this case, the sum of all values in the column representing the corresponding class without the value at
, which represents the true positives value (
). The formula for the number of false positives is shown in Equation (
10), where
N is the number of classes and
the entry of the confusion matrix at row
j and column
k.
The value for the false negatives of the i-th class can be calculated by adding the values of the i-th row of the confusion matrix except for the value at
, as depicted by Equation (
11).
From these values, it is possible to calculate the individual performance metrics. The mean value of the individual performance metrics is then calculated to obtain a global performance measure for the multi-class neural network.
7. Results
We have applied our search strategy and produced different MLPs of different sizes. We can see the results in
Figure 5. The left part of the figure shows the performance measurements of precision, recall, and F1-Score for different numbers of neurons in a single layer. It shows that initially, the performance increases rapidly with the number of neurons when that number is increased. The graph shows that the network achieved very good results with only a few neurons. However, we observed that the performance still increased with additional neurons. The network achieves very good performance with 110 neurons in the hidden layer. If we increase the number of the neurons, the supplementary computational cost increases according to Equation (
12), where
is the number of additional neurons in the hidden layer,
is the number of neurons of the input layer, and
is the number of neurons of the output layer.
This leads to necessary memory allocations of
weights and an increase in terms of clock cycles that is proportional to
to obtain a very small gain (less than 0.1). Therefore, we concluded that 110 artificial neurons in the hidden layer appear to be a good trade-off to minimize network complexity while maintaining a high detection rate. Following the proposition in [
8] and to allow deeper architectures to be explored more rapidly, we limited the number of neurons in each hidden layer to 100. The graph shows that the performance of a network with a single hidden layer is already quite high (F1-Score of 0.9985). It is also apparent that additional layers cannot yield much better results. On the contrary, additional layers appear to degrade the neural network’s performance.
The graph on the right-hand side of
Figure 5 shows the neural network’s performance for different depths, trained on the Lycos-IDS2017 dataset with all attacks included. The single-layer network performs quite well, with an F1-Score of 0.9199. However, additional layers significantly degrade the overall performance of the network.
The two confusion matrices in
Figure 6 show that our improved attack-sharing loss function improved the excellent results obtained with the original basic cross-entropy function. These results indicate that the additional regularization term successfully reduced the false negative rate of the classification caused by the disproportion of the training dataset towards the benign case. The confusion matrix on the left also shows a slight increase in false positives for the improved attack-sharing loss, representing a trade-off of our proposed solution. In terms of security, the false negative rate is more detrimental to the overall network security than the false positive rate, as falsely benign classified network traffic spreads into the network and can cause damage. These results, furthermore, show that a shallow neural network can classify the different attack classes in the simplified Lycos dataset. Based on this MLP architecture exploration, we chose a single-layer MLP.
Table 3 shows the performance measurements of our single-layer MLP trained with the original simplified CIC-IDS2017 dataset and the simplified version of the improved Lycos-IDS2017 dataset (first two rows). We compare our results with those published by Boxiang Dong et al. [
8] for a network with 10 layers of 100 neurons each. The reported results are the mean results of the published individual scores, since Dong et al. did not publish a global performance measurement.
The results in the first two rows show that the improved Lycos-IDS2017 dataset yields better results than the original CIC-IDS2017 dataset. A second result that is shown in
Table 3 is that our shallow neural network can substantially outperform the much deeper architecture.
Table 4 shows a comparison between our results obtained with a shallow neural network consisting of a single hidden layer with 110 ReLU-activated artificial neurons and the results proposed by Rosay et al. [
10] who implemented an artificial neural network consisting of two hidden layers, one of 64 and a second one of 32 SELU-activated neurons. To obtain the results in
Table 4, we trained our proposed neural network on the complete Lycos-IDS2017 dataset without simplification. This way, all attack classes participated in the training.
The first row of
Table 4 contains the mean values of the individual performance measurements, as described in
Section 6. The second row shows the performance of the aggregated confusion matrix. To calculate these values, we combined the results of all attacks while keeping the benign case separate. This way, we obtained a binary confusion matrix that distinguishes only the two cases, benign and attack. The third row of
Table 4 shows the performance measurements of the aggregated confusion matrix as they were presented by Rosay et al. [
10]. Comparing
Table 4 results show that our network could successfully classify network flows with an average F1-Score of about 92% If we compare our shallow MLP solution with the two hidden layers MLP from Rosay et al. [
10], rows 2 and 3, we see that they achieved similar performances. They obtained around 99% accuracy, confirming that shallow MLPs can classify network attacks as effectively as deeper networks. An advantage of a single hidden layer is that the entire hidden part of the inference step can be efficiently parallelized and accelerated. In addition, our shallow MLP uses the ReLU activation function. This activation function can be very effectively implemented. Rosay et al. use the SELU function, which implies calculating an exponential function in combination with several floating-point multiplications. Those mathematical operations are computationally very costly. Thus, our algorithm would run faster on most embedded systems.
Figure 7 shows the confusion matrix for the test results of our proposed neural network for the simplified Lycos-IDS2017 dataset (left side) and the complete Lycos-IDS2017 dataset (right side). For the simplified dataset, it shows that we achieved very good results, with only a few samples wrongly classified.
The confusion matrix for the complete dataset with all attacks shows that most attack classes were very well classified. The “webattack_bruteforce” class, however, was less well detected. Difficulties to detect this attack class were also reported in [
24]. Another aspect highlighted in the confusion matrix is that only a few of the provided attack samples were classified as benign. The small values visualize this in the first column of the confusion matrices shown in
Figure 7. The small number of attacks wrongly classified as benign indicates a low false negative rate, which is important for an intrusion detection system, as already explained.
Table 5 shows the individual precision and recall performance measures for each class of the complete Lycos-IDS2017 dataset obtained with our shallow neural network with 110 ReLU-activated neurons in a single hidden layer.
Table 6 compares our results for two additional intrusion detection datasets, WUSTL-IIOT-2021 and Bot IoT, with performance scores reported in the literature. All three considered datasets were used to find the structure that produced the best performance with our proposed architecture and search algorithm. The best classification performance was obtained with approximately 110 neurons in all cases. Therefore, we chose the same model size of 110 artificial neurons for all datasets. We trained a model for each intrusion detection dataset, utilizing early stopping to prevent overfitting.
Figure 8 shows the receiver operating characteristic (ROC) curve for the corresponding datasets. The curves show that we achieved excellent classification results for the Lycos-IDS2017 and Bot IoT datasets. However, the curve for the WUSTL-IIOT-2021 dataset shows that the model had difficulties classifying the backdoor attacks. This particular attack type had the smallest number of samples in the training dataset, which may have contributed to the decreased classification performance for this specific flow category.
For the training datasets from Washington University in St. Louis and the UNSW Institute for Cybersecurity, we obtained an F1-Score well above 90 percent.
Our F1-Score of 0.9564 for the WUSTL-IIOT-2021 dataset is well above the F1-Score of 0.832 reported by Gaber et al. [
25]. However, the authors only used the 17 most significant dataset features for their neural network in their paper. This, on its own, results in a considerable reduction in complexity but also comes with a decrease in model performance. Alani et al. [
26]’s DeepIIoT architecture achieved an F1-score of 99%, but they only performed a binary classification into benign and attack traffic categories. However, classifying the nature of an attack can be advantageous when it comes to mitigation strategies, and thus might justify a slight decrease in the detection rate. Alsamiri et al. [
29] reported an F1-Score of 0.83 for their multilayer perceptron implementation for the Bot IoT dataset. The authors only used the seven best-performing features. The comparison of those results with ours shows again the tradeoff between performance and accuracy. Koroniotis et al. [
27,
28] obtained an F1-Score with a binary classifier for the Bot IoT dataset of 0.999 with a much deeper neural network.
The bottom part of
Table 6 presents the classification results for the respective datasets with the two commonly used algorithms K Nearest Neighbor (K-NN) and Random Forest (RF) as presented in the literature. For the WUSTL-IIOT-2021 dataset, our proposed solution outperformed the classification results presented by Gaber et al. [
25]. We achieved an F1-Score close to the cited K-NN score for the Bot IoT dataset. Alsamiri et al. [
29] also presented an F1-Score of 0.97 for the Adaboost and ID3 algorithms. It is also worth noting that the K-NN algorithm has the highest computational cost and, therefore, the longest runtime.
8. Discussion
This article presents several contributions to finding an optimal MLP to detect and classify network attacks on the edge. We have proposed an MLP search strategy consisting of an iterative approach to successively add neurons and layers to the MLP, which is then trained and its performance evaluated to find a neural architecture that, on the one hand, maximizes performance and, on the other hand, minimizes the computational cost of the network. This search strategy led to the surprising result that an MLP with a single hidden layer performs the best in the researched task. This finding contrasts the widely accepted wisdom that only deep neural networks can perform well on complex tasks.
To deal with the inherent imbalanced nature of attack detection in digital communication networks, we have improved the special attack sharing loss function [
8] to further reduce the number of false negatives. The false negative rate is an essential characteristic of an intrusion detection system because attacks labeled as benign traffic cannot be further processed to refine and validate (or invalidate) the detection eventually. We have used a shallow MLP generated through a proposed simple search strategy to analyze different datasets representing different communication protocols on the Internet and in an IoT context.
We have shown that our shallow MLP performs well and could perform even better on the corrected CIC-IDS2017 dataset proposed by Rosay et al. [
10], which further underlines the importance of well-designed training datasets in machine learning. It achieved a 99.85% F1-Score on the much more balanced simplified subset of the Lycos-IDS2017 dataset. This is a very good result that allows us to consider implementing our shallow MLP in an actual intrusion detection system.
On the considerably imbalanced complete Lycos-IDS2017 dataset, we achieved an F1-Score of 91.99%, which is significantly lower than the results of the simplified dataset. However, we calculated the overall performance measures as the mean value of the individual performance metrics. Such an approach makes the resulting global measure quite sensible to the performance of the minority classes. We could also have used a weighted mean (as provided in the Scikit-learn library) to calculate the global metrics, which would have resulted in a higher performance score because of its emphasis on the majority classes. Nevertheless, this would have hidden that the model struggles with some minority classes. Our presented results thus give a more conservative judgment of the actual model performance.
When the objective of the neural network is more focused on detecting an attack than the actual classification, it is advantageous to consider measurements based on the aggregated confusion matrix. Comparing the reported results of the last two rows of
Table 4 shows that we were able to further improve on the performance published in the literature.
Another objective was to evaluate the capacity of our proposed neural network solution to classify network traffic in different networking scenarios. To validate our approach, we chose various network intrusion detection training datasets. The CIC-IDS2017 and its derived Lycos dataset represent a more general networking scenario of everyday Internet usage. In contrast to those two datasets, the WUSTL-IIOT-2021 and Bot IoT datasets include different types of smart devices and industrial applications. Those additional networking applications introduce new network protocols and connection patterns to the network traffic classification and attack detection task. Our results have shown that the proposed shallow neural network can detect attacks with a high F1-Score for both networking scenarios.
The presented results encourage us to extend our research in the future to further networking scenarios and to an implementation of our proposed neural network for the detection of network attacks in an embedded environment. Exploring additional model compression techniques like bit reduction or feature search and engineering also seems promising for successfully implementing our neural network on an embedded device. Another promising future research direction is combining our classification solution with an anomaly detection algorithm that could trigger alerts for possible threats not currently detected by the proposed network. An additional interesting research direction will be the combination of our approach with secure communication using Blockchain technology [
30].