Robust Intrusion Detection for Industrial Control Systems Using Improved Autoencoder and Bayesian Gaussian Mixture Model

Abstract

Machine learning-based intrusion detection systems are an effective way to cope with the increasing security threats faced by industrial control systems. Considering that it is hard and expensive to obtain attack data, it is more reasonable to develop a model trained with only normal data. However, both high-dimensional data and the presence of outliers in the training set result in efficiency degradation. In this research, we present a hybrid intrusion detection method to overcome these two problems. First, we created an improved autoencoder that incorporates the deep support vector data description (Deep SVDD) loss into the training of the autoencoder. Under the combination of Deep SVDD loss and reconstruction loss, the novel autoencoder learns a more compact latent representation from high-dimensional data. The density-based spatial clustering of applications with noise algorithm is then used to remove potential outliers in the training data. Finally, a Bayesian Gaussian mixture model is used to identify anomalies. It learns the distribution of the filtered training data and uses the probabilities to classify normal and anomalous samples. We conducted a series of experiments on two intrusion detection datasets to assess performance. The proposed model performs better than other baseline methods when dealing with high-dimensional and contaminated data.

1. Introduction

An industrial control system (ICS) is a general term that encompasses several types of control systems [1], such as supervisory control and data acquisition systems, and distributed control systems. Typically, these systems are employed in industries such as power plants, water, and wastewater facilities. With advancements in information technology, ICSs have been connected to the internet for increased efficiency. However, this has exposed ICSs to numerous security threats [2]. Attacks on ICSs can disrupt their normal operations, resulting in financial losses and potentially affecting the daily lives of humans. In recent years, several attacks against ICSs have been reported [3,4,5]. Therefore, it is crucial and urgent to develop security solutions to safeguard ICSs, and intrusion detection systems (IDSs) play a significant role in this regard [2].

Based on the detection technique used, the IDS can be divided into a misuse-based detection method or anomaly-based detection method [2,6]. A misuse-based detection method compares established attack patterns to new events to evaluate if the new events are attacks or not. This approach has a greater detection rate, but it is incapable of detecting unknown threats. The anomaly-based technique detects anomalies by constructing a normal profile from ICS data. Anomalies are defined as events that deviate from the profile. Anomaly-based detection methods can be divided into three categories [6]: statistics-based, knowledge-based, and machine learning-based. In this paper, we focus on IDS based on machine learning techniques.

To address the detection of anomalies using labeled datasets, some supervised machine learning techniques have been developed [7,8], such as random forest and decision tree. However, gathering attack data to create a labeled dataset is difficult and expensive [9]. As a result, training the detection model with only normal data makes sense. One frequently employed approach is the one-class support vector machine (OCSVM) [10,11]. However, these methods are affected by the curse of dimensionality, which could reduce the detection rate. Before training the detection model, some researchers suggested using the autoencoder (AE) to reduce the dimension first [9,12]. By rebuilding the input as much as possible, the AE learns a compressed latent representation. Furthermore, AE could detect anomalies by using the reconstruction error, which has a higher value for anomalous samples [13].

In addition to the issue of high-dimensional data, the presence of outliers in the training set also has a negative impact on performance. When outliers are included, models such as OCSVM [14] or AE [15] may also learn the distribution of outliers, which degrades performance. Motivated by the work of the deep support vector data description (Deep SVDD) neural network [16], which transforms data into a hypersphere where outliers are outside, an improved autoencoder (IAE) [17] was proposed. By incorporating the Deep SVDD loss during the training phase of AE, the IAE learns a compact representation. However, utilizing the distance calculated in the latent space makes it challenging to identify some outliers that may not be entirely excluded from the hypersphere. The Deep SVDD neural network can apply to the samples directly. But this approach suffers from the problem of hypersphere collapse [16].

In this paper, we propose a hybrid anomaly detection method to address the problems of high-dimensional data and the existence of outliers in the training set. In general, the novel model introduces two steps before anomaly detection: feature extraction and outlier removal. Motivated by the work [9,18], which uses the AE to extract features first, we apply the IAE to obtain latent representations. To further decrease the effect of outliers within the training set, we employ the density-based spatial clustering of applications with noise (DBSCAN) algorithm to remove some potential outliers. Finally, the distribution of the filtered dataset is modeled using a Bayesian Gaussian mixture model (BGMM). The following is a summary of the contributions:

• With the aim of extracting useful features from high-dimensional data, IAE is used to produce more compact latent representations. In detail, the Deep SVDD loss is combined with the reconstruction loss during the training of the IAE. The new latent representation improves the effectiveness of the anomaly detection model.
• With the improved latent representation, we first utilize DBSCAN to remove the potential outliers within the training set in order to decrease its impact on detection performance. BGMM is trained using the filtered training set, which uses probability to identify abnormalities. The outlier removal step benefits for the BGMM and improves detection performance.
• To show the effectiveness of the suggested methodology, we run a series of experiments on two intrusion detection datasets. Our method demonstrates more promising detection performance than some other baselines in terms of effectiveness and robustness. Additionally, an ablation study proves the improvement provided by both steps mentioned above.

The remainder of the paper is organized as follows. In Section 2, we review related works involving anomaly detection. We provide a full introduction to the proposed methods in Section 3. Then, the effectiveness of the suggested approach is discussed in Section 4. Finally, the relevant conclusions are presented in Section 5, together with directions for future research.

2. Related Work

In recent years, ICS security issues have received a lot of attention. Attacks against ICSs, such as Stuxnet [19] and BlackEnergy3 [20], can cause devastating consequences when compared to traditional information systems. IDS plays an important role in providing security. In particular, with the development of machine learning and deep learning, many research studies [8] have been proposed. Supervised machine learning methods can be used to classify the normal data and attacks when there are labeled datasets. However, gathering attack data is challenging [9], in contrast, collecting normal data is simple. This paper primarily reviews anomaly detection methods that use normal data only for training. In an anomaly-based intrusion detection method, the model is trained using normal data in order to learn a normal profile. Samples will be classified as abnormal if they differ from the established profile.

As a one-classification method, OCSVM is frequently used in the field of anomaly detection. It finds a hyperplane that separates normal data from the origin with a maximum margin. In [10], an improved OCSVM was developed. Modbus traffic packets were used to train the OCSVM model to generate a normal communication pattern, and a particle swarm optimization method was used to optimize the parameters of OCSVM. Ensemble learning and OCSVM were combined to create the intrusion detection technique named IT-OCSVM [11]. The authors combined the predictions of several OCSVM models using mean majority voting. The performance of detection using the suggested method is promising. However, OCSVM suffers from the issue of high-dimensional data [16].

Many deep learning-based anomaly detection works have been proposed [21,22,23]. One way to apply deep learning is by using it as a feature extraction module [9,24]. AE is a special kind of deep learning model that works to reconstruct the input as accurately as possible. To extract discriminative features for IDS, a stacked sparse AE is used [25]. Experiments have shown that it is effective in feature extraction, and it alleviates the performance degradation caused by high-dimensional data [9,12]. The reconstruction error can also be utilized as the anomaly score to detect anomalies directly. By utilizing the stacked denoising AE [26], malicious packets within ICSs can be detected.

These approaches, however, work under the presumption that the training set contains only normal data. However, it is hard to maintain in a practice environment. The anomaly detection model may also learn the anomalous samples well when the training dataset is contaminated by outlier samples. The performance of anomaly detection can be affected as a result, as shown in [17]. For example, in AE-based methods, the reconstruction error of anomalous samples may decrease to a level similar to that of normal data as the number of training epochs increases [15]. The Deep SVDD neural network [16] aims to learn a mapping that transforms normal data into a hypersphere while minimizing the volume of the hypersphere. Anomalies are identified as samples that are located outside of this hypersphere. However, the Deep SVDD neural network suffers from the problem of hypersphere collapse [16]. By adding a constraint supervised by the Deep SVDD loss for the latent space, Zhen Chen et al. [17] proposed IAE, which aims to position the anomalous samples outside the hypersphere in the latent space. As for the anomaly score, it uses the distance between the center and each sample in the latent space. Distance may not be a useful metric to distinguish between normal and anomalous data as some outliers might be inside the hypersphere. However, this method is effective for learning latent representations. In general, these two methods try to create a novel loss function for the neural network while considering the existence of outliers.

To reduce the effect of anomalous samples in the training dataset, ref. [15] proposed the use of an adversarial AE. They introduced an outlier labeling procedure during training using OCSVM. The identified outlier would not affect the training of normal samples. However, the adoption of the OCSVM may cause an increase in training time.

Based on the studies mentioned above, there are two views on dealing with high-dimensional and contaminated datasets. The first is the powerful capability of deep learning for feature extraction, which facilitates anomaly detection through representation learning. The other is an operation to minimize the influence of outliers in the training set. Our proposed method takes into account both of these factors.

3. Method

In this section, we present the proposed IDS model. We begin by introducing the entire detection framework. The IAE is then presented. In the end, a robust anomaly detection approach is developed using the latent representations derived from the IAE.

3.1. Overview

Before delving into the details of each part, we provide an overview of the proposed method, which is depicted in Figure 1. In general, there are two phases, training and testing, as shown in the figure.

There are three steps during the training phase. Given the ICS traffic features, we developed an IAE, which derives the compressed representations of original features. The combination of reconstruction loss and Deep SVDD loss is used to train the IAE. We used DBSCAN to remove the potential outlier points because their presence would negatively affect the performance of the anomaly detection model. Finally, a BGMM was trained to learn the data distribution using the filtered training set.

We applied the learned model to identify the anomalies in the testing set after training. Unlike the training process, the BGMM inspects the latent representation of the testing dataset directly during testing. The BGMM generates higher anomaly scores for anomalous samples compared to normal data. Each component of the proposed methodology for anomaly detection is thoroughly described in the subsequent subsections.

3.2. Improved Autoencoder

Compared with the traditional AE, the IAE incorporates the Deep SVDD as part of the loss function to learn a compact data representation with minimized volume. The AE and Deep SVDD are described further below, and the IAE is introduced at the end.

3.2.1. Autoencoder

The AE is a special type of neural network. The input data go through several hidden layers in the AE. It aims at reconstructing the input as much as possible. It has been applied in the field of feature extraction widely [27].

In detail, AE can be divided into two parts: the encoder and the decoder. Considering a set of input data $\mathit{X}=\{{\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_N\}$, where N is the number of samples in the dataset, the encoder learns a map function $\varphi$ for each sample ${\mathit{x}}_i$ and outputs compressed latent representation ${\mathit{z}}_i$. The decoder then attempts to reconstruct the input data from the compressed latent representation, ${\mathit{z}}_i$, and outputs the reconstructed data, ${\hat{\mathit{x}}}_i$. The training process of the AE aims to find the parameters that minimize the reconstruction loss. In this study, the mean squared error is used to calculate the reconstruction loss, which is given in (1).

$$L_{rec}={\parallel {\widehat{\mathit{x}}}_i-{\mathit{x}}_i\parallel }^2$$

(1)

When employing an AE to extract features from a dataset, the decoder is abandoned after training, only the encoder is maintained.

3.2.2. Improved Autoencoder

Support vector data description (SVDD) [28], like OCSVM, is a one-class classification method. When carrying out tasks for anomaly detection, the SVDD seeks to find the smallest hypersphere that encloses most of the normal samples. Lukas Ruff et al. [16] proposed the Deep SVDD neural network, which combines the SVDD with the neural network. The Deep SVDD introduces the SVDD as the loss function for the training of the neural network. The trained neural network learns a mapping function, and within the output space of the mapping function, most normal samples are enclosed by a hypersphere.

Let $\mathsf{\Phi }(·;\mathit{W})$ be the transform function of the neural network, with $\mathit{W}$ denoting the set of hidden layer weights. The $\mathsf{\Phi }({\mathit{x}}_i;\mathit{W})$ is the output of the neural network for sample ${\mathit{x}}_i$. The loss function of the Deep SVDD neural network is defined in (2).

$$L_{ds}=R^2+\frac{1}{\nu N}\sum _{i=1}^{N}max\{0,\parallel \mathsf{\Phi }({\mathit{x}}_i;\mathit{W})-\mathit{c}{\parallel }^2-R^2\}$$

(2)

The first term refers to the volume of the hypersphere with radius R. The second term is a penalty for points whose distance from the center $\mathit{c}$ is greater than R. The hyperparameter $\nu$ controls the trade-off between the volume of the hypersphere and the number of points outside it. Deep SVDD neural networks aim to map normal samples into a hypersphere with minimal volume based on the loss function. When using Deep SVDD for anomaly detection, the distances to the center $\mathit{c}$ can be calculated as anomaly scores.

But the Deep SVDD neural network suffers from the problem of hypersphere collapse [16]. When this happens, the output of the neural network for every input is constant, and the hypersphere radius R collapses to zero. However, we can use it as a regularizer to constrain the latent space of AE [17]. In this way, the encoder produces compact latent representations with minimized volume. Combining the two loss functions, the architecture of IAE is displayed in Figure 2. As seen in the figure, the Deep SVDD loss is applied to the output of the encoder only. Considering the map function $\varphi$ of the encoder, we can rewrite the joint loss function

$$L=L_{rec}+\lambda L_{ds}={\parallel {\widehat{\mathit{x}}}_i-{\mathit{x}}_i\parallel }^2+\lambda \{R^2+\frac{1}{\nu N}\sum _{i=1}^{N}max\{0,\parallel \varphi ({\mathit{x}}_i;\mathit{W})-\mathit{c}{\parallel }^2-R^2\}\}$$

(3)

where $\lambda$ is the weight controlling the relationship between reconstruction loss and Deep SVDD loss. It is worth noting that $\mathit{W}$ represents the layer weights within the encoder. Next, we describe the training algorithm in more detail, as shown in Algorithm 1. We start with k warm-up epochs during the training phase, where the AE is trained solely on the reconstruction loss. This allows the encoder to generate a stable latent representation. After that, the center value, $\mathit{c}$, is calculated as the mean value of the encoder output for all training data. The IAE is then trained with both Deep SVDD loss and reconstruction loss simultaneously. During training, we update the parameters of the neural network and update R. After calculating the distances $\mathit{S}$ between the encoder output and the center $\mathit{c}$, we calculate the new R using the distance $\mathit{S}$, following the approach in the original paper [16].

Algorithm 1: Training process of IAE.

   The latent representations are obtained once the IAE has been trained. We created an anomaly detection method based on the latent representation in the next subsection.

3.3. Anomaly Detection Based on DBSCAN and BGMM

In the previous section, we used the IAE to learn a compact representation of data. The distance within the latent space can be used to detect anomalies [17], but its performance is not satisfactory. Considering the presence of outlier points in the training set, we adopt a hybrid method for anomaly detection. First, we remove the outliers from the training set using DBSCAN. Next, we use a BGMM model to fit the remaining training data. In the following subsections, we provide a detailed explanation of our method.

3.3.1. Density-Based Spatial Clustering of Applications with Noise

DBSCAN is a clustering method [29,30]. Its intuition is to find the high-density regions in the sample space. The radius, $ϵ$, and the number of neighbors, $minPts$, are two important pre-defined parameters. Within the radius, if the neighbors of one observation are higher than the $minPts$ (including the observation itself), the regions can be classified as high-density regions. In the meantime, some points within the low-density regions would be considered outliers.

In detail, with the DBSCAN algorithm, samples in the dataset can be classified into three types of points: core points, border points, and outlier points. Core points are those with $minPts$ neighbors within a radius of $ϵ$. The neighbors of core points belong to the same cluster of core points, and they are density-reachable. The points within the cluster that are not core points are called border points. Outlier or noise points are those that are not density-reachable by any core points. An example can be seen in Figure 3a, where there are eight points in a simple example, and two clusters can be concluded. In the meantime, two noise points are recognized. To illustrate the DBSCAN algorithm further, we create a synthetic dataset with three clusters. The clustering results are displayed in Figure 3b. Some points with a larger distance from other points have been classified as noise.

Presuming that the normal ICS data locate in higher density regions than outliers, we can employ DBSCAN to filter the outlier points first. With the latent representations $\mathit{Z}=\{{\mathit{z}}_1,{\mathit{z}}_2,\cdots ,{\mathit{z}}_N\}$, we can use the DBSCAN algorithm to label these observations as normal or noise. We denote the label results using $Y=\{y_1,y_2,\cdots ,y_N\}$, where $y_i\in \{0,1\}$. The points are given "label 1" if they are outlier points and "0" if they are core or boundary points. We compute the neighbors of each point and identify core points using the pre-defined parameters $minPts$ and $ϵ$. The neighbors of core points and core points themselves are then connected to form clusters. Finally, a point is considered noise if it does not neighbor any core points.

The two parameters in the algorithm significantly affect the clustering results. As a rule of thumb, the $minPts$ can be set as $2\ast d$ [29], where d is the dimension of input data. However, in our study, we conducted experiments on a dataset consisting of tens of thousands of samples. It is reasonable to set it at a higher value. In this study, the $minPts$ are set at a certain percentage of the number of training sets. Then, using the k-distance graph, where $k=minPts$, we select the value of epsilon [30]. We calculate the k-nearest neighbor distance for each observation and then sort the distances from lowest to highest. After that, we compute the “elbow” point in the k-distance graph using the Kneedle algorithm [31], and we utilize this value as $ϵ$.

3.3.2. Bayesian Gaussian Mixture Model

We first filter the training set using the label results Y, keeping the observations $z_i$ with $y_i=0$. After that, the BGMM is used to learn the distribution for the cleaned training dataset. Gaussian mixture model (GMM) is a probabilistic model that is commonly used for clustering. In this study, we use it to model the normal data and detect anomalous samples with probability density. We lay out the foundations of GMM first.

GMM attempts to fit its probability density using a linear combination of K Gaussian distributions (also known as components) with the observation $\mathit{z}$ extracted from the IAE. Its equation can be written as

$$p\left(\mathit{z}\right)=\sum _{k=1}^K{\pi }_k\mathcal{N}\left(\mathit{z}\right|{\mathbf{\mu }}_{\mathbf{k}},{\mathsf{\Lambda }}_k^{-1})$$

(4)

where the $\mathcal{N}\left(\mathit{z}\right|{\mathbf{\mu }}_{\mathbf{k}},{{\mathsf{\Lambda }}_{\mathbf{k}}}^{-1})$ represents the k-th components with the parameter ${\mathbf{\mu }}_{\mathbf{k}},{\mathsf{\Lambda }}_{\mathbf{k}}$. The ${\mathbf{\mu }}_{\mathbf{k}}$ and ${\mathsf{\Lambda }}_{\mathbf{k}}$ are the mean and precision matrices, respectively, and parameter ${\pi }_k$ is the mixing coefficient, which satisfies ${\sum }_k{\pi }_k=1$ and $0\le {\pi }_k\le 1$. Let $\mathit{v}$ denote the K-dimensional latent variable corresponding to $\mathit{z}$. The latent variable $\mathit{v}$ is a binary random variable satisfying $v_k\in \{0,1\}$ and ${\sum }_k{v}_k=1$. Taking the mixing coefficients into account, the distribution of $\mathit{v}$ can be written as

$$p\left(\mathit{v}\right)=\prod _{k=1}^K{\pi }_k^{v_k}$$

(5)

Let the filtered dataset be denoted by $\mathit{Z}=\{{\mathit{z}}_1,{\mathit{z}}_2,\cdots ,{\mathit{z}}_N\}$ where N is the number of samples in the filtered dataset, and let all latent variables be denoted by $\mathit{V}$; considering the parameters and latent variables, the conditional distribution of observations is given by

$$p\left(\mathit{Z}\right|\mathit{V},\mathbf{\mu },\mathsf{\Lambda })=\prod _{n=1}^N\prod _{k=1}^K\mathcal{N}{\left({\mathit{z}}_n\right|{\mathbf{\mu }}_k,{\mathsf{\Lambda }}_k^{-1})}^{v_{nk}}$$

(6)

In the maximization likelihood framework, we can use the expectation–maximization (EM) algorithm to estimate the parameters. However, there are some limitations to using the EM algorithm to estimate the parameters of the GMM. One issue is that it suffers from the presence of singularities [32]. In this study, we adapt the Bayesian version of GMM and use variational inference to estimate the parameters of the GMM. This approach treats all of the parameters as random variables and assigns prior distributions to them, using conjugate prior distributions to simplify calculations. The Dirichlet distribution and Gaussian–Wishart prior are introduced for $\mathbf{\pi }$, ${\mathbf{\mu }}_k$, and ${\mathsf{\Lambda }}_k$, respectively, as shown in

$$p\left(\mathbf{\pi }\right)=\mathrm{Dir}\left(\mathbf{\pi }\right|{\mathbf{\alpha }}_0)$$

(7)

and

$$p(\mathbf{\mu },\mathsf{\Lambda })=p\left(\mathbf{\mu }\right|\mathsf{\Lambda })p\left(\mathsf{\Lambda }\right)=\prod _{k=1}^K\mathcal{N}\left({\mathbf{\mu }}_k\right|{\mathbf{m}}_0,{\left({\beta }_0{\mathsf{\Lambda }}_k\right)}^{-1})\mathcal{W}\left({\mathsf{\Lambda }}_k\right|{\mathbf{a}}_0,b_0)$$

(8)

where $\mathcal{W}$ is the Wishart distribution, and $\{{\mathit{\alpha }}_{\mathbf{0}},{\mathit{m}}_0,{\beta }_0,{\mathit{a}}_0,b_0\}$ are the parameters of the prior distribution. In the Dirichlet distribution, we use the same parameter, ${\alpha }_0$, for each component. The goal of variational inference for BGMM is to find an approximation of the posterior distribution $p(\mathit{V},\mathbf{\pi },\mathbf{\mu },\mathsf{\Lambda }|\mathit{Z})$. To simplify, we denote $\mathsf{\Theta }$ as $\{\mathbf{\pi },\mathbf{\mu },\mathsf{\Lambda }\}$. We refer to the [32]; the log marginal likelihood can be decomposed by

$$lnp\left(\mathit{Z}\right)=\mathcal{L}\left(q\right)+\mathrm{KL}(q\parallel p)$$

(9)

where the functional $\mathcal{L}\left(q\right)$ is the lower bound of $lnp\left(\mathit{Z}\right)$. Moreover, the $\mathrm{KL}(q\parallel p)$ is the Kullback–Leibler (KL) divergence between the variational posterior distribution $q(\mathit{V},\mathsf{\Theta })$, and the true posterior $p(\mathit{V},\mathsf{\Theta }|\mathit{Z})$. These equations are defined as

$$\mathcal{L}\left(q\right)=\sum _V\int q(\mathit{V},\mathsf{\Theta })ln\left\{\frac{p(\mathit{Z},\mathit{V},\mathsf{\Theta })}{q(\mathit{V},\mathsf{\Theta })}\right\}d\mathsf{\Theta }$$

(10)

$$\mathrm{KL}(q\parallel p)=-\sum _V\int q(\mathit{V},\mathsf{\Theta })ln\left\{\frac{p\left(\mathit{V},\mathsf{\Theta }|\mathit{Z}\right)}{q\left(\mathit{V},\mathsf{\Theta }\right)}\right\}d\mathsf{\Theta }$$

(11)

We attempt to maximize the $\mathcal{L}\left(q\right)$ with respect to the $q(\mathit{V},\mathsf{\Theta })$, which is equivalent to minimizing the $\mathrm{KL}(q\parallel p)$. When the $q\left(\mathit{V},\mathsf{\Theta }\right)$ is approximated as the true posterior distribution $p\left(\mathit{V},\mathsf{\Theta }|\mathit{Z}\right)$, we have the maximum of the lower bound [32]. With some necessary calculations (for detailed calculations, readers can refer to [32]), the approximate posterior distribution is shown below. We list the detailed variables in the equation in the Appendix A.

$$q^{★}\left(\mathit{V}\right)=\prod _{n=1}^N\prod _{k=1}^K{r}_{nk}^{v_{nk}}$$

(12)

where $r_{nk}$ plays the role of responsibilities. Moreover, the optimization of $\mathbf{\pi }$ and $({\mathbf{\mu }}_{\mathbf{k}},{\mathsf{\Lambda }}_k)$ are calculated as below.

$$q^{★}\left(\mathbf{\pi }\right)=\mathrm{Dir}\left(\mathbf{\pi }\right|\mathbf{\alpha })$$

(13)

$$q^{★}({\mathbf{\mu }}_k,{\mathsf{\Lambda }}_k)=\mathcal{N}\left({\mathbf{\mu }}_k\right|{\mathbf{m}}_k,{\left({\beta }_k{\mathsf{\Lambda }}_k\right)}^{-1})\mathcal{W}\left({\mathsf{\Lambda }}_k\right|{\mathbf{a}}_k,b_k)$$

(14)

With the optimal distribution analysis above, we can estimate the parameters by following the steps in the EM algorithm. After the parameters in the prior distribution are initialized, we calculate the (12) in the E-step. Then in the M-step, we keep the responsibilities fixed and use these values to update the distribution in (13) and (14). By updating the parameters while cycling between the two steps until convergence, we can estimate the parameters of BGMM.

3.3.3. Anomaly Detection

We introduce the whole training algorithm in Algorithm 2 by combining the DBSCAN and BGMM. We first filter the noise points in the latent representation ${\mathit{z}}_i$ of the training set. We use the Kneedle algorithm to calculate the value of $ϵ$ using the given parameters $minPts$. The training set is then labeled by DBSCAN, and the labeled results indicate whether the samples are considered noise points. To train the BGMM model, we take samples $z_i$ from the training set where their labeled results are $y_i=0$ by DBSCAN. With some necessary parameters in the prior distribution initialized, we begin the E-step. The M-step is then carried out to update the parameters. The E-step and M-step are repeated before the convergence. Finally, we have a well-trained BGMM model.

Algorithm 2: Training process of the combination of DBSCAN and BGMM.

In the testing phase, we first extract latent representations by IAE for the test samples. Then, they are inspected by the well-trained BGMM. The anomaly score is finally calculated using the negative log-likelihood produced by the BGMM, where the anomalous samples have a higher value than the normal samples.

4. Evaluation

In this section, we evaluate the performance of the proposed method. The datasets utilized in the experiments are presented first. After that, the comparative methods that serve as the baselines are described. Finally, we introduce the experimental setup and analyze the performance of the proposed method.

4.1. Dataset

In this study, we conducted the experiments using two datasets: the Gas Pipeline dataset [33] and NF-BoT-IoT-v2 dataset [34], in order to assess the performance of the proposed method.

The first dataset, Gas Pipeline [33], was collected using a laboratory-scale gas pipeline environment. Some attacks have been used to simulate anomalous samples. There are six types of attacks, NMRI, CMRI, MSCI, MPCI, DoS, and Reconnaissance. In total, there are 61,156 normal samples and 35,863 anomalous samples. Each sample involves one captured network transaction pair, including network traffic features and payload content features. For every observation, there are 26 numeric features.

The second dataset, NF-BoT-IoT-v2 [34], was created by the Cyber Range Lab of the Australian Center for Cyber Security using Ostinato and Node-RED tools. Apart from normal samples, there are four attacks: DoS, DDoS, Reconnaissance, and Theft. Since there are a large number of records, we randomly sample the records in the dataset, just as in our previous work [35]. There are 65,150 normal samples and 78,598 anomalous samples after sampling. Every record in the dataset is a NetFlow-based feature. Except for some category features, we apply the log function to numeric features to decrease the effect of large feature differences. Additionally, the category features are handled using the one-hot encoder method. There are roughly 200 features left after preprocessing.

Every observation in both datasets is scaled using the min–max normalization into the range of [0, 1]. We divided the dataset into a training set and a testing set with a 4:1 ratio. In order to simulate the contaminated dataset, we remove all anomalous data from the training set. After that, we randomly sample data from the removed anomalous data with a specific ratio (referring to the anomaly ratio) with respect to the remaining training set. The pure training set is then combined with the chosen anomalous samples to form a new training set. It should be noted that we trained the model without any help from the label during training. We used the testing set directly, as it already contained both normal and anomalous data.

4.2. Compared Baselines

We compare the performance of our method against seven baselines, including four traditional machine learning methods and three deep learning-based methods, as listed below.

• OCSVM. In the experiments, we used the radial basis function kernel as the kernel function. We selected $\nu$ from $\{0.01,0.1\}$, and $\gamma$ from $\{1\times {10}^{-5},1\times {10}^{-4},$$1\times {10}^{-3},$$1\times {10}^{-2},1\times {10}^{-1},1\}$. The best results are reported.
• Isolation forest (IForest). We set the hyperparameters, the samples, and the number of estimators as the original paper recommended, which were 256 and 200, respectively.
• Kernel density estimation (KDE). The kernel function also uses the radial basis function kernel. The bandwidth was set as $0.001$.
• BGMM. The BGMM was used to show the influence of the feature extraction and outlier removal steps. Its hyperparameters were set to be the same as our proposed method.
• AE. The reconstruction error was used as the anomaly score. The network architecture and hyperparameters of the AE are identical to those in our method, except for Deep SVDD loss.
• Deep SVDD. As in the original paper, we trained an AE first, and the encoder was used to train the Deep SVDD neural network. During training, the value of $\nu$ was set to 0.1.
• IAE. IAE trains the data with reconstruction loss and Deep SVDD loss. The distance between samples and the center in the latent space is used as the anomaly score. The IAE uses the same settings as our method.

For comparison, we directly used the trained IAE model in our method, which means that both the IAE and our method use the same latent representations. Therefore, we can demonstrate that the BGMM can produce a better anomaly score than the distance calculated from the IAE.

4.3. Experiment Settings

The suggested method in this study was implemented using the Python programming language. The PyTorch framework was used to program the IAE and other deep learning methods. All neural network methods employed Adam [36] as their optimizer with a learning rate of 0.001. The batch size was set to 256, and the epochs to 200. With the aim of a warm start, the first 20 epochs were trained with reconstruction loss only. A weight of 1 × 10⁻⁵ was used for the Deep SVDD loss. The value of $\nu$ in the Deep SVDD loss function was set to 0.1.

For hidden layer settings in the IAE, we used a five-layer hidden layer architecture. The architecture used by the encoder and decoder is symmetrical. For the NF-BoT-IoT-v2 dataset, we set “144-80-16-80-144” in the hidden layer. Moreover, the Gas Pipeline dataset was “20-14-8-14-20”. We used PReLU [37] as the activation function in the hidden layer. The activation function in the output layer was a sigmoid function.

We implemented DBSCAN, BGMM, and other traditional machine learning methods using scikit-learn [38]. In terms of the $minPts$ in the DBSCAN, we observe that larger values may decrease the performance because they lead to more outliers being labeled as normal. As a result, we set the value as a fraction of the training set chosen from $\{1\%,5\%\}$ and report the best results. Since BGMM could choose the optimal components by itself, with the components that provide insufficient contributions removed [32], we fixed the number of components to 10, and the number of training iterations to 100.

We used the area under the receiver operating characteristic curve (AUROC), which is frequently employed in the task of anomaly detection, as the evaluation metric. To guarantee accurate results and prevent randomness during the training phase, we repeated each approach five times.

4.4. Results

We examine the effectiveness of the suggested approach in the following content of this section. We first contrast it with various baselines. Additionally, several anomaly ratios are utilized to demonstrate their robustness in handling contaminated datasets. To demonstrate the effects of the different components of our method, we conducted thorough ablation research. Finally, we examined the impact of the settings for the hyperparameter.

4.4.1. Detection Performance

To begin the performance analysis, we report the results of two lower anomaly ratios, 5% and 10%. The total performance results, which include the mean value and standard deviation of the AUROC for each approach, are shown in

Table 1. AUROC comparisons with different baseline methods for two datasets. There are eight methods, including our proposed method in the first column. Both mean values and the standard deviation of the AUROC are reported.

Method	Gas Pipeline		NF-BoT-IoT-v2
	5%	10%	5%	10%
OCSVM	$86.89\pm 0.26$	$86.42\pm 0.06$	$60.02\pm 1.39$	$58.57\pm 0.92$
KDE	$87.04\pm 0.28$	$74.27\pm 0.30$	$86.40\pm 0.11$	$83.83\pm 0.14$
IForest	$88.51\pm 0.44$	$87.89\pm 0.50$	$85.53\pm 0.46$	$81.80\pm 0.56$
BGMM	$89.65\pm 1.33$	$79.75\pm 5.22$	$77.17\pm 5.06$	$72.16\pm 5.03$
AE	$83.55\pm 6.17$	$73.84\pm 8.51$	$84.91\pm 2.45$	$82.80\pm 4.11$
Deep SVDD [16]	$66.24\pm 5.61$	$64.19\pm 4.67$	$70.51\pm 6.69$	$63.20\pm 4.25$
IAE [17]	$76.90\pm 4.81$	$78.03\pm 2.18$	$77.33\pm 4.88$	$80.39\pm 3.69$
Ours	$\mathbf{89}.\mathbf{89}\pm \mathbf{2}.\mathbf{96}$	$\mathbf{88}.\mathbf{04}\pm \mathbf{0}.\mathbf{91}$	$\mathbf{92}.\mathbf{00}\pm \mathbf{0}.\mathbf{73}$	$\mathbf{91}.\mathbf{73}\pm \mathbf{1}.\mathbf{06}$

Bold font indicates best results.

.

The detection performance of our method shows the greatest performance for both datasets, as can be seen from the table above. The majority of methods show a decreasing trend when the anomaly ratio rises from 5% to 10%. First, we analyze the results for the Gas Pipeline dataset. The four traditional machine learning methods outperform the remaining three deep learning baselines for this dataset. When the anomaly ratio is 5%, BGMM produces the best results out of these four methods. Despite an increase in the anomaly ratio to 10%, OCSVM and IForest maintain greater values. IAE performs well for both anomaly ratios, and it shows a higher performance compared to AE when the anomaly ratio is 10%. However, Deep SVDD performs poorly compared to other methods.

When it comes to the NF-BoT-IoT-v2 dataset, both OCSVM and BGMM perform worse than they do in the Gas Pipeline dataset, which may be due to the curse of dimensionality. AE continues to perform well in terms of both anomaly ratios. The performance of Deep SVDD is also the lowest in this dataset. In this dataset, KDE and IForest both maintain higher values. However, compared to KDE and IForest, our method shows greater improvement.

We conducted more experiments with more varied anomaly ratios to further demonstrate the robustness of the proposed method. The performances for both datasets are shown in Figure 4, with the anomaly ratio increasing from 5% to 25% with a step of 5%.

Since it can be difficult to eliminate some anomalous samples with higher anomaly ratios, most techniques generally exhibit decreasing trends for both datasets. The IAE and AE display different patterns in Figure 4a. IAE performs better than AE in general. However, compared to other promising baselines, such as IForest and OCSVM, our approaches have not shown significant improvement. For the NF-BoT-IoT-v2 dataset, the experimental results are displayed in Figure 4b. In most cases, OCSVM produces the worst results. Our method exhibits a greater improvement than the best baseline method, KDE.

In conclusion, our proposed detection method exhibits robustness and offers promising detection results with a contaminated dataset, compared with other baseline methods.

4.4.2. Ablation Study

As previously stated, we improved the approach from two aspects. One is the compact latent data representations generated by the IAE, and the other is the outlier removal step by DBSCAN. In this part, we perform an ablation study to demonstrate the effects of these two parts. There are five variants, depending on whether or not IAE or DBSCAN is used, including methods that use original features or AE to extract features. The difference between each method can be inferred from the name. The detailed comparisons for the two datasets are shown in Figure 5, where the x-axis is the anomaly ratio used to exhibit the robustness for dealing with contaminated datasets.

For the Gas Pipeline dataset in Figure 5a, our approach exhibits comparable detection performance to the method employing the original features with DBSCAN. Since the dimension of this dataset is not particularly high, employing the original features can also produce good results. Additionally, the introduction of DBSCAN demonstrates improvement; for instance, our methods maintain higher accuracy than the “IAE + BGMM” method for all anomaly ratios.

In the next part, we analyze the NF-BoT-IoT-v2 dataset with higher dimensions and more detail. Figure 5b shows a higher difference between different methods than the results of the Gas Pipeline dataset. Starting with the simplest method, BGMM, which learns the distribution directly, it yields the poorest outcomes compared to the other approaches. When adding AE or IAE into the feature extraction step, both methods have an increment of 10% over BGMM only under different anomaly ratios. These two methods have similar results as their curves almost entirely overlap. However, since there are outliers in the training set, both methods suffer from performance degradation.

Both methods utilizing AE and IAE show improvement when the DBSCAN module is introduced. The effects of IAE are also starting to show. As can be seen from the figure, our methods produce better results than those utilizing AE alone because the outliers have been removed and IAE has more compact representations than AE. However, the improvement is lower when the anomaly ratio is higher. It should be noted that the methods using DBSCAN on the original features produce promising results when the anomaly ratio is lower, indicating the importance of the outlier removal step. However, when the anomaly ratio rises to 20% or 25%, the results of this method become worse. With a higher anomaly ratio, it is more difficult to remove outliers from high-dimensional data. Compared to the "DBSCAN + BGMM" method, our method has two advantages with the introduction of IAE: it has a more representative distribution and reduces computational costs by reducing dimensions.

In conclusion, the IAE and DBSCAN modules both enhance the performance of anomaly detection, especially with the higher-dimensional dataset.

4.4.3. Analysis of Hyperparameters

In the proposed method, we automatically set some hyperparameters, such as the radius of DBSCAN or the number of components in BGMM. However, the $minPts$ of DBSCAN is determined empirically. As previously stated, this value is very important for finding outliers. Therefore, we analyze this parameter using different anomaly ratios.

When $minPts$ is set to a value that is too high, most samples will be merged into one cluster. As we compute the $ϵ$ using the Kneedle algorithm, its value will increase with the increase of $minPts$ in most cases. Some outliers may be included in clusters with bigger $minPts$ and $ϵ$ and are incorrectly labeled as normal. We set the value of $minPts$ as a proportion of the number of the training dataset, ranging from 0.5% to 5% with the 0.5% step. For simplicity, we only run tests on the NF-BoT-IoT-v2 dataset. A comparison of the results is shown in Figure 6.

Overall, the AUROC shows a decreasing trend as the anomaly ratio or the number of $minPts$ increases. The AUROC remains greater than 0.80 for all alternative values of $minPts$, with an anomaly ratio of no more than 15%. When the $minPts$ parameter is set to 0.5% or 1%, the maximum value for AUROC is obtained. However, we observed that DBSCAN incorrectly labeled some normal samples as noise, which indeed degraded the performance. This suggests that we can further improve the detection rate with optimized parameters for DBSCAN, and we leave this as a direction for future work.

5. Conclusions and Future Work

Due to the increasing cyber security threats to ICSs, it is critical to develop efficient security solutions. Anomaly detection-based IDS are essential tools for ensuring security. However, the high-dimensional data and presence of outliers in the training set pose challenges to detection accuracy. In this study, we propose a hybrid intrusion detection model to address these issues.

The proposed method improves the effectiveness of anomaly detection in two aspects. The IAE in the first part derives data representations from ICS traffic features. In the second part, before training the anomaly detection model, we introduce an outlier removal step. In more detail, we provide IAE that incorporates the Deep SVDD loss function during training. Together, the additional loss and reconstruction loss enable the IAE to learn a more compact latent representation. Based on the improved latent representation, the DBSCAN method is employed to first remove the outlier from the training set. This process is important because it reduces the influence of outliers. The BGMM eventually learns the probability distribution for the training set. The results of the experiments conducted on two intrusion datasets reveal that the proposed method outperforms other baseline methods in terms of the detection rate and robustness to outliers.

There are numerous possibilities for the future direction of our work. Despite the removal of outliers from the training set by DBSCAN, as previously mentioned, it also incorrectly classified some normal data, negatively impacting the performance because some normal data could not be learned. Using an advanced optimization method to select parameters for DBSCAN would be important to reduce the number of false positives. Additionally, by using measurements such as the reconstruction error or distance during the training phase of the AE, we could directly remove some outliers with greater confidence. In this way, during training, we could reduce the effect of outliers on the latent representation of normal data. Finally, the normal data could be modeled using a more advanced model than the BGMM.