Mutation-Based Multivariate Time-Series Anomaly Generation on Latent Space with an Attention-Based Variational Recurrent Neural Network for Robust Anomaly Detection in an Industrial Control System

Abstract

Anomaly detection involves identifying data that deviates from normal patterns. Two primary strategies are used: one-class classification and binary classification. In Industrial Control Systems (ICS), where anomalies can cause significant damage, timely and accurate detection is essential, often requiring analysis of time-series data. One-class classification is commonly used but tends to have a high false alarm rate. To address this, binary classification is explored, which can better differentiate between normal and anomalous data, though it struggles with class imbalance in ICS datasets. This paper proposes a mutation-based technique for generating ICS time-series anomalies. The method maps ICS time-series data into a latent space using a variational recurrent autoencoder, applies mutation operations, and reconstructs the time-series, introducing plausible anomalies that reflect multivariate correlations. Evaluations of ICS datasets show that these synthetic anomalies are visually and statistically credible. Training a binary classifier on data augmented with these anomalies effectively mitigates the class imbalance problem.

1. Introduction

Anomaly detection is the task of predicting anomalous data that exhibit different patterns from normal data. It plays a crucial role in various fields such as finance [1], industrial control systems (ICS) [2], and cybersecurity [3]. There are two main approaches to classifying anomalies from normal data: one-class classification and binary classification. (1) One-class classification involves training a classifier using only one type of data (typically normal data) to predict anomalous data [4,5]. This strategy is commonly adopted in most domains because anomalous data is often rare and difficult to collect compared to normal data. (2) On the other hand, if anomalous data are available, the anomaly detection problem can be reduced to binary classification [6,7]. In this setting, a prediction model is trained with both normal and anomalous data.

One-class classification-based anomaly detection models are generally known to have a high false alarm rate [8]. The primary reason for this drawback is that the prediction model struggles to learn a sophisticated decision boundary due to the absence of anomalous instances. Consequently, researchers attempt to address the anomaly detection problem through binary classification, aiming to accurately separate anomalous instances from normal data. Most of this research concentrates on independently and identically distributed (i.i.d) data, with only limited studies addressing anomaly generation for time-series data.

The lack of research on time-series anomaly generation exacerbates the issue of building robust anomaly detection models in ICS environments. ICS is core to various domains like the manufacturing industry, electrical grids, and transportation systems. These systems share a closed nature, which makes data collection challenging. Moreover, the data inherently have time-series characteristics. As a result, this leads to data imbalance, making it difficult to train robust anomaly detection models and consequently reducing the reliability of performance evaluation. Therefore, in this paper, we focus on generating pseudo-time-series anomalies in the ICS environment.

Based on our observation, there are three major challenges in generating time-series anomalies. (1) Unlike i.i.d data, time-series data exhibit dependencies between data samples over time. In other words, when generating anomalies for time-series data, it is crucial to reflect this temporal dependency; (2) In the case of multivariate data, an anomaly in one feature can affect other features due to the correlations between them. However, systematically calculating these correlations and assigning values manually is not scalable [9]; (3) Anomalous data are inherently rare compared to normal data, making it difficult to determine an accurate probability distribution for these data. This scarcity complicates the process of generating data through sampling from a probability distribution.

To overcome the identified challenges, our insights are as follows. (1) By using variational inference, we map normal data into a latent space that follows a well-known probability distribution [10]. Variational inference is a method for training generative models, where the model maximizes the evidence lower bound (ELBO); (2) By ensuring that the latent space follows a well-known probability distribution, we can easily sample latent vectors and generate new data from them. Additionally, since we already know the statistical properties of the probability distribution, we can appropriately tamper with the latent vectors to generate anomalous data; (3) Inspired by dynamic software testing methods such as fuzzing, we adopt mutation to modify latent vectors [11,12]. These insights guide our approach to effectively generate time-series anomalies while addressing the challenges of temporal dependency, multivariate correlation, and data scarcity.

In this paper, we propose a method for generating pseudo anomalous data for industrial process data by synthesizing the aforementioned insights. The proposed method consists of two main parts. First, we map industrial time-series data to latent variables using a neural network. The learned latent variables encapsulate not only the characteristics of each time step of the data but also the correlations between features. Next, we tamper with the latent variables using mutation operations inspired by fuzz testing. We define mutation types to reproduce several known types of time-series anomaly patterns. In evaluations using several well-known ICS time-series datasets, the proposed method successfully generated time-series anomalies with various patterns. The contributions of this paper are as follows:

• We define patterns of time-series anomalies by analyzing several time-series datasets and existing studies.
• We propose a robust time-series anomaly generation algorithm using mutations and a variational recurrent autoencoder.
• We generate time-series anomalies from well-known ICS datasets using the proposed algorithm and comprehensively evaluate the quality of the synthetic anomalies.

The remainder of this paper is organized as follows. Section 2 presents existing studies on anomaly detection from the perspectives of one-class classification and binary classification; Section 3 analyzes the patterns of time-series anomalies; Section 4 proposes a method for generating time-series anomalies using AVRAE and mutation; Section 5 thoroughly evaluates the quality of the time-series anomalies generated by the proposed method using widely used anomaly detection ICS time-series datasets; Section 6 discusses several limitations of the proposed method; Finally, in Section 7, we provide conclusions and suggest directions for future research.

2. Related Work

Anomaly detection is a major research topic in various application domains, including cybersecurity. Most studies in this field adopt either one-class classification or binary classification detection strategies, depending on the availability of anomalous data during the training of detection models. In this section, we present existing research related to each approach (Section 2.1, Section 2.2 and Section 2.3). Furthermore, since this paper focuses on anomalies observed in time-series data, we discuss recent studies on generating time-series data and anomalous data in Section 2.4 and Section 2.5, respectively.

2.1. Statistical Anomaly Detection

The most traditional method to detect an anomaly is to utilize statistical techniques such as calculating the mean and standard deviation of a dataset. By identifying data points that fall outside of a certain threshold, these methods can flag potential outliers.

W. Yu et al. [13] introduces a generalized probabilistic monitoring model (GPMM) designed to handle both random and sequential data for process monitoring. It unifies various probabilistic linear models and establishes connections between different monitoring methods. Using the expectation-maximization (EM) algorithm, the model estimates parameters, derives monitoring statistics, and investigates the equivalence between these statistics and those from classical multivariate methods. The model’s effectiveness is demonstrated through numerical examples and application to the Tennessee Eastman process.

W. Yu et al. [14] presents an unsupervised fault detection and diagnosis method called Sparse Distribution Dissimilarity Analytics (SDDA), which combines distribution dissimilarity with a lasso penalty. This method addresses shortcomings in existing techniques by maximizing the dissimilarity between normal and abnormal data distributions, enabling accurate detection and diagnosis of process faults, including those with small magnitudes. The method is validated through both simulations and real industrial processes, showing superior performance compared to traditional methods.

W. Yu et al. [15] presents a novel fault detection method designed for complex industrial systems. The proposed MoniNet framework integrates both temporal and spatial information using a cascaded monitoring network, which enhances the detection accuracy of process anomalies. The method is validated using real industrial data, demonstrating its effectiveness in identifying faults more accurately compared to traditional methods.

2.2. Anomaly Detection in One-Class Classification

As mentioned earlier, anomalous behaviors occurring in enterprise networks or systems are typically very rare. Consequently, most research interprets anomaly detection as a one-class classification problem. L. Shen et al. [4] proposed a temporal hierarchical one-class model and an end-to-end learning method for time-series anomaly detection. This model is designed with a dilated recurrent neural network and incorporates multi-scale clustering to better capture temporal dynamics. The cluster centers are encouraged to be orthogonal.

H. Xu et al. [5] developed a calibrated one-class classifier that learns a more refined normality boundary. The calibration of this model involves penalizing uncertain predictions and discriminating normal samples from simulated abnormal behaviors.

S. Mauceri et al. [16] focused on the representation of time-series data rather than designing a novel classifier. This study represented given time-series data based on their dissimilarities to a set of so-called prototypes. The study evaluated the Cartesian product of 12 dissimilarities and 8 prototypes and used a one-class nearest neighbor classifier to detect anomaly samples.

L. Gjorgiev and S. Gievska [17] explore the use of variational autoencoders (VAEs) combined with Mahalanobis distance for anomaly detection in time-series data. The study evaluates various deep learning architectures on the BATADAL challenge dataset, which focuses on detecting cyber-attacks in water distribution systems. The results indicate that simpler VAE models using Mahalanobis distance can effectively detect anomalies, demonstrating significant promise in time-to-detection performance.

2.3. Anomaly Detection in Binary Classification

While most studies on time-series anomaly detection focus on the one-class classification strategy using only normal data, there have been recent attempts to identify anomalous data through binary classification. Z. Ghrib et al. [6] proposed a hybrid approach to recognize fraudulent credit card transactions. This study generated latent representations of the given data using a long short-term memory (LSTM) [18] based autoencoder pretrained on normal data, and then classified these representations using a support vector machine (SVM) [19].

P. Primus et al. [7] proposed a method for detecting anomalous sounds. Instead of presenting a novel detection model, this study focused on collecting anomalous samples from a given dataset by careful selection of unrelated data to serve as proxy outliers. Then, a detection model based on ResNet [20] was trained as a binary classifier using normal samples and proxy outliers.

I. Ullah et al. [21] proposed an RNN-based model for detecting intrusions in Internet of Things (IoT) networks. The proposed model, designed with LSTM, bidirectional LSTM, or gated recurrent unit (GRU) [22], was evaluated on various IoT datasets. This study trained the detection model as a binary classifier and employed techniques such as weighted loss and borderline SMOTE [23] to overcome the limitations of imbalanced datasets.

K. Gundersen et al. [24] proposed a binary time-series classification model for detecting gas emissions from the ocean. This study adopted a Bayesian convolutional neural network to detect gas leaks and introduced Monte Carlo dropout [25] to maximize generalization.

F. Liu et al. [26] proposed a model for detecting anomalous data in quasi-periodic time-series. This study first split the quasi-time-series into successive quasi-periods through two-level clustering and then detected anomalies using a hybrid attentional LSTM-CNN model.

2.4. Time-Series Data Generation

Many real-world datasets are time-series, which exhibit higher dynamicity and complexity compared to i.i.d. data. Consequently, time-series generation models should address issues arising from temporal dependencies. G. Forestier et al. [27] proposed a time-series generation method based on dynamic time warping (DTW) barycenter averaging (DBA). This method generates new time-series by calculating a weighted average of the time-series within a given dataset. By assigning different weights to each time-series, a variety of rich patterns can be generated. Additionally, this study presented several weight selection methods to ensure diversity in time-series synthesis.

Recently, deep neural network (DNN)-based models have been employed to more effectively capture the temporal dependency and dynamicity in data synthesis. Powerful generative models based on variational inference [10] and generative adversarial networks (GAN) [28] are among these approaches. C. Zhang et al. [29] proposed a GAN-based model for synthesizing power consumption data in smart grids. This study represented power consumption data using two attributes (level and pattern) and conditionally modeled the data probabilistically over users, days, and months, enabling the model to learn temporal attributes.

L. Zhou et al. [30] presents a novel model called LS4, a deep latent state space model for time-series generation. It addresses the limitations of existing ordinary differential equations (ODE)-based models, particularly their struggles with sharp transitions in time-series data and high computational costs. LS4 leverages a convolutional representation to enhance speed and efficiency, significantly outperforming previous models in accuracy and computational efficiency, especially on datasets with irregular sampling and long sequences.

2.5. Pseudo Anomaly Generation

Synthetic data generation plays a crucial role in various domains for augmenting imbalanced datasets and ensuring privacy protection. It is well known that building robust anomaly detection models requires balanced datasets containing both normal and anomalous data [31]. However, to construct high-quality datasets, it is essential to secure relatively rare anomalous data. M. Salem et al. [32] proposed a strategy to transform normal data into anomalous data using Cycle-GAN [33]. Cycle-GAN employs a training method that enables mutual transformation between images from different domains. This study utilized this characteristic to convert data into images and transform template data (normal data) into anomalies. The synthetic anomalies are then appended to the dataset for training detection models.

M. Pourreza et al. [34] introduced G2D, a GAN-based general framework for generating anomalies. This study categorizes synthetic data into three stages based on the learning progress of the generator: random samples, outliers surrounding the normal samples, and samples following the data distribution. Random samples and outliers are considered anomalies and used for training anomaly detection models. As a result, the detection models trained with G2D demonstrate stable performance as the proportion of outliers in the dataset increases.

H. Shen et al. [35] presented a novel method for unsupervised anomaly detection in industrial images. This method involves generating high-quality pseudo-anomaly images and enhancing normal image features. Experiments on various real datasets show performance improvements, specifically by enhancing normal image features to boost the model’s prediction accuracy. The method also reduces the uncertainty of single models by integrating various anomaly scores through ensemble detection.

Y. Lin et al. [36] introduced the FastLogAD system, designed for rapid anomaly detection in log data. It uses a mask-guided anomaly generation (MGAG) technique to generate pseudo-abnormal logs and a discriminative abnormality separation (DAS) model for efficient anomaly identification. FastLogAD significantly outperforms existing methods by achieving anomaly detection speeds at least ten times faster and competitive performance metrics, representing a significant advancement in log anomaly detection with an emphasis on speed and efficiency in handling security-related data.

T. Hu et al. [37] introduced AnomalyDiffusion, a new anomaly generation model that enables accurate anomaly detection with limited data. AnomalyDiffusion significantly improves generation accuracy and diversity by integrating anomaly location and appearance information. Experimental results demonstrate that this model outperforms existing methods in both generation accuracy and diversity, showing high performance in downstream anomaly detection tasks.

While numerous studies have proposed methods for generating anomalous data, only a limited number of these focus on time-series data. Most synthetic anomaly generation techniques are confined to i.i.d data, such as images. In contrast, data collected from industrial processes often exhibit time-series characteristics, making i.i.d-based anomaly generation techniques unsuitable for generating time-series anomalous data.

3. Patterns of Time-Series Anomaly

Understanding the characteristics and patterns of anomalous data is as crucial as the techniques for generating time-series or anomalies. Several studies have explored the patterns of time-series anomalies [38,39,40,41]. We synthesize the findings of these studies to comprehensively analyze anomaly patterns. Based on this analysis, we design mutation operations in Section 4 to reproduce these patterns. Specifically, we focus on continuous data collected from industrial processes [42,43], rather than discrete data such as log data.

P. Boniol et al. [41] analyzed various publicly available time-series datasets and categorized the anomalous data within these datasets into several patterns. Broadly, this study distinguishes between anomaly patterns occurring at single data points and interval anomaly patterns (collective anomalies) that span multiple data points.

Anomaly patterns for single data points are further divided into point anomalies, which exceed expected value ranges, and contextual anomalies, which do not. Additionally, the concept of multiple anomalies, where several anomalous data points are present, is mentioned. Y. Bao et al. [40] and Z. Tang et al. [39] proposed anomaly detection models for health monitoring data and, as part of their research, analyzed the anomaly patterns within the data. These studies fundamentally consider data that deviate statistically from the norm as anomalies.

In addition to conducting a literature review, we analyzed well-known ICS datasets to examine the anomaly patterns contained within these datasets. Specifically, we analyzed the SWaT [42] and the HIL-based augmented ICS security dataset (HAI) [43]. The secure water treatment (SWaT) [42] dataset is derived from a water treatment testbed designed for ICS and security research. This testbed consists of six processes: raw water intake, chemical dosing, ultrafiltration (UF), reverse osmosis (RO), RO filtration, and UF backwash. In this study, three types of attackers were modeled, performing various attacks such as network packet sniffing and physical access.

The HAI [43] dataset also originates from a testbed primarily for water treatment, consisting of four major processes: the boiler process, turbine process, water treatment process, and hardware-in-the-loop simulator. The dataset was collected over more than ten days, capturing three types of attacks: process variable response prevention, setpoint attack, and control output attack.

Figure 1 illustrates the anomaly patterns found in the HAI and SWaT datasets. The blue and red lines represent normal data and anomalies, respectively. As seen in the figure, the anomaly patterns in both ICS datasets mostly correspond to point anomalies or contextual anomalies as defined by P. Boniol et al. [41]. The SWaT dataset includes some collective anomalies.

Furthermore, according to Figure 1, the collective anomalies in the SWaT dataset can be categorized as minor types, as described by Y. Bao et al. [40] and Z. Tang et al. [39]. Based on our literature review and analysis of the anomaly patterns in ICS datasets, we focus on generating point anomalies, contextual anomalies, and collective anomalies from given normal ICS time-series data.

4. Mutation-Based Multivariate Time-Series Anomaly Generation in ICS

This section describes the mutation-based ICS time-series anomaly generation model. Figure 2 illustrates the ICS time-series anomaly generation process. To effectively generate anomalies for ICS time-series data, we adopt an attention-based variational recurrent autoencoder (AVRAE) [44]. AVRAE is fundamentally designed with an RNN architecture and incorporates variational inference and an attention mechanism to better learn the characteristics of the time-series data. In the figure, the blue and red rectangles represent the RNN cells of the encoder and decoder, respectively, while the green circles denote the hidden states produced by the encoder or decoder.

Our anomalous data generation process is divided into two main phases: the training phase (left side of Figure 2) and the generation phase (right side of Figure 2). The training phase aims to train AVRAE as a robust time-series generation model. Each timestep’s latent vector produced by AVRAE is forced to follow a well-known probability distribution. Additionally, the attention layers of AVRAE effectively learn the relationships between the hidden states (latent vectors) produced by the encoder and those produced by the decoder. Once AVRAE training is completed, the process moves to the generation phase. In the generation phase, the hidden states generated by the trained encoder of AVRAE are mutated and passed through the attention layers. AVRAE then generates time-series anomalies using the mutated hidden states from the encoder and the hidden states from the decoder, where we define several mutation operators specifically designed to generate point anomalies, contextual anomalies, or collective anomalies.

The remainder of this section is organized as follows. In Section 4.1, we describe the details of AVRAE, including the objective function and attention mechanisms. Section 4.2 presents the mutation operators for generating ICS time-series anomalies. Finally, Section 4.3 proposes an algorithm for ICS time-series anomaly generation that combines AVRAE with the mutation operators.

4.1. Attention-Based Variational Recurrent Autoencoder

AVRAE is central to our ICS time-series anomaly generation. There are two main reasons for adopting AVRAE for anomalous data generation. First, AVRAE extends variational inference to time-series data, allowing control over the distribution of hidden states produced at each timestep by the RNN. Second, AVRAE’s attention mechanism enables the generation of highly plausible time-series by leveraging the relationships between the encoder’s and decoder’s hidden states.

Since our approach involves applying mutation operations to the hidden states to generate anomalous data, AVRAE, which maps actual data to controllable hidden states, is a highly suitable model. This section formally describes AVRAE’s objective function, the time-series evidence lower bound, and the inference process.

4.1.1. Evidence Lower Bound

Variational inference is an approximation method that uses relatively simple distributions to handle complex probability distributions. It is used to optimize a latent variable model for given data, replacing complex distributions with simpler, more computationally feasible ones, thereby reducing computational cost and increasing efficiency. ELBO is a typical objective used in variational inference.

$$\begin{array}{cc}\hfill logp\left(\mathbf{x}\right)& =KL\left(q\left(\mathbf{z}\right)\right|\left|p\left(\mathbf{z}\right|\mathbf{x})\right)+{\mathbb{E}}_{q\left(\mathbf{z}\right)}[logp(\mathbf{z},\mathbf{x})]-{\mathbb{E}}_{q\left(\mathbf{z}\right)}[logq\left(\mathbf{z}\right)]\hfill \\ \hfill & \ge {\mathbb{E}}_{q\left(\mathbf{z}\right)}[logp(\mathbf{z},\mathbf{x})]-{\mathbb{E}}_{q\left(\mathbf{z}\right)}[logq\left(\mathbf{z}\right)]\hfill \\ \hfill & =ELBO\left(q\right)\hfill \end{array}$$

(1)

Equation (1) is the derivation of ELBO. Originally, we aim to train the model to maximize the loglikelihood $logp\left(\mathbf{x}\right)$, where $\mathbf{x}$ is the observable data and $\mathbf{z}$ is the latent variable. $p(.)$ represents the true distribution, and $q(.)$ represents the variational distribution. The $logp\left(\mathbf{x}\right)$ can be decomposed into the Kullback–Leibler divergence (KLD) $KL\left(q\right(\mathbf{z}\left)\right|\left|p\right(\mathbf{z}\left|\mathbf{x}\right))$ and the $ELBO\left(q\right)$. Since the KLD $KL\left(q\right(\mathbf{z}\left)\right|\left|p\right(\mathbf{z}\left|\mathbf{x}\right))$ is always non-negative, $logp\left(\mathbf{x}\right)$ is always greater than or equal to $ELBO\left(q\right)$.

Therefore, variational inference seeks to maximize $ELBO\left(q\right)$ instead of directly maximizing $logp\left(\mathbf{x}\right)$. This ELBO fundamentally assumes that the data $\mathbf{x}$ is i.i.d. Hence, to handle probability distributions over time-series data with variational inference, ELBO must be extended to time-series.

$$\begin{array}{cc}\hfill ELB{O}_{ts}\left(q\right)& ={\mathbb{E}}_{q(\mathbf{z},{\mathbf{h}}_{1:T}|{\mathbf{x}}_{1:T})}[logp(\mathbf{z},{\mathbf{h}}_{1:T},{\mathbf{x}}_{1:T})]-{\mathbb{E}}_{q(\mathbf{z},{\mathbf{h}}_{1:T}|{\mathbf{x}}_{1:T})}[logq(\mathbf{z},{\mathbf{h}}_{1:T}|{\mathbf{x}}_{1:T})]\hfill \\ \hfill & ={\mathbb{E}}_{q(\mathbf{z},{\mathbf{h}}_{1:T}|{\mathbf{x}}_{1:T})}[logp\left({\mathbf{x}}_{1:T}\right|\mathbf{z})]-KL(q\left(\mathbf{z}\right|{\mathbf{h}}_{1:T})\left|\right|p\left(\mathbf{z}\right|{\mathbf{h}}_{1:T}))\hfill \\ \hfill & -KL\left(q\left({\mathbf{h}}_{1:T}\right|{\mathbf{x}}_{1:T})\right|\left|p\left({\mathbf{h}}_{1:T}\right)\right)\hfill \end{array}$$

(2)

Equation (2) formulates $ELB{O}_{ts}\left(q\right)$, which is an extended version of ELBO for time-series data. $ELB{O}_{ts}\left(q\right)$ not only extends the data to a time-series but also considers an additional sequential latent variable ${\mathbf{h}}_{1:T}$ to account for the sequential nature of the data.

Consequently, maximizing $ELB{O}_{ts}\left(q\right)$ is equivalent to maximizing the log-likelihood of the time-series data ${\mathbf{x}}_{1:T}$ given the latent variable $\mathbf{z}$, while minimizing the Kullback–Leibler divergence between the true distribution and the variational distribution for both the latent variable $\mathbf{z}$ and the sequential latent variable ${\mathbf{h}}_{1:T}$.

4.1.2. Inference

AVRAE follows an encoder–decoder architecture composed of RNNs. At each timestep, the RNN processes the input data $x_t$ and the hidden state ${\mathbf{h}}_{t-1}$ from the previous timestep to produce a new hidden state ${\mathbf{h}}_t$. This hidden state ${\mathbf{h}}_t$ is propagated to the next timestep, allowing the RNN to reflect the sequential nature of the data. However, basic RNNs perform all inference operations deterministically. Therefore, AVRAE introduces stochasticity into the inference process to enable variational inference.

$$\begin{array}{cc}\hfill {\mathbf{h}}_t& \sim q_{\theta }\left({\mathbf{h}}_t\right|{\mathbf{h}}_{t-1},{\mathbf{x}}_t)\hfill \end{array}$$

(3)

Equation (3) represents the stochastic process of the RNN-based encoder at timestep t, where $q_{\theta }$ is the variational distribution parameterized by $\theta$. The hidden state ${\mathbf{h}}_t$ is sampled from the variational distribution $q_{\theta }$, which is conditioned on the hidden state ${\mathbf{h}}_{t-1}$ from the previous timestep and the input data ${\mathbf{x}}_t$ at the current timestep.

$$\begin{array}{cc}\hfill {\mathbf{h}}_t& \sim \mathcal{N}({\mathit{\mu }}_{{\mathbf{h}}_t},\mathrm{diag}\left({\mathit{\sigma }}_{{\mathbf{h}}_t}^2\right)),\mathrm{where}[{\mathit{\mu }}_{{\mathbf{h}}_t},{\mathit{\sigma }}_{{\mathbf{h}}_t}]=\varphi ({\mathbf{h}}_{t-1},{\mathbf{x}}_t)\hfill \end{array}$$

(4)

Equation (4) describes the sampling process assuming $q_{\theta }$ is a Gaussian distribution. The function $\varphi (.)$ uses ${\mathbf{h}}_{t-1}$ and ${\mathbf{x}}_t$ to produce the mean ${\mathit{\mu }}_{{\mathbf{h}}_t}$ and variance ${\mathit{\sigma }}_{{\mathbf{h}}_t}$ of the Gaussian distribution. Then, ${\mathbf{h}}_t$ is sampled from the Gaussian distribution parameterized by ${\mathit{\mu }}_{{\mathbf{h}}_t}$ and ${\mathit{\sigma }}_{{\mathbf{h}}_t}$. However, as is well known, sampling is a non-differentiable operation, so the reparameterization trick is typically used in variational inference [10,45,46]. We use a location-scale transformation to ensure that the reparameterized hidden state still follows a Gaussian distribution: ${\mathbf{h}}_t={\mathit{\mu }}_{{\mathbf{h}}_t}+{\mathit{\sigma }}_{{\mathbf{h}}_t}\odot ϵ$ ($ϵ\sim \mathcal{N}(0,I)$).

The decoder fundamentally mirrors the encoder and thus has similar inference, with three key exceptions. First, the decoder requires the latent vector $\mathbf{z}$ (referred to as context) produced by the encoder. In AVRAE, the hidden state ${\mathbf{h}}_T$ from the encoder’s final timestep is used as $\mathbf{z}$. Second, unlike the encoder, the decoder uses the output from the previous timestep as the input data for the current timestep. The initial piece of input data for the decoder is the start symbol <start>. Third, AVRAE processes the decoder’s outputs through attention layers to reconstruct the encoder’s input data ${\mathbf{x}}_{1:T}$.

These attention layers include cross-attention and self-attention. Cross-attention combines the encoder’s hidden states ${\mathbf{h}}_{1:T}^e$ and the decoder’s hidden states ${\mathbf{h}}_{1:T}^d$ to calculate attention weights, which are then used to produce the output ${\mathbf{o}}_{1:T}^c$. Additionally, cross-attention employs a look-ahead mask to prevent future information from being referenced during sequence generation. Next, self-attention takes ${\mathbf{o}}_{1:T}^c$ as input, calculates attention weights, and produces the output ${\mathbf{o}}_{1:T}^s$. Finally, AVRAE uses a dense layer to reconstruct ${\mathbf{x}}_{1:T}$ from ${\mathbf{o}}_{1:T}^s$, resulting in ${\widehat{\mathbf{x}}}_{1:T}$.

4.1.3. Training

The training of AVRAE is relatively straightforward. AVRAE is trained to maximize Equation (2). More specifically, ${\mathbb{E}}_{q(\mathbf{z},{\mathbf{h}}_{1:T}|{\mathbf{x}}_{1:T})}[logp\left({\mathbf{x}}_{1:T}\right|\mathbf{z})]$ is maximized by minimizing the error between the encoder’s input ${\mathbf{x}}_{1:T}$ and the decoder’s output ${\widehat{\mathbf{x}}}_{1:T}$. $KL\left(q\left(\mathbf{z}\right|{\mathbf{h}}_{1:T})\right|\left|p\left(\mathbf{z}\right|{\mathbf{h}}_{1:T})\right)$ and $KL\left(q\left({\mathbf{h}}_{1:T}\right|{\mathbf{x}}_{1:T})\right|\left|p\left({\mathbf{h}}_{1:T}\right)\right)$ are minimized as the variational distribution becomes closer to the true distribution.

AVRAE adopts the standard normal distribution as the true distribution, so the KLD decreases as the hidden states of the encoder ${\mathbf{h}}_{1:T}^e$ and the decoder ${\mathbf{h}}_{1:T}^d$ follow the standard normal distribution more closely.

4.2. Mutation Operators

We define three mutation operators to generate ICS time-series anomalies of the type analyzed in Section 3, utilizing the aforementioned characteristics. Algorithms 1–3 are the mutation operators for generating point anomalies, contextual anomalies, and collective anomalies, respectively. All three algorithms are applied to the hidden state sequence ${\mathbf{h}}_{1:T}^e$ produced by the encoder of AVRAE and return the mutated sequence ${\widehat{\mathbf{h}}}_{1:T}^e$.

Algorithm 1: Mutation operator ${\mathcal{M}}_p$ for point anomaly generation.

input: the encoder’s hidden state sequence ${\mathbf{h}}_{1:T}^e$

output: the mutated hidden state sequence ${\widehat{\mathbf{h}}}_{1:T}^e$

$t\leftarrow \mathcal{U}(1,T)$;

$\mathbf{v}\leftarrow \mathcal{N}(0,I)$;

${\mathbf{h}}_{1:T}^e\left(t\right)\leftarrow \mathbf{v}$;

${\widehat{\mathbf{h}}}_{1:T}^e\leftarrow {\mathbf{h}}_{1:T}^e$;

Algorithm 2: Mutation operator ${\mathcal{M}}_{ctx}$ for contextual anomaly generation.

input: the encoder’s hidden state sequence ${\mathbf{h}}_{1:T}^e$

output: the mutated hidden state sequence ${\widehat{\mathbf{h}}}_{1:T}^e$

$t_s\leftarrow \mathcal{U}(1,T)$;

$t_d\leftarrow \mathcal{U}(t_s,T)$;

$\mathbf{v}\leftarrow {\mathbf{h}}_{1:T}^e\left(t_s\right)$;

${\mathbf{h}}_{1:T}^e\left(t_d\right)\leftarrow \mathbf{v}$;

${\widehat{\mathbf{h}}}_{1:T}^e\leftarrow {\mathbf{h}}_{1:T}^e$;

Algorithm 3: Mutation operator ${\mathcal{M}}_{col}$ for collective anomaly generation.

input: the encoder’s hidden state sequence ${\mathbf{h}}_{1:T}^e$

output: the mutated hidden state sequence ${\widehat{\mathbf{h}}}_{1:T}^e$

$t_s\leftarrow \mathcal{U}(1,T)$;

$t_d\leftarrow \mathcal{U}(t_s,T)$;

${\mathbf{v}}_{t_s:t_d}\leftarrow$ a vector of arbitrary value with length ($t_d-t_s$);

${\mathbf{h}}_{1:T}^e\left(t_s:t_d\right)\leftarrow {\mathbf{v}}_{t_s:t_d}$;

${\widehat{\mathbf{h}}}_{1:T}^e\leftarrow {\mathbf{h}}_{1:T}^e$;

Algorithm 1 introduces a point anomaly into the given hidden state sequence. First, an index t is randomly selected to specify the element in the hidden state sequence to be altered. Then, a random value $\mathbf{v}$ is sampled from the standard normal distribution to replace the t-th element of the hidden state sequence.

Algorithm 2 is the mutation operator ${\mathcal{M}}_{ctx}$ for generating contextual anomalies. This algorithm randomly selects two indices $t_s$ and $t_d$ that are smaller than the total length of the sequence T. Then, it replaces the value at the $t_d$-th position in the hidden state sequence with the value $\mathbf{v}$ from the $t_s$-th position. The rationale behind this design is that contextual anomalies inherently have values within the normal range but are considered abnormal within a given context. Therefore, Algorithm 2 generates anomalies by altering the context of normal values within the hidden state sequence.

Lastly, Algorithm 3 is the mutation operator ${\mathcal{M}}_{col}$ for generating collective anomalies. This algorithm first randomly selects two indices $t_s$ and $t_d$, both less than the total length of the sequence T, to specify the segment to mutate. Then, it replaces the values in this segment of the hidden state sequence with a sequence ${\mathbf{v}}_{t_s:t_d}$ composed of arbitrary values, generating the mutated sequence ${\widehat{\mathbf{h}}}_{1:T}^e$.

Depending on the method used to generate ${\mathbf{v}}_{t_s:t_d}$, various types of collective anomalies can be produced. While there are multiple approaches, this study recommends replacing with a sequence of constant vectors, a sequence of random vectors, or a sequence from another segment. The first two methods are the simplest ways to create ${\mathbf{v}}_{t_s:t_d}$, while the latter extends contextual anomalies into collective anomalies. It is noteworthy that the three algorithms presented so far only mutate the hidden state sequence and do not directly generate actual time-series anomalies.

4.3. Anomaly Generation

As previously mentioned, we generate anomalies by altering the latent space rather than the actual data space. More specifically, we use AVRAE to map ICS time-series data into the latent space. In particular, the encoder of AVRAE produces hidden states at each timestep while processing the time-series. Additionally, since these hidden states follow a well-known probability distribution (in this study, the standard normal distribution), it is easy for us to predict or control the values of the hidden states.

We apply mutations to the hidden states produced by the encoder of AVRAE. This approach has two major advantages. First, the hidden states (the latent vectors) effectively encapsulate the correlations among the features of the observations. Therefore, changing only a part of the latent vector affects all the features of the observations. Second, the hidden states from the encoder are used by the decoder’s attention layer (cross-attention) for time-series generation. The attention layer learns the relevance between the elements of the hidden state sequences produced by the encoder and the decoder. Thus, if a segment of the hidden state sequence from the encoder is altered, the subsequent data generated by the decoder are also influenced.

Algorithm 4 outlines the process for generating anomalous ICS time-series proposed in this study. This algorithm is very straightforward. First, the encoder of the trained AVRAE processes ${\mathbf{x}}_{1:T}$ to produce the hidden state sequence ${\mathbf{h}}_{1:T}^e$. As mentioned in Section 4.1, we assume the standard normal distribution as the true distribution, so each ${\mathbf{h}}_t^e$ that makes up ${\mathbf{h}}_{1:T}^e$ follows the standard normal distribution. Then, one of the three mutation operators $\{{\mathcal{M}}_p,{\mathcal{M}}_{ctx},{\mathcal{M}}_{col}\}$ presented in Section 4.2 is randomly selected as $\mathcal{M}$. Although $\mathcal{M}$ is chosen randomly in this study, if there is a need to control the type of anomaly to be generated, specifying a particular mutator is allowed. Using $\mathcal{M}$, we generate the mutated sequence ${\widehat{\mathbf{h}}}_{1:T}^e$ from ${\mathbf{h}}_{1:T}^e$. Finally, the decoder of AVRAE generates the ICS time-series ${\widehat{\mathbf{x}}}_{1:T}$ from ${\widehat{\mathbf{h}}}_{1:T}^e$. Since ${\widehat{\mathbf{x}}}_{1:T}$ is generated from the mutated sequence ${\widehat{\mathbf{h}}}_{1:T}^e$, it contains anomalies. This generated anomalous ICS time-series not only considers the correlations among the features of the actual data but also reflects the temporal dependencies through the attention layers of AVRAE.

Algorithm 4: Mutation-based ICS time-series anomaly generation.

input: the trained AVRAE, the ICS time-series ${\mathbf{x}}_{1:T}$

output: the anomalous ICS time-series ${\widehat{\mathbf{x}}}_{1:T}$

${\mathbf{h}}_{1:T}^e\leftarrow$ produce the hidden state sequence from ${\mathbf{x}}_{1:T}$ with the AVRAE’s encoder;

$\mathcal{M}\leftarrow$ randomly select mutation operator in $\{{\mathcal{M}}_p,{\mathcal{M}}_{ctx},{\mathcal{M}}_{col}\}$;

${\widehat{\mathbf{h}}}_{1:T}^e\leftarrow \mathcal{M}\left({\mathbf{h}}_{1:T}^e\right)$;

${\widehat{\mathbf{x}}}_{1:T}\leftarrow$ product the anomalous ICS time-series from ${\widehat{\mathbf{h}}}_{1:T}^e$ with the AVRAE’s decoder;

5. Evaluation

In this section, a series of experiments is conducted to evaluate the mutation-based ICS time-series anomaly generation proposed in this paper for the ICS dataset. To our best knowledge, there is no prior research on the generation of anomalous data for time-series. Therefore, instead of comparing performance with other studies, the plausibility of the generated anomalies is assessed, and binary classification is performed using them. From this, two research questions (RQs) are defined as follows:

• RQ1: Are the synthetic ICS time-series anomalies visually and in the embedding space similar to real data?
• RQ2: Does a binary classifier trained on the synthetic ICS time-series anomalies perform better than a one-class classifier?

5.1. Dataset Description

To evaluate the mutation-based ICS time-series anomaly generation proposed in this study, the HAI [43] and SWaT [42] datasets were used. HAI is a testbed (and dataset) collected using a hardware-in-the-loop (HIL) simulator composed of turbines, boilers, and a water treatment system. In this testbed, normal and attack scenarios were repeatedly executed in an unmanned supervised control and data acquisition (SCADA) operating environment, and data were collected accordingly. This testbed has been continuously enhanced since 2017, and the dataset has been updated accordingly.

For our experiments, we used HAI 22.04. This version of the dataset provides six training datasets (each file ranging from 45 MB to 136 MB) and four test datasets (each file ranging from 33 MB to 69 MB). Each dataset consists of 86 features. The training datasets comprise only normal data, while the test datasets include anomaly data along with label information.

The SWaT dataset was collected from a six-stage water treatment process, where each stage is autonomously controlled by PLCs. This testbed is designed to closely mimic a real water treatment system, ensuring that the collected data can be applied to actual systems. In SWaT, communication among sensors, actuators, and PLCs is realized using a combination of wired and wireless channels, allowing for extensive experiments in realistic environments. The SWaT dataset has also been enhanced over time, with corresponding updates to the dataset. For this experiment, datasets collected in 2015 were used. Although the training and test datasets are not explicitly separated, the dataset provides two normal datasets (each file sized 127 MB) and one attack dataset (file sized 113 MB). Excluding the labels, this dataset consists of 51 features.

Furthermore, before presenting the analysis of the experimental results, it is worth noting that various attack methods can be included in actual attack scenarios against ICS. Consequently, the types of anomalies can also be diversified. However, the HAI and SWaT datasets used in this evaluation only categorize the data as either normal or attack. This study does not aim to present a model with high anomaly detection performance. Nonetheless, we conducted anomaly detection experiments to verify that synthetic anomalies help train detection models. Additionally, the primary purpose of these experiments is to distinguish attack data from normal data, rather than categorizing the types of attacks in detail.

5.2. Experimental Settings

All experiments were conducted on the same machine with the following specifications: Intel(R) Core(TM) i9-11900 2.50 GHz, 32 GB RAM, 64-bit Ubuntu 20.04 LTS, and NVIDIA GTX 3080 Titan. The implementation of AVRAE was implemented using Python code, with the deep learning library PyTorch 2.3.0.

Table 1. The architecture of AVRAE.

Layer	Encoder			Decoder
	Type	Input Size	Output Size	Type	Input Size	Output Size
Layer 1	LSTM	$100\times d$	$100\times p$	LSTM	$100\times p$	100 × p
Layer 2	LSTM	$100\times p$	$100\times p$	LSTM	$100\times p$	$100\times p$
Layer 3				Cross-Attention	$100\times p$, $100\times p$	$100\times p$
Layer 4				Self-Attention	$100\times p$, $100\times p$	$100\times p$
Layer 5				Dense	$100\times p$	$100\times d$

shows the architecture of AVRAE used in this study. AVRAE has a nearly symmetrical encoder and decoder. The encoder consists of two LSTM layers, with the first layer taking data of size d at each timestep (sequence length of 100) and producing an output of size $100\times p$. For HAI, p is set to 1024, and for SWaT, it is set to 256. The second layer takes the output of the first layer as input and produces an output of size $100\times p$.

The decoder uses the hidden state of the last timestep of the encoder as its initial hidden state, taking a vector of size p at each timestep as input and producing an output of size $100\times p$. Similarly, the second layer of the decoder takes the output of the first layer as input and produces an output of size $100\times 1024$. The reason the input size of the first layer of the decoder is p is that it receives the timestep-wise output of the second layer as feedback.

Then, the cross-attention layer takes the outputs of both the encoder and the decoder as input and produces a sequence of size $100\times p$. The self-attention layer takes the output of the cross-attention layer as input and outputs a vector sequence of size $100\times p$. Finally, the output of the self-attention layer is restored to a vector sequence of size $100\times d$, the same shape as the input to the encoder, by the dense layer.

AVRAE is trained using the Adam optimizer [47]. The Adam optimizer is an adaptive learning rate optimization algorithm designed for training deep learning models, combining the advantages of two other extensions of stochastic gradient descent, namely, AdaGrad [48] and RMSProp [49], to compute individual adaptive learning rates for different parameters. The learning rate was set to $5\times {10}^{-4}$, the mini-batch size to 64, and AVRAE was trained for 1024 epochs.

5.3. Assessment on Quality of Synthetic Anomaly

This section assesses the quality of the synthetic ICS time-series anomaly both visually and statistically. Visually inspecting the quality of generated time-series anomaly data is crucial because it allows evaluation of whether an anomaly detection model accurately captures real anomaly situations and whether the data patterns are realistic. This helps in intuitively understanding the detection model’s performance and identifying potential areas for improvement.

Figure 3 shows the ICS time-series anomalies generated by the proposed method. In the figure, the blue line represents the real data used as the source for AVRAE to generate anomalous data, and the red line represents the generated anomaly. Each subplot depicts the changes in the values of a variable over time in the dataset (the y-axis represents the variable’s value, and the x-axis represents the time sequence). The first row of Figure 3 represents point anomaly generation, the second row represents context anomaly generation, the third row represents collective anomaly generation with a constant value, the fourth row represents collective anomaly generation with a random value, and the fifth row represents collective anomaly generation with swap. The sections where the mutation operation is applied are marked with green circles.

The results of this experiment were extremely interesting. In the case of point anomaly generation, the anomalous values significantly differed from the surrounding context. Additionally, the direction of the outliers varied depending on the variables, which indicates that AVRAE learned the multivariate relationships of the HAI dataset.

The context anomaly generation creates anomalies by swapping the hidden states at two arbitrary timesteps in the embedding space. As a result, the range of anomaly values occurring early did not exceed the surrounding context. In fact, the anomaly values matched the values at the swapped location (i.e., the normal values of the later time). Conversely, the values at the later time remained unchanged, and the exact cause of this phenomenon has not been determined yet, although it is suspected to be due to the influence of the attention layer.

The collective anomaly generations with constant value and random value both exhibited similar effects. Anomalous data ignoring the pattern of the data were placed in a randomly selected interval. Lastly, the collective anomaly generation with swap selects two intervals of the same length at random and swaps their hidden states in the embedding space. Except for the anomaly occurring within the interval, the effect is similar to the context anomaly generation. In this method as well, the anomaly pattern occurring early matched the pattern at the swapped location, while the pattern at the later time remained unchanged. This phenomenon is also suspected to be due to the influence of the attention layer.

Figure 4 shows the results of ICS time-series anomaly generation using the SWaT dataset. Overall, results similar to the experiments using HAI were observed. However, for SWaT, the quality of the generated time-series was lower compared to HAI. This issue was analyzed as a limitation of AVRAE rather than a problem with the anomaly generation method. The SWaT dataset exhibited much more subtle variations in the values of each variable compared to HAI. This issue appears to have caused a decrease in the data reconstruction performance of the decoder in AVRAE.

Figure 5 visualizes the original source data and the synthetic time-series anomalies given to AVRAE in a lower dimension using principal component analysis (PCA) [50] and t-distributed stochastic neighbor embedding (SNE) [51]. Although PCA is not ideally suited for visualizing high-dimensional data, it can be used to examine the data’s variance. On the other hand, t-SNE excels at projecting high-dimensional data into a low-dimensional space to reveal local similarities within the data. In the figure, the two subplots on the left represent the HAI dataset, while the two subplots on the right depict the SWaT dataset. In each subplot, blue dots represent the original source data, and red dots represent anomalies. Each dot corresponds to a $100\times d$ vector, meaning the input to AVRAE’s encoder. Additionally, the anomalies used in this figure were generated by the collective anomaly generation method, which produced the greatest variations compared to the original source data.

In the HAI dataset, it was confirmed that the original data and the synthetic anomalies had similar distributions in both the PCA and t-SNE plots. This indicates that the synthetic anomalies share structural similarities and variance characteristics with the original data. However, it is evident that the distributions of the two datasets did not completely overlap. This suggests that new variance was introduced during the generation of synthetic anomalies through mutation.

This observation is even more pronounced in the SWaT dataset. The PCA plot shows that most of the data in the SWaT dataset has nearly similar variance characteristics. However, some synthetic anomalies appeared somewhat distinguishable from the original data. Additionally, in the t-SNE plot, the synthetic anomalies exhibited a distribution similar to the original data but were relatively more clustered. These two observations could imply that the anomaly generation method proposed in this study might be less effective for the SWaT dataset compared to the HAI dataset. We interpret this as the differences in the values of the features in the SWaT dataset being very subtle compared to the HAI dataset, which resulted in the mutation-induced changes in hidden states ultimately producing anomalies that differ from the original data.

Figure 6 shows the kernel density estimation (KDE) between the original source data and the synthetic anomaly used as input for AVRAE. In the figure, the first row displays the results for the HAI data, while the second row presents the results for the SWaT dataset. Additionally, the blue area represents the distribution of the original data, and the red area indicates the distribution of the synthetic anomaly. The synthetic anomaly used in Figure 6 was generated through collective anomaly generation, similar to Figure 5. As seen in the figure, the distributions of the original data and the anomaly are almost identical.

Table 2. Jensen–Shannon divergence between the original source data and the synthetic anomaly.

Variable (HAI)	P1_B4022	P1_PP04	P2_SIT01	P1_PIT02	P1_FCV03D	P1_FCV01Z
JSD	0.13	0.075	0.065	0.037	0.022	0.02
Variable (SWaT)	FIT503	PIT503	PIT501	FIT501	FIT502	LIT401
JSD	0.138	0.08	0.077	0.034	0.015	0.004

shows the Jensen–Shannon divergence (JSD) for each variable measured using KDE. The upper part of the table presents the results for the HAI dataset, and the lower part shows the results for the SWaT dataset. Furthermore, six variables with the largest JSD values were selected from each dataset.

As shown in the table, even the variable with the largest JSD value had a value close to 0. This indicates that the ICS time-series anomaly generated by the proposed method is not only visually similar to the actual anomaly (as shown in Figure 1) but also statistically does not deviate significantly from the original source data.

Nevertheless, it is worth noting from Figure 6 and Table 2 that there is a difference in similarity between the original source time-series and the anomaly depending on the features. For example, in Figure 6, the KDE between the source data and the synthetic anomaly for HAI’s P1_B2016 is almost similar, whereas a significant difference is observed in P1_B4022. This observation suggests that the level of realism and indistinguishability may vary depending on the features.

The answer for RQ1. Various experimental results presented in this section confirm that the ICS time-series anomaly generated by mutation-based anomaly generation is visually and statistically quite similar to the actual anomaly. However, as shown in the experiments using the SWaT dataset, the proposed method somewhat depends on the reconstruction performance of AVRAE. Therefore, to improve the performance of the proposed method, it is necessary to enhance the model that maps the original data (such as AVRAE) into the embedding space.

5.4. Comparison between One-Class and Binary Classification

The application of techniques for artificially generating ICS time-series anomalies is clear. When given a dataset containing a small amount of anomalies or even only normal data, augmenting the data with anomalies can help mitigate class imbalance to some extent. Particularly, in the absence of such augmentation techniques, anomaly detection often relies on a one-class classification strategy. Therefore, in this section, the proposed synthetic ICS anomaly is generated to augment the given ICS dataset, and binary classification models are trained and compared with several one-class classification models.

Table 3. Performance comparison between one-class and binary classification.

Type	Model	HAI					SWaT
		ACC	REC	PRE	F1	ROC	ACC	REC	PRE	F1	ROC
OC	OCSVM	0.77	0.5	0.04	0.07	0.32	0.61	0.83	0.22	0.35	0.14
	IF	0.94	0.25	0.11	0.14	0.36	0.13	1.0	0.13	0.23	0.18
	LOF	0.97	0.31	0.29	0.3	0.35	0.97	0.31	0.28	0.29	0.35
	LSTM-AE	0.93	0.22	0.31	0.26	0.69	0.94	0.55	0.99	0.71	0.8
BIN	k-SVM	0.37	0.79	0.06	0.11	0.54	0.86	0.71	0.47	0.57	0.85
	RF	0.05	1.0	0.05	0.1	0.47	0.62	0.74	0.22	0.34	0.78
	KNN	0.67	0.35	0.05	0.1	0.53	0.85	0.1	0.24	0.1	0.72
	LSTM-BIN	0.94	0.56	0.44	0.49	0.73	0.95	0.81	0.78	0.79	0.82

shows the anomaly detection performance of several one-class classification models (OC) and binary classification models (BIN) for the HAI and SWaT datasets. The one-class classification models adopted are one-class support vector machine (OCSVM), isolation forest (IF), local outlier factor (LOF), and LSTM-based autoencoder (LSTM-AE). The binary classification models adopted are kernel SVM (k-SVM), random forest (RF), k-nearest neighbor (KNN), and LSTM-based binary classifier (LSTM-BIN). The performance indicators used are accuracy (ACC), recall (REC), precision (PRE), F1 (F1 score), and ROC (ROC-AUC). These models are trained to take the ICS time series of length 100 as input and determine whether an anomaly is present. Specifically, any sequence containing non-normal data for at least one timestep is considered an anomaly.

Note that no model selection process was conducted to enhance performance, aiming to confirm the differences in performance among models based on the dataset differences. Additionally, the LSTM-AE and LSTM-BIN were designed to have as similar complexity as possible, specifically the same number of parameters. The LSTM-AE consists of an encoder and decoder, each composed of a single LSTM layer with a hidden state size of 256, with a dense layer added at the end of the decoder for reconstruction of the original input sequence. The LSTM-BIN consists of two LSTM layers with a hidden state size of 256, applying a dense layer to the average of the hidden states from the last LSTM layer to classify the given sequence. Both the LSTM-AE and LSTM-BIN were trained for 2048 epochs.

Last but not least, all one-class classification models were trained on a training set composed only of normal data, while the binary classification models were trained on a dataset where the original normal data, synthetic normal data (no mutation) by AVRAE, and synthetic anomalies were mixed in a 1:1:1 ratio. In other words, the OC models were trained on the standard dataset, while the BIN models were trained on the dataset augmented with synthetic anomalies. Additionally, the standard training sets for HAI and SWaT do not inherently include anomaly (attack) data. The performance of each model was measured using a test set that did not include any training data and did not contain the synthetic anomalies. Therefore, this experiment not only demonstrates the validity of the binary classifier in anomaly detection but also highlights the effectiveness of the synthetic anomaly.

The results of this experiment were quite intriguing. Basically, accuracy was generally higher for one-class classification models in the case of HAI, while it was higher for binary classification models in the case of SWaT. However, since both HAI and SWaT are imbalanced datasets that overwhelmingly contain normal data, accuracy is not a reliable performance metric for these models. On the other hand, for both HAI and SWaT, the model with the highest F1 score was LSTM-BIN. In fact, most models except for LSTM-BIN exhibited higher recall than precision. This discrepancy is primarily due to the imbalance in the ICS datasets.

Generally, a high recall indicates that the model correctly identifies actual positive samples (anomalies), whereas low precision means that the proportion of actual positives among the samples predicted as positive by the model is low. This suggests that, given the relative scarcity of positive class (anomalies) compared to the negative class (normal data), many samples predicted as positive by the model are likely to be false positives.

Another possible interpretation is that the models, while relatively effectively identifying samples from the positive class, generate many false positives because the differentiation between the negative and positive classes is not pronounced. In other words, while the models successfully detect positive class samples, they frequently confuse them with the negative class, leading to lower precision.

Despite this, LSTM-BIN demonstrated a relatively high F1 score because its recall and precision were not significantly different from each other. This indicates that LSTM-BIN, by using data from both classes, learned an optimal decision boundary that better distinguishes between positive and negative classes. Additionally, LSTM-BIN, being based on a neural network, can learn complex patterns in the data, including non-linearities, allowing it to form a more sophisticated decision boundary compared to other binary classification models. In contrast, LSTM-AE, which is trained only on normal data, learns a less refined decision boundary compared to LSTM-BIN.

The answer for RQ2. Comparing the performance of one-class classification models and binary classification models using the pure ICS dataset and the anomaly-augmented ICS dataset revealed that the neural net-based binary classifier exhibited slightly higher performance. In other words, it was confirmed that dataset augmentation through the generation of ICS time-series anomalies indeed aids in training sophisticated anomaly detection models.

6. Limitations

The mutation-based ICS time-series anomaly generation proposed in this paper has been demonstrated to be effective through various experiments in terms of visual/statistical plausibility and dataset augmentation. This technique is novel, particularly in its application to time-series data, and can be utilized to address imbalances in various time-series datasets beyond ICS datasets. Despite these advantages, the proposed anomaly generation technique has several clear limitations.

• Currently, this study is limited to anomaly generation for numerical time-series data. From a broader perspective, sequential data encompass various forms of data, including text. Although there is potential for this technique to be applied to other types of sequential data, further investigation is required.
• The quality of the synthetic anomaly heavily depends on the performance of the generation model. In this study, AVRAE was adopted as the generation model. While AVRAE is a robust generation model, it still falls short of perfectly capturing the temporal dynamics inherent in time-series datasets.
• The most significant limitation of this technique is its inability to generate scenario-based anomalies. The mutation operation fundamentally relies on randomness. In other words, the generated ICS anomalies do not reflect the causes and consequences of specific cyberattacks. Nevertheless, the proposed technique has sufficient utility because it generates anomalies that consider the correlations between variables in the data through AVRAE.

7. Conclusions

In this paper, a mutation-based ICS time-series anomaly generation method is proposed. This technique first utilizes the encoder of the powerful generation model, AVRAE, to map observations into latent space. Then, it applies carefully designed mutation operations to alter the latent representation. The decoder of AVRAE reconstructs the observations from the mutated latent representations. To design appropriate mutation operations, commonly used ICS datasets were analyzed to derive the types of anomalies they contain. Based on this analysis, mutation operations were proposed to generate point anomalies, contextual anomalies, and collective anomalies. Evaluations using the HAI and SWaT datasets confirmed that the proposed method could generate ICS time-series anomalies that are both visually and statistically plausible.

Furthermore, by using the augmented ICS dataset with synthetic anomalies, various one-class classification models and binary classification models were compared. The results empirically demonstrated that a robust binary classifier could be trained using the augmented dataset. This study presents an effective means to augment ICS datasets, which are often plagued by imbalance issues. The proposed technique is transferable to other time-series datasets with similar characteristics, although its application to sequential datasets of a different nature, such as text, requires further research. Additionally, the method’s dependence on the performance of the underlying generation model and its inability to generate scenario-based anomalies were identified as limitations. Consequently, future research will naturally extend to applying the proposed method to various datasets and exploring more powerful anomaly detection models.