Chemical Gas Source Localization with Synthetic Time Series Diffusion Data Using Video Vision Transformer

Abstract

Gas source localization is vital in emergency scenarios to enable swift and effective responses. In this study, we introduce a gas source localization model leveraging the video vision transformer (ViViT). Utilizing synthetic time series diffusion data, the source grid is predicted by classifying the grid with the highest probability of gas occurrence within the diffusion data coverage. Through extensive experimentation using the NBC-RAMS simulator, we generate large datasets of gas diffusion under varied experimental conditions and meteorological environments, enabling comprehensive model training and evaluation. Our findings demonstrate that the ViViT outperforms other deep learning models in processing time series gas data, showcasing a superior estimation performance. Leveraging a transformer architecture, the ViViT exhibits a robust classification performance even in scenarios influenced by weather conditions or incomplete observations. Furthermore, we conduct an analysis of accuracy and parameter count across various input sequence lengths, revealing the ability of the ViViT to maintain high computational efficiency while achieving accurate source localization. These results underscore the effectiveness of the ViViT as a model for gas source localization, particularly in situations demanding a rapid response in real-world environments, such as gas leaks or attacks.

1. Introduction

Chemical gases, when released into the atmosphere, spread over time and cause widespread harm to humans across large areas [1,2]. Identification of the gas source is important for rapid response in the event of a gas diffusion accident. For instance, in industrial accidents like toxic gas leaks or the dispersal of combustion gases resulting from fires, pinpointing the gas source aids in analyzing the causes of incidents and mitigating further damage. Similarly, in military contexts where gas bomb attacks occur, accurate identification of the gas source enables the implementation of effective defense measures. Gas source localization involves estimating the initial location of gas emission using diffusion data, playing a critical role in responding to the gas diffusion incidents [3]. Strategies for gas source localization typically rely on static sensor networks [4,5,6,7] or mobile sensors [8,9], depending on the deployment configuration. Our focus lies in leveraging static sensor networks, where sensors are deployed to monitor gas concentrations over time, facilitating the detection of changes in diffusion patterns within predefined areas.

To estimate the origin position of a gas source, researchers utilize a gas dispersion process model that incorporates the origin position as a modeling variable. Subsequently, the origin position is inferred by back estimating the modeling variable from the modeled gas diffusion distribution and the measured diffusion distribution information. However, pinpointing the exact location of chemical gases poses challenges due to their irregular diffusion patterns and the complex atmospheric conditions that influence their dispersion [3,6,10]. In response to these challenges, studies on gas source localization within static sensor networks have taken diverse approaches. Foremost among these are optimization methodologies and probabilistic approaches based on Bayesian inference. Optimization methods [11,12] encompass gradient-based techniques that leverage various adaptations of least squares methodologies, representing a field ripe with substantial research advancements. Additionally, direct search methods [13] have made notable progress in this domain. Furthermore, endeavors employing genetic algorithms [14,15] have led to significant breakthroughs. Probabilistic methodologies employ techniques such as Markov chain Monte Carlo [16], sequential Monte Carlo [17], differential evolution Monte Carlo [18], and polynomial chaos quadrature [19], demonstrating notable advancements.

Recent advancements have seen the emergence of approaches employing machine learning models such as support vector machine and kernel ridge regression for gas source localization [4]. Moreover, active research endeavors are underway exploring methods utilizing deep neural networks, renowned for their efficacy in processing complex data, including time series data [20]. Yeon et al. [21] employed a deep neural network (DNN) to estimate gas source locations in 2D coordinates from sensor array data. Cho et al. [22] utilized DNN and random forest classifiers to track gas sources using data generated from computational fluid dynamics (CFD) simulations. Bilger et al. [5] implemented convolutional long short-term memory neural network (CNN-LSTM) architectures to extract temporal and spatial features from gas concentration data collected from the real environment, enabling the estimation of gas source locations in static sensor networks amidst varying air conditions. Similarly, Yamamoto et al. [23] leveraged a long short-term memory deep neural network (LSTM-DNN) to estimate gas source locations using gas sensor arrays and anemometers. Kim et al. [24] proposed a model integrating feedforward neural networks and recurrent neural networks with long short-term memory (LSTM-RNN) to predict gas source locations using data derived from CFD simulations. While RNN or LSTM traditionally processes time series data, recent attention has shifted towards transformers [25], originally introduced for natural language processing, due to their remarkable performance in handling time series data [26]. Operating on the attention mechanism, the transformer model adeptly learns correlations within input tokens. Son et al. [27] deployed a transformer model to recursively track chemical gas dispersion and estimate source localization. The utilization of a 2D convolutional layer to convert each gas diffusion frame into a token facilitated this process. Transformer models have recently found applications across diverse machine learning tasks, including image and video processing, showcasing a performance surpassing that of existing models [28,29,30]. In image and video processing, transformers excel by extracting multiple patches or tubes with spatial or spatio-temporal characteristics, surpassing the simple tokenization of individual frames.

In this study, we introduce a deep neural network utilizing a video vision transformer (ViViT) for gas source localization. This model processes time series gas diffusion data and predicts the grid of gas sources within the area where diffusion data are available. We rigorously train and evaluate the model using diffusion data under diverse experimental conditions and meteorological environments generated via the nuclear biological chemical reporting and modeling system (NBC-RAMS). To validate the effectiveness of the ViViT in gas source localization, we conduct a comparative analysis with existing deep learning methods. Our results not only demonstrate a superior performance compared to other models but also highlight the structural efficiency of the ViViT, which achieves a high performance even with fewer input frames. Section 2 details the experimental data sources and the model structure employed in our study. Section 3 outlines the experimental setup, method validation, and performance analysis against other models. Finally, Section 4 provides the conclusion.

2. Materials and Methods

2.1. Data Source

We acquire time series gas dispersion data using the NBC-RAMS simulator, which simulates gas diffusion distribution following a gas bomb explosion under specific meteorological and experimental conditions in predefined spatial regions. NBC-RAMS established by the Agency for Defense Development of the Republic of Korea is a simulation program with purposes and functions similar to the Hazard Prediction and Assessment Capability software (version 6) [31] developed by the Defense Threat Reduction Agency in the United States. Previous studies have utilized NBC-RAMS for similar purposes [27,32]. NBC-RAMS comprises a meteorological model and a pollution diffusion model. The meteorological model generates a three-dimensional gridded meteorological field, incorporating factors such as wind orientation, wind velocity, and temperature, accounting for topography, land cover, and building heights within the calculation area based on input weather information. The pollution diffusion model calculates the transport and diffusion process of pollutants, including chemical, biological, radiological, and smoke agents within the calculation area, utilizing the generated meteorological field.

The data generation area encompasses a 5 km × 5 km region within the metropolitan area of South Korea, divided into grids of 200 m × 200 m. Gas concentrations are computed at the center of each grid. Simulations extend for 30 min from the time of the gas incident, with data recorded every minute starting from 1 min post-explosion. Each incident scenario yields gas diffusion data comprising 30 frames of size $(H,W,C)$, where H = 25, W = 25, and C = 1. To diversify the gas diffusion data, we vary parameters such as the gas generation grid, warhead payload mass, and meteorological environments. SF6 is used as the gas in the diffusion simulation, with the payload mass ranging from 0.0005 kg to 0.0025 kg in 0.0005 kg increments. Meteorological conditions are set by adjusting the wind speed from 1 m/s to 3 m/s in 0.5 m/s intervals and the wind direction from 0° to 315° in 45° intervals. The temperature remains constant at 10 °C. We generate data by performing the various experimental conditions described above for each $N_C=H\times W$ grid, representing a potential gas source. The generated dataset comprises 200 experimental conditions per 625 grids, totaling 125,000 instances comprising 3.75 million frames. Of these, 70% are allocated for training, 20% for validation, and 10% for testing. Figure 1a presents a sample of data generated using NBC-RAMS. Frames depict gas dispersion at 2 min intervals from 1 to 27 min post-explosion. The gradual spread of gas over time and its varying distribution across each time step are discernible. The presence of a gas source point, represented by a white flask, and its subsequent diffusion are apparent, illustrating the influence of both inherent chemical properties and the weather environment on dispersion. As time progresses, gas disperses, and concentrations at the source diminish due to atmospheric dilution.

We conduct data preprocessing, encompassing sequence extraction and normalization, on the acquired data. During sequence extraction, we extract T consecutive frames from each simulation data. Following gas generation, the immediate environment experiences dynamic instability due to the sudden introduction of chemical substances, potentially causing irregularities in gas diffusion distribution. These irregularities may induce data instability, thereby impeding model generalization. As time elapses post-explosion, the surrounding environment stabilizes, and the gas diffusion pattern becomes consistent. Hence, we commence the extraction process from the 6th frame among the 30 frames obtained for each scenario. The extracted consecutive frames undergo validation. If the number of frames devoid of any concentration exceeds $\frac{T}{3}$, the frames are disregarded, and the extraction process recommences from the subsequent frame. This ensures that the dataset encompasses ample gas diffusion information. Post-extraction, min–max normalization is applied to ensure that all frames fall within the range of 0 to 1. The preprocessed time series frames $\mathbf{V}\in {\mathbb{R}}^{T\times H\times W\times C}$ serve as input to the model, with the label $y\in {\mathbb{R}}^{N_C}$ representing the grid where gas occurred in the input $\mathbf{V}$. Figure 1b depicts the result of preprocessing the samples portrayed in Figure 1a.

2.2. Proposed Method

We present a deep neural network for gas source localization using the ViViT. This model takes time series gas diffusion data $\mathbf{V}$ as the input and accomplishes gas source localization by classifying the grid where gas occurred among $N_C$ grids. Transformers, which employ attention mechanisms, were initially introduced in [25] to process time series data. They have since been applied not only to natural language processing [26] but also to various fields such as image and video classification [28,29], demonstrating a remarkable performance. Our model is designed based on the video vision transformer [30], which employs the transformer structure for video classification. The model structure comprises embedding layers, transformer encoders, and a classifier, as illustrated in Figure 2.

Within the embedding layers, the input $\mathbf{V}$ undergoes conversion into a sequence of tokens ${\mathbf{z}}^0\in {\mathbb{R}}^{\left(N+1\right)\times d}$ expressed as

$$\begin{array}{c}\hfill {\mathbf{z}}^0=\left(z_{cls},\mathbf{E}{x}_1,\cdots ,\mathbf{E}{x}_N\right)+\mathbf{p},\end{array}$$

where $z_{cls}\in {\mathbb{R}}^d$ represents the learned classification token [26]. Each $\mathbf{E}{x}_i\in {\mathbb{R}}^d$ token corresponds to a tublet extracted from the video ${\mathbf{x}}_i\in {\mathbb{R}}^{t\times h\times w\times C}$ and linearly projected, while $\mathbf{p}\in {\mathbb{R}}^{\left(N+1\right)\times d}$ denotes a learned positional embedding. Nonoverlapping spatio-temporal tublets $x_i$ are extracted from $\mathbf{V}$. Given the tublet dimensions of t, h, and w for the temporal, height, and width, respectively, the total number of extracted tublets is $N=n_t\times n_h\times n_w$, where $n_t=\lfloor \frac{T}{t}\rfloor$, $n_h=\lfloor \frac{H}{h}\rfloor$, and $n_w=\lfloor \frac{W}{w}\rfloor$. Figure 3 illustrates the extraction of tublets spanning the spatio-temporal input $\mathbf{V}$. These tublets $x_i$, for i = 1, 2, ⋯, N, are linearly projected by $\mathbf{E}$ into tokens $\mathbf{E}{x}_i$. Subsequently, $z_{cls}$ is prepended to these tokens, resulting in the sequence $\left(z_{cls},z_1,\cdots ,z_N\right)$, where $z_i=\mathbf{E}{x}_i$. This sequence, augmented by the learned positional embedding $\mathbf{p}$, forms the input ${\mathbf{z}}^0$ for the transformer encoders.

The sequence of tokens ${\mathbf{z}}^0$ passes through L transformer layers, resulting in ${\mathbf{z}}^L\in {\mathbb{R}}^{\left(N+1\right)\times d}$, where the dimension of each token remains d throughout each layer. Each layer consists of multi-headed self-attention (MSA) [25], layer normalization (LN) [33], and multi-layer perceptron (MLP) blocks:

$$\begin{array}{cc}\hfill {\mathbf{y}}^l& =\mathrm{MSA}\left(\mathrm{LN}\left({\mathbf{z}}^l\right)\right)+{\mathbf{z}}^l,\hfill \\ \hfill {\mathbf{z}}^{l+1}& =\mathrm{MLP}\left(\mathrm{LN}\left({\mathbf{y}}^l\right)\right)+{\mathbf{y}}^l,\hfill \end{array}$$

where l = 0, ⋯, L− 1. The MSA consists of k self-attentions operating in parallel, while the MLP block comprises two dense layers and the GELU activation function [34] between the layers, as developed in [30]. The number of nodes in the two dense layers of the MLP is set to $d_{ff}$ and d, respectively.

Among the output tokens of the transformer encoders, $z_{cls}^L\in {\mathbb{R}}^d$ is forwarded to the classifier, which predicts the label $\widehat{y}\in {\mathbb{R}}^{N_C}$ of the grid where gas detonation occurred. $z_{cls}^L$ is the token created by passing the classification token $z_{cls}$ through the encoder. The classifier comprises LN and a single dense layer (CLS), expressed as

$$\begin{array}{c}\hfill \widehat{y}=\mathrm{CLS}\left(\mathrm{LN}\left(z_{cls}^L\right)\right).\end{array}$$

The model is trained using the Adam optimizer [35], aiming to minimize the loss function:

$$\begin{array}{c}\hfill L(y,\widehat{y})=-\frac{1}{N_B}\sum _{i=1}^{N_B}\sum _{c=1}^{N_C}\log \left(\frac{\exp \left({\widehat{y}}_{i,c}\right)}{{\sum }_{j=1}^{N_C}\exp \left({\widehat{y}}_{i,j}\right)}\right)y_{i,c}+\frac{1}{2}\lambda {\parallel W\parallel }_2^2,\end{array}$$

where $N_B$ is the number of data included in the batch. The term ${\parallel W\parallel }_2^2$ represents the $L2$ regularization term, with the coefficient $\lambda$ set to ${10}^{-4}$. To prevent overfitting, dropout is applied to the embedding layer and the transformer layers with a dropout rate of $0.05$. The initialization of $z_{cls}$ and $\mathbf{p}$ is set to 0 and is trained during the training process. The initial learning rate of the Adam optimizer is ${10}^{-3}$.

3. Results

3.1. Experiment Setting

The model undergoes training and validation using the respective datasets and is subsequently evaluated on the test dataset. The weights of the model are updated using the training data, with the validation loss computed using the validation data at each epoch. The model achieving the lowest validation loss across all epochs is selected as the optimal one for evaluation. The following hyperparameters are utilized in the experiments: L = 10, k = 8, $d_{ff}$ = 512, d = 128, t = 2, h = 13, and w = 13. During training, the total epochs and batch sizes are set to 300 and 512, respectively. Evaluation metrics include the f1-score, precision, recall, and accuracy, commonly employed for a classification performance analysis. Additionally, the average error distance, computed as the distance between the predicted grid center and the actual grid center of all test data, is used as an evaluation metric, considering each grid unit corresponds to a length of 200 m.

To facilitate the comparison and analysis of performance, CNN-LSTM and 3D-CNN models are considered. The CNN-LSTM model comprises a CNN block, LSTM block, and DNN block, akin to those introduced in [5], with a hidden dimension size of 625 to accommodate the larger width and height of frames than those of [5]. The 3D-CNN model utilizes the VideoGasNet architecture proposed in [36], which comprises six Conv-Pool structures [37] and two fully connected layers. The convolution layers in the Conv-Pool structure have a kernel size of (3, 3, 2), considering that the input tokens of the ViViT span two channels of data, and the dropout rate is set to 0.5, consistent with CNN-LSTM. The configuration of the fully connected layers mirrors that of the DNN blocks of CNN-LSTM. The training settings for the control groups are the same as those of the ViViT, with $L2$ regularization also applied.

3.2. Performance Analysis

Table 1. Performance evaluation scores of models on test data.

Model	f1-Score	Precision	Recall	Accuracy	Average Error Distance [m]	Params [M]
CNN-LSTM	0.9110	0.9189	0.9134	0.9217	17.7394	9.86
3D-CNN	0.8102	0.8298	0.8163	0.8456	37.1616	6.74
ViViT	0.9363	0.9413	0.9383	0.9478	12.6894	2.10

presents the performance metrics and the number of parameters for each model when the input sequence length is T = 10. The ViViT achieves metrics of 0.93 or higher across all classification performance metrics, demonstrating the highest accuracy in source grid prediction. Additionally, it exhibits the lowest error distance, with an average distance of 12.689 m between the predicted source location and the actual one. CNN-LSTM shows a classification performance of 0.91 or higher and an average error distance of 17.7394 m, ranking second in performance. 3D-CNN exhibits metrics of 0.81 or higher, showing the lowest performance among the comparison groups, with an average error distance of 37.1616 m, approximately three times larger than that of the ViViT. The ViViT comprises 2.10 million parameters, which is approximately 4.6 times fewer than CNN-LSTM and 3.3 times fewer than 3D-CNN. Despite the potential for learning more complex nonlinear relationships with a larger number of parameters, the ViViT demonstrates a superior performance with the fewest parameters.

Figure 4 illustrates three samples from the test data along with the predicted gas source locations by each model. For each sample, the results display 10 data sequences used as the input on the left and the prediction results of each model alongside the ground truth on the right. In Figure 4a, the gas distribution widens over time, indicating a spreading pattern observed across the entire area. Since the distribution of grids showing high concentrations in each frame does not change significantly, it can be seen that the influence of weather does not have a significant effect on gas diffusion, especially when the wind is weak. This sample not only allows for the observation of the entire distribution that changes over time but also confirms that all models accurately predict the source grid because the change in the concentration distribution is monotonic. In Figure 4b, the distribution of gas is observed to move to the bottom right over time, indicating a stronger influence of the wind compared to Figure 4a. Although the results of CNN-LSTM and 3D-CNN are predicted to be close to the actual source location, they do not accurately predict the source location when the gas distribution moves or changes due to the influence of weather. On the other hand, the ViViT accurately localizes the source grid, overcoming these difficulties. In the case of Figure 4c, the distribution of gas moves significantly to the left over time due to the influence of weather, and even in the last two frames, it extends beyond the detection area with no observable values. CNN-LSTM and 3D-CNN fail to accurately estimate the location of the source, with larger errors compared to Figure 4b. However, the ViViT accurately predicts the source grid even under such conditions. Through these examples, it is evident that the ViViT outperforms other models in gas source localization. The ViViT can accurately predict the grid of the source even if the distribution changes significantly over time due to the influence of weather, and even if the distribution is outside the area of interest or some observations are missing. Considering the limitation that it is difficult to respond when gases escape the designated area through diffusion, the ViViT offers an efficient solution to this challenge. The rationale behind the superior performance of the ViViT lies in its utilization of a transformer structure, enabling it to effectively acquire spatio-temporal features compared to other models for processing time series data. While training a model with a transformer structure demands a substantial dataset, such data can be readily acquired through simulations. Moreover, an alternative approach involves loading the parameters of a pretrained model and fine-tuning only some of the parameters with a small dataset.

3.3. Sequence Length Variants

We vary the sequence length T∈ {8, 10, 15, 20, 25} and analyze its impact on the performance of the models for gas source localization. For T = 20 and T = 25, CNN-LSTM and 3D-CNN are trained for 500 epochs. Given that the number of parameters of these models increases with the number of input frames, we extend the training epochs to ensure convergence during the learning process.

Figure 5 illustrates the accuracy and number of parameters of the models with changes in sequence length. It is evident from Figure 5a that all models tend to improve their performance as the number of input frames increases. The ViViT consistently demonstrates the highest performance across all sequence lengths. Even in the T = 8 condition, utilizing the fewest frames, the ViViT exhibits superior performance compared to all the other length conditions of the other models. Using a larger number of input frames allows for more information utilization and potentially a higher performance. However, in practical scenarios, there is a trade off in data acquisition time. To respond swiftly to gas leaks or incidents, a model capable of accurate prediction with a limited number of frames is crucial. Thus, the ability of the ViViT to achieve accurate source localization with a small number of frames offers a significant advantage in real-time applications. While the performance of CNN-LSTM improves with increasing frames, its accuracy gradually converges, showing a similar performance between the T = 20 and T = 25 conditions. 3D-CNN consistently exhibits a lower performance across all sequence lengths.

We evaluate the computational efficiency of the transformer-based model structure in Figure 5b. The transformer architecture used in the ViViT ensures that the number of parameters to be learned in the encoder or classifier remains constant even as the number of input frames increases. Although the parameters for $z_{cls}$ and positional embedding $\mathbf{p}$ increase with the sequence length, they contribute only a fraction of the total number of parameters. In contrast, CNN-LSTM and 3D-CNN exhibit linear increases in parameters proportional to the number of input frames. The parameters of CNN-LSTM grow with the size of the hidden state of the LSTM and the number of nodes in the first layer of the DNN block. Furthermore, the sequential computation of LSTM outputs for each frame adds computational inefficiency. Similarly, the parameter increase in 3D-CNN is proportional to the product of the number of filters in the last convolutional layer, the dimensions of the output tensor of the last convolutional layer, and the number of nodes in the following dense layer. These models, with their larger parameter counts, demand extensive training and computational resources, making real-world deployment challenging. Even with increased epochs, CNN-LSTM and 3D-CNN fail to surpass the performance of the ViViT, which is particularly evident in the T = 20 and T = 25 conditions. Notably, the excessive parameters of 3D-CNN hinder training convergence in the T = 20 and T = 25 conditions, resulting in a slight performance decrease. Thus, the consistent performance of the ViViT across varying sequence lengths and its constant parameter count make it highly efficient for real-world applications.

4. Conclusions

In this study, we introduce a gas source localization model utilizing the ViViT. The model processes time series diffusion data to predict the source grid by classifying the grid with the highest probability of gas occurrence within the covered area. Through extensive experimentation using the NBC-RAMS simulator, we generate large datasets of gas diffusion under varied experimental conditions and meteorological environments. Subsequently, we train and evaluate the model using these data. Our findings demonstrate that the ViViT outperforms other deep learning models in processing time series gas data, showcasing a superior estimation performance. Leveraging a transformer architecture, the ViViT exhibits robust classification performance even in scenarios heavily influenced by weather conditions or when observations are incomplete. Furthermore, we conduct an analysis of accuracy and parameter count across various input sequence lengths. The ViViT not only achieves a high estimation performance with a small number of frames but also maintains high computational efficiency by keeping the number of learned parameters constant regardless of sequence length. These results underscore the effectiveness of the ViViT as a model for gas source localization, particularly in situations demanding a rapid response in real-world environments, such as gas leaks or attacks. In future work, since the data generated through simulation is limited to one type of gas even though various experimental conditions are used, we aim to conduct experiments by generating gas dispersion data with various volatile characteristics. Additionally, we intend to explore the application of the ViViT in real-world scenarios, leveraging data collected from gas diffusion experiments conducted in real environments.