Detection of Harmful H₂S Concentration Range, Health Classification, and Lifespan Prediction of CH₄ Sensor Arrays in Marine Environments

Abstract

Underwater methane (CH₄) detection technology is of great significance to the leakage monitoring and location of marine natural gas transportation pipelines, the exploration of submarine hydrothermal activity, and the monitoring of submarine volcanic activity. In order to improve the safety of underwater CH₄ detection mission, it is necessary to study the effect of hydrogen sulfide (H₂S) in leaking CH₄ gas on sensor performance and harmful influence, so as to evaluate the health status and life prediction of underwater CH₄ sensor arrays. In the process of detecting CH₄, the accuracy decreases when H₂S is found in the ocean water. In this study, we proposed an explainable sorted-sparse (ESS) transformer model for concentration interval detection under industrial conditions. The time complexity was decreased to O (n log_n) using an explainable sorted-sparse block. Additionally, we proposed the Ocean X generative pre-trained transformer (GPT) model to achieve the online monitoring of the health of the sensors. The ESS transformer model was embedded in the Ocean X GPT model. When the program satisfied the special instructions, it would jump between models, and the online-monitoring question-answering session would be completed. The accuracy of the online monitoring of system health is equal to that of the ESS transformer model. This Ocean-X-generated model can provide a lot of expert information about sensor array failures and electronic noses by text and speech alone. This model had an accuracy of 0.99, which was superior to related models, including transformer encoder (0.98) and convolutional neural networks (CNN) + support vector machine (SVM) (0.97). The Ocean X GPT model for offline question-and-answer tasks had a high mean accuracy (0.99), which was superior to the related models, including long short-term memory–auto encoder (LSTM–AE) (0.96) and GPT decoder (0.98).

1. Introduction

Marine combustible ice is an ideal energy source for low-carbon societies [1,2,3] because its combustion produces ten times more energy than gasoline, coal, or natural gas [4,5,6,7]; so, coastal countries compete to develop combustible ice resources. Methane (CH₄) gas formed after the mining of marine combustible ice can be transported by laying submarine pipelines, and pipeline safety monitoring has become an important work during transportation. Once the CH₄ pipeline leaks, it will not only cause damage to the natural environment but also directly affect the energy supply problem and even cause social panic. Therefore, it is of great theoretical value and practical significance to strengthen research on underwater CH₄ sensor arrays technology [8,9] and solve technical problems such as underwater CH₄ pipeline leakage monitoring and positioning. In addition, hydrothermal activities of submarine “hot springs” and submarine volcanic activities will also release CH₄ gas, and the monitoring of the CH₄ information also requires underwater CH₄ sensor arrays. In the process of underwater CH₄ detection, hydrogen sulfide (H₂S) gas is often present, which is harmful for the performance of the CH₄ sensor [10,11] and seriously affects the health and life of the CH₄ sensor arrays.

H₂S gas and other sensors’ fault situations affect CH₄ detection accuracy [12,13,14,15,16]. To study whether or not the CH₄ sensor works properly, the accurate concentration of H₂S gas in the external environment should be determined at first. Different concentrations of H₂S gas have different effects on CH₄ sensor poisoning; therefore, the different levels of H₂S gas poisoning can assess the risk of the detection mission and promote the smooth execution of the mission. To detect the working conditions of sensor arrays, health management decision-making is essential. A solution must be established to ensure that the health status of all sensors in the system is clear and that the system is stable.

Data acquisition, failure detection and diagnosis, failure recovery and prediction, health evaluation, and maintenance decision-making are featured in the prognostics and health management (PHM) decision synthesis technique [17,18,19,20]. The reliability and safety of these systems can be enhanced for the purpose of making health management decisions [21,22,23,24]. According to collected data, a system’s health status can be predicted and evaluated to make more informed health management decisions [25,26,27,28]. Previous research has been used to complete these types of health evaluations and to monitor conditions [29,30,31,32]. On the basis of this theory and technical applications in previous work, the current health management decision-making method can be divided into three categories: data-based maintenance decision-making, model-based maintenance decision-making, and reliability-based decision-making [25,26,27,28,33]. The environment has a significant effect on sensor systems, which have changeable working conditions and complex structures. Because the baselines of the same concentrations change at various times, the failure range is hard to define. Therefore, to maintain suitable decision-making conditions for these sensor systems, it is essential to develop appropriate models.

D–S evidence theory [33,34,35], Bayes theory [36,37,38,39], and fuzzy set theory [40,41,42,43,44], as the traditional reliability-based methods, have faced significant challenges because of the various data types and information uncertainty. Conflicting evidence has caused results from the D–S evidence theory to differ from the user’s understanding. Using the D–S evidence theory to address conflicting results under system failure remains challenging. The basis of Bayes theory method is prior probability. When a prior probability is known, it is easy to obtain accuracy results; however, when a prior probability is not known, it is difficult to obtain, and the ability to apply this information is limited. When considering maintenance decision-making tasks, because of the logical reasoning of fuzzy set theory, several subjective factors can affect the description of information. As a result, objectivity in the representation and processing of this information will be lacking.

In this study, we found that it was difficult to apply these maintenance decision-making methods. Shen et al. [31] proposed multifunctional sensor health reliability to assess the working state of the sensor. The quantitative description of health information is called health reliability (HRD). The failure of a single sensor was not effectively revealed when too many sensors were used [32]. To enhance the poison concentration, residual life, and health-level information of the sensor’s health, we developed a nested intelligent health management and life prediction system.

An explainable sorted-sparse (ESS) transformer model was proposed for toxic gas concentration interval detection of CH₄ sensor arrays in ocean conditions. The proposed model is a data-based supervised-learning method, requiring label data for training and presenting high accuracy for industrial promotion and application. The time complexity can be decreased to O (n log_n) using the ESS block. In addition, we propose the Ocean X generative pre-trained transformer (GPT) model for intelligence health management and lifetime prediction of CH₄ sensor arrays by embedding the GPT model with ESS transformer models. The proposed model provides a substantial amount of expert information about sensor array failures and electronic noses by text and speech. This concept also used a question-and-answer system framework.

(1)

To limit the time complexity to O (n log_n), we changed the traditional self-attention mechanism to ESS attention. The proposed model used the idea of sorting the product weights of the query and the key value from high to low. Using the original distribution of the training data, we retained the first third and sparse the rest with an explainable mask.

(2)

We enhanced the attention of the Ocean X GPT model with a Rotary Position Embedding (RoPE) attention mechanism. This attention not only had RoPE position information but also retained the original information. Applying the idea of a residual network, we added the original data to the query and key, which were mapped by RoPE. Then, we mapped the total return data into the RoPE again. For the question-and-answer task, the model obtained the position information between the question and the answer from the first RoPE operation. Then, we increased the accuracy of the answer according to the second RoPE operating by combining the question and the target answer.

(3)

We proposed a real-time interactive health management and life prediction system. The basic framework was the Ocean X GPT model with an ESS transformer model embedded inside. When performing the task, according to the keyword in the question, the program jumped into the ESS transformer model and waited for input data. After we entered random validation data x into the trained toxic gas concentration interval detection model (ESS transformer), the model returned the corresponding concentration information, poisoning grade, and remaining life information as a voice broadcast.

2. Theoretical Fundamentals

2.1. ESS Mask

Figure 1 shows the proposed method, which was the result of the ESS mask of temporal and spatial dimensions in a factorized self-attention model [45]. We called the proposed factorized attention “fixed attention”, as shown in Figure 1a. Fixed attention has two mechanisms (FA1 and FA2), where FA1 represents the triangular region (light blue), and FA2 represents the vertical fixed region (sky blue) with L-separated distances. For each current token (deep blue), it is possible to traverse to its left (light blue) until encountering the first token selected by FA2 (sky blue). We called the factorized attention “strided attention”, as shown in Figure 1b. Strided attention has two mechanisms (SA1 and SA2), where SA1 represents the galaxy-shaped region (light blue), and SA2 represents the hackly bar-shaped region (sky blue). In SA1, each token can focus only on the L tokens adjacent to the left. In SA2, each token can focus only on the token to its left. We selected these attended tokens by counting them to the left of their position, and there was one token for every L. L equaled 3 from the original model. Based on this model and the decoder mask shown in Figure 1c, we proposed the explainable sorted–fixed mask shown in Figure 1d.

To map a matrix of input embeddings X to an output matrix, we used a self-attention layer. We parameterized this layer according to a connectivity pattern $T=\{T_1, \dots , T_n\}$, where T_i is the set of indices of the input vectors to which the ith output vector attends. We used a weighted sum of transformations of the input vector for the output vector, as follows:

$${\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{d}\left(\mathrm{X};T\right)=\left(\mathrm{a}\left(x_i;T_i\right)\right)}_{iϵ\left\{1, \dots , l\right\}}$$

(1)

where $\mathrm{a}(x_i;T_i)$ denotes the traditional attention score; $T_i=\{j:j\le i\}$ denotes the full self-attention for autoregressive models, which enabled each element to attend to its own position as well as each of the previous positions.

More precisely, the set T_i is divided into p non-overlapping subsets, and the subset m is represented as $A_i^{\left(m\right)}\subset T_i,m=1,\dots ,p$; so, the maximized path of the output situation i and random j is p + 1. For example, if (j, c, d, e, …, i) is the index path of i and j, $j\in A_c^{\left(1\right)},c\in A_d^{\left(2\right)},d\in A_e^{\left(3\right)}\dots$.

We defined the two sparse patterns shown in Figure 1d: (1) one head attended to the previous l locations, and (2) the other head attended to every lth location, where l reflects the stride and was close to $\sqrt{n}$ (called “strided attention”). If the data had a structure that was naturally aligned with the stride (e.g., some types of music), this proved to be a convenient formula. For Equation (2), l = 6 corresponds to the number of sensor arrays:

$$A_i^{\left(1\right)}=\left\{k,k+1,\dots ,i\right\}$$

(2)

where k is the maximum of (0, i − l).

$$A_i^{\left(2\right)}=\left\{j:(i-j)\right\} modl=0$$

(3)

For the second pattern called “fixed attention”, we propagated the summary of previous locations to all future cells as follows:

$$A_i^{\left(1\right)}=\left\{j:⌊{\displaystyle \frac{j}{l}}⌋=⌊{\displaystyle \frac{j}{l}}⌋\right\}, where k=\max \left(0,i-l\right)$$

(4)

where k denotes the maximum of (0, i − l).

$$A_i^{\left(2\right)}=\left\{j:j mod k\in \left\{k-c,\dots ,k-1\right\}\right\}$$

(5)

where c is a hyperparameter.

In fixed attention, it is important to pay attention to tokens in other locations regardless of the current token location. This kind of attention gives more attention to the global weight information, where c = 5 corresponds to the number of sensor arrays.

Another concept of this sparse mask includes the probability distribution information of the device array data used in this study, which required a tedious derivation of its formula to verify results (see Section 2.2).

2.2. ESS Attention

Inspired by the prob-sparse self-attention mechanism from Informer [45], we apply ESS attention as our basic model. We observed the same relationship between function detection and self-attention with the prob-sparse self-attention mechanism (see details in Supplementary Materials).

The original attention can be defined as follows:

$$A\left(q_i,k,v\right)=\sum _j{\displaystyle \frac{k\left(q_i,k_j\right)}{{\sum }_{l}k\left(q_i,k_l\right)}}{V}_j=D_{p\left(k_{j|q_i}\right)}\left[V_j\right]$$

(6)

where $p\left(k_j\right|q_i)=k(q_i,k_j)/{\sum }_{l}k(q_i,k_l),$ and $k(q_i,k_j)$ selects the asymmetric exponential kernel $\exp \left(q_i,k_j^T\right)/\sqrt{n}$.

We proposed the max-mean measurement to replace the previous methods. The specific proof method is given in the Supplementary Materials. Calculating the sparsity score for each query would result in additional computation, and that assumption can present the dot-product results following a long-tail distribution. Therefore, each query sparsity score can be calculated with only some of the sampled keys. Therefore, under the long-tail distribution, it was only necessary to compute M by randomly sampling the set of dot-product pairs and by filling the other pairs with 0 values. Sparse Top-U was selected as Q; and for M, the max-operator was stable and insensitive to 0.

$$\overline{N}\left(q_i,k\right)=\underset{j}{\max }{\displaystyle \frac{q_i,k_j^T}{\sqrt{d}}}-{\displaystyle \frac{1}{L_K}}\sum _{j=1}^{L_K}{\displaystyle \frac{q_i,k_j^T}{\sqrt{d}}}$$

(7)

$$\stackrel{def}{\Rightarrow }\underset{j}{\max }\{q_i,k_j^T\}-\underset{j}{\mathrm{mean}}\{q_i,k_j^T\}$$

According to the max-mean method, the result of multiplying the query and key matrix is shown in Figure 2. From Figure 2, the total data were $\overline{N}\left(q_i,k\right)$, where i is from 1 to 607. By comparing the results shown in Figure 2 with the following data, we observed that the first third of the data had a larger value range, which helped the model distinguish among the different types of data. This distinction information among different classifications was mostly in this area. Therefore, we only kept the first third and took the average value later. Compared with Figure 1d, we retained one-third of the whole 25 × 25 mask.

We identify a significant difference between the original and proposed models (Figure 3). Specifically, the matrix product of the query and key matrix did not directly calculate the score. We first multiply Q by the sorted-sparse mask and then by K, thus limiting the time complexity to O (n log_n). The pseudocode of ESS attention is given in Algorithm 1.

Algorithm 1 ESS Attention
Function	ESS Attention (Xinput)
｜	Q, K ← Xinput
｜	$\mathrm{Q}, \mathrm{K} \in$ (Batch, Head, 608, 25)
｜	K ← Randint (len(X) = 25)
｜	QK_sample ← Q ∗ K
｜	M_top ← Sort (QK_sample)
｜	Visualized (M_top)
｜	QSortedSparse ← Q ∗ Mask
｜	Score = QSortedSparse ∗ KT
End

For the QK_sample, the output is as follows:

$${\mathrm{Q}\mathrm{K}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}=\mathrm{Q}\ast \mathrm{K}$$

(8)

where Q and K are from the X_input.

Then, the sorted of QK_sample is as follows:

$$\mathrm{S}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{e}\mathrm{d}\left({\mathrm{Q}\mathrm{K}}_{\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}}\right)=\overline{N}\left(q_i,k\right)$$

(9)

where i is from 1 to 607.

The visualization of M_top is shown in Figure 2. After analyzing the information in Figure 2, we confirmed that only the first third of the data was taken. We combined it with the ESS mask, and then we made the last mask. The sorted-sparse query is as follows:

$$Q_{\mathrm{s}\mathrm{o}\mathrm{r}\mathrm{t}\mathrm{e}\mathrm{d} \mathrm{s}\mathrm{p}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{e}}=Q\ast \mathrm{m}\mathrm{a}\mathrm{s}\mathrm{k}$$

(10)

2.3. Enhanced Rotary Positional Embedding Attention

Typically, using a self-attention mechanism, the position information of individual tokens can be leveraged through the use of transformer-based language modeling [46]. On the basis of this original mechanism, $q_m^T{k}_n$ typically allows for knowledge to be shared among tokens at various locations. To add the relative location information, we used a function g to formulate the inner product of query q_m and key k_n, which considered their relative position m − n and the word embeddings x_m, x_n as the input variables. We proposed the inner product to encode only the relative form of the location information as follows:

$$〈f_q\left(x_m,m\right),f_k(x_n,n)〉=g(x_m,x_n,m-n)$$

(11)

To solve the functions f_q (x_m; m) and f_k (x_n; n), we looked to align the previous relationship using a similar encoding mechanism.

We used a dimension d = 2 to define a simple case. Accordingly, we used a two-dimensional (2D) plane for the geometric property of the vectors as follows:

$$f_q\left(x_m,m\right)=\left(W_q{x}_m\right)e^{im\theta }$$

(12)

$$f_k\left(x_n,n\right)=\left(W_k{x}_n\right)e^{in\theta }$$

$$g\left(x_m,x_n,m-n\right)=Re[\left(W_q{x}_m\right)\left(W_k{x}_n\right)\ast e^{i(m-n)\theta }]$$

where Re[·] denotes the real part of a complex number; (W_k x_n)* denotes the conjugate complex number of (W_k x_n); and θ∈R denotes a preset nonzero constant. We then calculated f_{{q, k}} in a multiplication matrix as follows:

$$f_{\{q,k\}}\left(x_m,m\right)=\left(\begin{array}{cc}cos m\theta & -sin m\theta \\ sin m\theta & cos m\theta \end{array}\right)\left(\begin{array}{cc}{W}_{\left\{q,k\right\}}^{\left(11\right)}& W_{\left\{q,k\right\}}^{\left(12\right)}\\ W_{\left\{q,k\right\}}^{\left(21\right)}& W_{\left\{q,k\right\}}^{\left(22\right)}\end{array}\right)\left({\displaystyle \genfrac{}{}{0pt}{}{x_m^{\left(1\right)}}{x_m^{\left(2\right)}}}\right)$$

(13)

where ($x_m^{\left(1\right)},x_m^{\left(2\right)})$ is the expression of x_m in the 2D coordinates; g denotes the matrix and solves Equation (11) under the 2D case. Specifically, the relative position embedding was straightforward: we rotated the affine-transformed word-embedded vector according to the number of angle multiples of its location index and thus interpreted the RoPE intuition.

To simplify the 2D results to any $x_i\in R^d$, where d is even, we divided the d-dimension space into d/2 subspaces and combined them in the merit of the linearity of the inner product, transforming f_{{q, k}} as follows:

$$f_{\{q,k\}}\left(x_m,m\right)=R_{\Theta ,n}^l{W}_{\left\{q,k\right\}}{x}_m$$

(14)

where the rotary matrix with preset parameters $\Theta =\{{\theta }_i={\mathrm{10,000}}^{-{\displaystyle \frac{2\left(i-1\right)}{d}}},\mathrm{I}\in [1,2,\dots ,d/2]\}$ is as follows:

$$R_{\Theta ,n}^l\left\{\begin{array}{c}cos n{\theta }_1\\ \begin{array}{c}sin n{\theta }_1\\ 0\end{array}\\ \begin{array}{c}\begin{array}{c}0\\ \begin{array}{c}⋮\\ 0\end{array}\end{array}\\ 0\end{array}\end{array} \begin{array}{c}-sin n{\theta }_1\\ \begin{array}{c}cos n{\theta }_1\\ 0\end{array}\\ \begin{array}{c}\begin{array}{c}0\\ \begin{array}{c}⋮\\ 0\end{array}\end{array}\\ 0\end{array}\end{array} \begin{array}{c}0\\ \begin{array}{c}0\\ cos n{\theta }_2\end{array}\\ \begin{array}{c}\begin{array}{c}sin {\theta }_2\\ \begin{array}{c}⋮\\ 0\end{array}\end{array}\\ 0\end{array}\end{array} \begin{array}{c}0\\ \begin{array}{c}0\\ -sin n{\theta }_2\end{array}\\ \begin{array}{c}\begin{array}{c}cos n{\theta }_2\\ \begin{array}{c}⋮\\ 0\end{array}\end{array}\\ 0\end{array}\end{array} \begin{array}{c}\cdots \\ \begin{array}{c}\cdots \\ \cdots \end{array}\\ \begin{array}{c}\begin{array}{c}\cdots \\ \begin{array}{c}⋮\\ \cdots \end{array}\end{array}\\ \cdots \end{array}\end{array} \begin{array}{c}0\\ \begin{array}{c}0\\ 0\end{array}\\ \begin{array}{c}\begin{array}{c}0\\ \begin{array}{c}⋮\\ cos n{\theta }_{l/2}\end{array}\end{array}\\ sin n{\theta }_{l/2}\end{array}\end{array} \begin{array}{c}0\\ \begin{array}{c}0\\ 0\end{array}\\ \begin{array}{c}\begin{array}{c}0\\ \begin{array}{c}⋮\\ -sin n{\theta }_{l/2}\end{array}\end{array}\\ cos n{\theta }_{l/2}\end{array}\end{array}\right\}$$

(15)

The original attention can be represented as follows:

$$q_m^T{k}_n={\left(R_{\Theta ,m}^l{W}_q{x}_m\right)}^T\left(R_{\Theta ,n}^l{W}_k{x}_n\right)=x^T{W}_q{R}_{\Theta ,n-m}^l{W}_{k{x}_n}$$

(16)

where $R_{\Theta ,n-m}^l={{(R}_{\Theta ,m}^l)}^T{R}_{\Theta ,n}^l$.

For a computationally efficient realization of the multiplication of $R_{\Theta }^l$ and $x\in R^d$, we took advantage of the sparsity of $R_{\Theta ,n}^l$ in Equation (15) as follows:

$$R_{\Theta ,n}^l =\left(\begin{array}{c}{x}_1\\ x_2\\ \begin{array}{c}\begin{array}{c}{x}_3\\ x_4\end{array}\\ ⋮\\ \begin{array}{c}{x}_{l-1}\\ x_l\end{array}\end{array}\end{array}\right) \otimes \left(\begin{array}{c}cos n{\theta }_1\\ cos n{\theta }_1\\ \begin{array}{c}\begin{array}{c}cos n{\theta }_2\\ cos n{\theta }_2\end{array}\\ ⋮\\ \begin{array}{c}cos n{\theta }_{{\displaystyle \frac{l}{2}}}\\ cos n{\theta }_{{\displaystyle \frac{l}{2}}}\end{array}\end{array}\end{array}\right) + \left(\begin{array}{c}-x_2\\ x_1\\ \begin{array}{c}\begin{array}{c}{-x}_4\\ x_3\end{array}\\ ⋮\\ \begin{array}{c}{-x}_{l-1}\\ x_l\end{array}\end{array}\end{array}\right) \otimes \left(\begin{array}{c}sim n{\theta }_1\\ sim n{\theta }_1\\ \begin{array}{c}\begin{array}{c}sin n{\theta }_2\\ sin n{\theta }_2\end{array}\\ ⋮\\ \begin{array}{c}sin n{\theta }_{l/2}\\ sin n{\theta }_{l/2}\end{array}\end{array}\end{array}\right)$$

(17)

The RoPE principal formula derivation and the relationship with relative position embedding are given in Figure S1. According to the RoPE mechanism, we proposed an enhanced-RoPE mechanism in our model to improve the question-and-answer position information to increase accuracy. The enhanced-RoPE attention is shown in Figure 4.

As shown in Figure 4, we proposed an enhanced-RoPE mechanism. The specific operation applied the residual network. To solve the problem of gradient explosion and disappearance with an increase in the number of model layers, we developed the residual network. Image processing and other traditional neural networks often feature a lot of pooling and convolutional layers. Because each layer extracts features from the previous layer, degradation generally occurs as the number of layers increases and other problems emerge. To avoid these types of problems caused by deep neural networks, we adopted the jump connection method for the residual network, which we defined as follows:

$$y=F\left(x,\left\{W_i\right\}\right)+x$$

(18)

where $y$ denotes the output of the module, x denotes the input of the module, and $F\left(x,\left\{W_i\right\}\right)$ represents the learned residual mapping.

We multiplied x by a linear map W_s so that F and W_s x had the same dimension in Equation (18).

$$y=F\left(x,\left\{W_i\right\}\right)+W_{s}x$$

(19)

The identity mapping mitigated the degradation problem and could be simple, where Ws is used only to match the dimension of x. In this study, we used a one-layer residual network that was similar to a linear fully connected layer residual network. Algorithm 2 shows how it works.

Algorithm 2 Enhanced-RoPE Attention
Function	Enhanced-RoPE Attention (Xinput)
｜	Q, K ← Xinput
｜	QRoPE←$R_{\theta ,n}^l$ · Q, KRoPE←$R_{\theta ,n}^l$ · K
｜	Score = (QRoPE + Q) ∗ (KRoPE + K) T
｜	ScoreRoPE ←$R_{\theta ,n}^l$·Score
End

In Algorithm 2, we first applied something like a residual network to the query and key entered into the RoPE:

$$Q_{RoPE}=x{W}_q{R}_{\Theta ,m}^l$$

(20)

$$K_{RoPE}=x{W}_k{R}_{\Theta ,n}^l$$

We added operations like residual networks to $Q_{RoPE}$ and $K_{RoPE}$. Then, we calculated the score of attention:

$$Score=\left(F\left(x_q,\left\{W_i\right\}\right)+W_s{x}_q\right)\ast {\left(F\left(x_k,\left\{W_j\right\}\right)+W_s{x}_k\right)}^T$$

(21)

where $F\left(x_q,\left\{W_i\right\}\right),$ $x{W}_q{R}_{\Theta ,m}^l,$ and $x{W}_k{R}_{\Theta ,n}^l$, respectively.

We operated RoPE operation again on the score. The Score_RoPE was proposed to be combined with the Value as follows:

$${Score}_{RoPE}=x_{score}{W}_{score}{R}_{\Theta ,n-m}^l$$

(22)

2.4. Ocean X GPT Question-Answering System with Embodied Intelligence

Decoder: A stack of identical layers (N = 3) was included in the decoder. The decoder inserted a third sublayer as well as two sublayers in each encoder layer. This enabled the decoder to perform multi-head attention over the encoder stack’s output. Around each sublayer, we employed residual connections followed by layer normalization, similar to the encoder. For the structural part of the model, see the legend in the Supplementary Materials.

Algorithm 3 shows the Ocean X GPT question-and-answer system with embodied intelligence.

Algorithm 3 Ocean X GPT Question-and-Answer System with Embodied Intelligence
	Question ← “Input:”
	While True:
	temp_sentence == input (“…”)
	Question ← temp_sentence
	If Question == “The environment outside”
	|   b ← model. Predict (XRandom_inside). Argmax (−1)
	|   If b == 0:
	|     | a == “answer 0”
	|   elif b == 1:
	|    |               a == “answer 1”
	|   elif b == 2:
	|     |              a == “answer 2”
	|   elif b == 3:
	|      |   a == “answer 3”
	else a ← GPT.answer (Question)
	Return a

Algorithm 3 integrates the concept of embodied intelligence into the system. Based on the data collected by the external sensor, it broadcasts the working status and environment of the sensor in real time. This function’s health management concept made strong engineering sense.

The proposed health management system was based on a question-and-answer dialog model. First, we entered the questions into the model in text form. When the content of questions contained the “outside environment”, the model would jump into the ESS transformer model and return the output result based on the external data. Because the input result in this part was random, we created dynamic data, which could be combined well with various situations under real working conditions. The sensor arrays would know the corresponding gas concentration and remaining working time according to external data by the pre-training model. These judgments enabled the sensor arrays to make corresponding decisions and create embodied intelligence. When the question did not contain the keywords (e.g., the “outside environment”), the model would output the question in the form of a voice broadcast according to the trained offline question-and-answer corpus, which helped the questioner learn the knowledge of the deep-sea CH₄ detection.

2.5. Encoder and Decoder Stacks

Encoder: A stack of identical layers (N = 4) was included in the encoder for the toxic gas concentration detection task shown in Figure S2a. All layers had two sublayers. The first layer featured a multi-head sorted-sparse attention mechanism. The second layer featured a simple, position-wise fully connected feed-forward network. We employed a residual connection to surround each sublayer after layer normalization. Each sublayer output was Layer–Norm (x + Sublayer(x)), where Sublayer(x) was implemented by the sublayer. All model sublayers, as well as the embedded layers, produced outputs of dimension d_model = 16 to facilitate these residual connections.

Decoder: A stack of identical layers (N = 6) was included in the decoder for the Ocean X GPT model shown in Figure S2b. In each encoder layer, the decoder had a third sublayer in addition to the two sublayers. This encoder applied multi-head attention over the encoder stack’s output. We used residual connections around the sublayers like the encoder after layer normalization. Table S1 lists parameters used in the interval detection task for toxic gas concentration. We considered the complexity of the toxic gas data and decreased the parameter head to three to ensure that sufficient attention was provided to determine the relationship among data in the training process. We used the classic transformer model without changes in the middle layer. To avoid overfitting during training, we used the dropout operation in Layers 3 and 4, which caused the test set to perform poorly. Table S2 lists the parameters used in the Ocean X GPT model. We set the number of layers to six and the parameter head to eight.

3. Experiment, Results and Discussion

3.1. Setup of Experiment

The experimental system provided all the data in this study. Figure 5a illustrates the CH₄ sensor array diagram. This experimental system featured the following six parts: (1) the air chamber and gas circuit parts, (2) the communication module, (3) the host computer, (4) the air path, (5) the air chamber, and (6) the environmental stress simulator. The vibration stress amplitude ranges from ±1.5 mm and the frequency ranges from 0 to 120 Hz. The angle range of the sway stress was ±15°. The temperature increase ranged from 0 to 10 °C.

After testing and training, the sensor array studied in this paper will be installed on the underwater robot, and the underwater methane detection will be carried by the robot. In practical engineering, the robot is driven by a propeller, and the propeller is controlled by a motor. When the motor works, it will produce vibration stress (noise) in a certain frequency range. We use the vibration motor to load it on the gas chamber to simulate the vibration stress generated by the robot working, with amplitude ±1.5 mm and frequency 0–120 Hz. When diving, the impact of ocean current will also cause the underwater robot to sway. We use the sway motor to pull the gas chamber sway to simulate the robot sway, and the sway Angle is ±15°. The temperature of sea water will also change. A heating wire is installed in the gas chamber as a heater to make the temperature change in the gas chamber to simulate the change in an underwater temperature environment. The change in the temperature environment is 0–10 °C.

Loading environmental factors, such as vibration, sway, and heating, into the test enables the examination of whether the sensor array can work normally and fail under complex working conditions. Due to the applied vibration and sway stress, although it will cause periodic interference to the sensor array, the interference signal is superimposed on the sensor output and has little impact on the detection accuracy, so it is not discussed in detail. In addition, the heating factor is loaded into the test, which is also to examine whether the sensor array can work normally under variable temperature conditions. Although the applied temperature stress will also cause interference to the sensor array, the normal working temperature of the sensor is about 450 °C, and small temperature changes will not have a great impact on the detection accuracy, so this paper does not focus on the analysis.

The sensing transmitter is composed of a sensing resistor R_S, a fixed sampling resistor R₀, an adjustable potentiometer R_L, a programmed potentiometer R_W, a heating voltage V_h, and a working voltage V_in. Two-way communication and data transmission enabled communication with the host computer. The primary platform of the test system was the host computer. It sent down control commands and received temperature, air pressure, humidity, sway, vibration, and gas concentration detection information from the communication module. It also collected output signals from the sensor arrays and extracted characteristic information. Figure 5b shows the sensor signal pickup circuit structure. Figure 5c shows the MQx6 and MQ4 gas sensor cylinder core structure. Pattern recognition is realized by using a 2 × 3 array composed of two types of sensors: MQx6 and MQ4. We used Windows 10 on a 2.8 GHz Intel CPU with 16 GB RAM.

In this research work, the gas static test method is used, and the experimental system in Figure 5a is used for gas concentration test. The specific method is to use the standard air bag to take high-purity (99.99%) H₂S Gas and high-purity (99.99%) CH₄ gas, and then use the standard syringe to take the measured gas from the air bag and inject it into the air chamber of the experimental system so that the standard concentration of H₂S gas and standard concentration of CH₄ gas can be prepared. Then, the experiment of the methane sensor array placed in the air chamber is realized. The designed volume of the air chamber used in our test is 13.6 L. Under normal circumstances, the air chamber is filled with air. Since the consumption of O₂ is very small in the test process, it can reach a degree of neglect, so the influence of O₂ is not deeply discussed in this paper.The yellow, green, and blue curves are MQx6 sensors; purple, red, and brown are MQ4 class sensors. A 2 × 3 sensor array is formed. Figure S3 shows the relationship of toxic gas concentration interval detection.

In Figure 6a, the hydrogen sulfide concentrations are 8 ppm, 6 ppm, and 4 ppm. In Figure 6b, the hydrogen sulfide concentrations are 14 ppm, 16 ppm, and 18 ppm. In Figure 6c, the hydrogen sulfide concentrations are 24 ppm and 25 ppm. In Figure 6d, the hydrogen sulfide concentrations are 35 ppm, 38 ppm, and 34 ppm. The methane background concentration is 10 ppm (8 ppm). The sensor array consists of three MQ4 sensors and three MQx6 sensors. For methane gas, the sensitivity of the MQ4 sensor is higher than that of the MQx6 sensor. For H2S gas, the sensitivity of the MQx6 sensor is higher than that of the MQ4 sensor.

3.2. Flowchart of Question-and-Answer Health Management Systems with Embodied AI

To develop the Ocean X GPT model, we created an AI model based on multi-sensor data under changing operating conditions to support a question-and-answer health management system. Figure 7 shows the experimental method’s flowchart.

The health management process featured the following three steps:

Step 1: For the input, the model used the voice signal from the user and converted it to a text signal, which was entered into the trained model to be processed. Some of these previously designed questions contained the external environment information, and the corresponding answers to these questions had an impact on the working instrument. Based on the sensor array information that was returned, we observed the current external environment information. According to the trained ESS transformer model, we calculated the health condition and remaining working time of the machine, which enabled the corresponding engineering decisions. This concept of making hardware decisions through software algorithms is called embodied AI.

Step 2: In the Ocean X GPT health management section, the model translated text into machine language by word embedding. The model iterated the corresponding answer through the information from the knowledge base. When the problem included the keyword “external environment, etc.”, it jumped to the ESS transformer model to return the value y. The operation of connecting the ESS transformer model and Ocean X GPT model enabled the health management system to be an online monitoring system. Each x entered the model randomly, which simulated the random situation in a real working condition and returned the corresponding y value, which corresponded to the answer for that health management level.

Step 3: In the answer part, the model converted the answer into text by NLG, and then converted it into speech through speech synthesis to complete the broadcast. We used python’s speech-to-text conversion plug-in to implement the dialog system. The machine also provided feedback to the answer for the user through a voice broadcast, which replicated human–computer interaction.

3.3. Validation of Anomaly Detection Method and Inference

3.3.1. Toxic Gas Concentration Interval Detection Evaluation Metrics

We evaluated metrics of the toxic gas concentration interval detection methods. We used precision rate in Equation (23), recall rate in Equation (24), F measure in Equation (25), and accuracy rate (Acc) in Equation (26):

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(23)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(24)

$$F1score={\displaystyle \frac{\left(1+x^2\right)\times Precision\times Recall}{{\alpha }^2\times Precision\times Recall}}$$

(25)

$$Acc={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(26)

where FP and FN are the false positives and false negatives, respectively. TP and TN are the true positives and true negatives, respectively, and $\alpha$ = 1.

For indicators of model performance, we adopted the mean squared error (MSE), mean absolute error (MAE), and root mean squared error (RMSE) of the predicted and true targets. For MAE, MSE, and RMSE, the corresponding calculations are given in the following equations:

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|y_{\_test}-y_{pre}\right|$$

(27)

$$MSE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(y_{\_test}-y\_pre\right)}^2$$

(28)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(y_{\_test}-y\_pre\right)}^2}$$

(29)

where $y$_pre denotes the predicted tool wear value in the dataset, $y$_test represents the true tool wear value in the dataset, and n is the number of test samples.

3.3.2. ESS Transformer Toxic Gas Concentration Interval Detection Results and Discussion

We trained 50 iterations in the ESS transformer model. We set the batch size to two in the training process. We set the dynamic learning rate to 0.001. As the number of training epochs rose, the learning rates dropped every 10 epochs by 0.0005, which enabled the model to find the global optimum more quickly. The training and validation loss of the toxic gas concentration interval detection model is shown in Figure S4.

The confusion matrix of the toxic gas concentration interval detection task result is shown in Figure 8. We used a 10-fold validation method to test the results and set the number of verifications to 760. As shown in Figure 8, the accuracy rate was close to 100%. Mode 1 corresponded to the 0–10 ppm toxic gas interval, and Mode 4 corresponded to the 30–40 ppm toxic gas.

To compare toxic gas concentration interval detection accuracy and assess the method’s performance, we used two related methods. The comparative methods were convolutional neural networks (CNNs) + support vector machine (SVM) transformer encoder and ESS transformer.

Table 1. Toxic gas concentration detection metrics for the three methods.

Model	Training Time (s)	Accuracy (%)	Recall (%)	Testing Time (s)
CNN + SVM	480	97.5%	97%	0.1
Transformerencoder	600	98.3%	98%	0.2
ESS transformer model	520	99.9%	99%	0.2

lists the results for each of these methods. The CNN + SVM method had the lowest training time (480 s). Because of the transformer’s high parallel computing ability, the transformer encoder model had the highest training time (600 s). The proposed model had a higher recall score of 99% and a higher accuracy of 99.9%.

3.3.3. Health Levels and Lifetime Prediction Results and Discussion

As shown in Figure 9, the establishment of system health is closely related to prior knowledge of the sensor’s life. Therefore, we connected the four classification results for toxic gas concentration detection to the remaining lifetime of the CH₄ sensor with toxic gas and broadcast the speech through the Ocean X GPT model.

The mechanism of H₂S poisoning of CH₄ sensors is a complex chemical process. It is considered to be a chemical reaction between sensitive materials and sulfur and is specifically manifested as catalyst sulfur poisoning and sensitive material sulfur poisoning [10]. This poisoning affects the three most important indicators of activity, selectivity, and life of the CH₄ sensors [11]. The phenomenon of CH₄ sensor poisoning caused by H₂S can be understood as the adsorption of poison at the surface’s active center and further conversion to more stable surface compounds. This action leads to the active site being passivated or permanently occupied. On the one hand, H₂S gas makes the active component of the precious metal catalyst become inactive metal sulfide or sulfate. On the other hand, the sulfurization or sulfation of SnO₂-sensitive materials makes it lose its role of supporting the active component and regulating the microstructure of the active component. The degree of CH₄ sensor poisoning caused by H₂S gas can be expressed by a grading system. The assessment is based on the remaining time that the sensor continues to work after the poisoning has occurred. According to the requirement of the one-time working time of the sensor for CH₄ detection, we classified the H₂S poisoning phenomenon into four grades. Figure 10 shows the H₂S poisoning level of the CH₄ sensor.

According to the general technical standards of gas sensors and specific engineering application requirements, we selected deviation from the CH₄ detection value ±10% as the failure criterion of the CH₄ sensor. After a qualified aging screening, we extracted and divided 10 MQ4 sensor samples (MQ4 and MQx6) into five groups for testing. We similarly assumed that the performance of the ten sensor arrays used for life calibration experiments to that of six sensor arrays in the toxic gas concentration internal detection experiment in real working conditions. We conducted the test under normal atmospheric conditions. To simulate real working conditions, we loaded environmental factors, such as vibration, rocking, and heating, into the test. We tested the failure time of H₂S poisoning of the sensor in a fixed concentration interval according to relevant regulations. Figure 10 shows the relationship between H₂S concentration and the failure time of the CH₄ sensor. This figure can be used as a health management system reference to classify toxic gas concentration intervals.

As shown in Figure 10, the IV Level was the lowest level, in which case the detection would not be affected. The concentration of toxic gas (H₂S) was 0–10 ppm. The III Level was the critical warning, which could cause the sensor detection accuracy to move to the edge of the error band and would not be able to maintain sensor detection accuracy within a specified time. The concentration of toxic gas (H₂S) was 10–20 ppm. The II Level was higher, which caused the CH₄ sensor to fail to complete the specified task within the specified time, but it could continue to work after recovery. The concentration of toxic gas (H₂S) was 20–30 ppm. The I Level was the highest level, which caused fatal damage to the sensor, failed to complete the specified task within the specified time, and whose performance could not be restored. The concentration of toxic gas (H₂S) was 30–40 ppm.

Therefore, according to Figure 9 and Figure 10, we obtained the following information about the health of the system:

The concentration of toxic gas internal detection at Mode 0 was 0–10 ppm, the health level was IV Level, and the remaining working time was about over 16 h.

The concentration of toxic gas internal detection at Mode 1 was 10–20 ppm, the health level was III Level, and the remaining working time was about 10–16 h.

The concentration of toxic gas internal detection at Mode 2 was 20–30 ppm, the health level was II Level, and the remaining working time was about 6–10 h.

The concentration of toxic gas internal detection at Mode 3 was 30–40 ppm, the health level was I Level, and the remaining working time was about 3–6 h.

3.3.4. Offline Question-and-Answer System Experimental Results and Discussion

We evaluated questions and answers from the CH₄ sensor Wikipedia and model basics. These contents were the key factor affecting the working state of CH₄ sensors. We trained the offline question-and-answer task for the health system for 500 iterations. Throughout this process, we set the batch size to two, and set the dynamic learning rate to 0.0001. As the number of training epochs rose, the learning rate dropped every 10 epochs by 0.0005. Therefore, the model was able to find the global optimum more quickly. Figure S5 shows the training and validation loss of the toxic gas concentration interval detection model.

Table 2. Offline question-and-answer accuracy by Ocean X GPT.

Question	Generated Answer Token	Correct	Accuracy Rate (%)
What are the application areas of CH4 sensors?	37	✓	99.7%
What are the H2S poisoning phenomena of sensors?	40	✓	99.4%
What are the hazards of CH4 gas?	39	✓	99.9%
What is the transformer algorithm?	46	✓	99.5%
What are the implications of detecting ocean CH4?	37	✓	99.2%
What are the components of an array sensor?	40	✓	99.6%
What are the components of the signal collector? What are the gas identification methods?	41 56	✓ ✓	99.3% 98.9%
What is the mechanism of sensor poisoning caused by H2S?	52	✓	98.7%
What is the degree of poisoning of the CH4 sensor arrays caused by H2S gas?	84	✓	98.9%
What is the level of H2S?	52	✓	99.1%
What is the significance of predicting failure?	15	✓	99.9%

provides the accuracy of the offline question-and-answer system by Ocean X GPT. According to the results given in Table 2, the 12 questions were essentially correct, as they pertained to deep-sea CH₄ detection (average accuracy was >99%). The question “What is the significance of predicting failure?” had the highest accuracy (99.9%), and the question “What is the mechanism of the sensor poisoning caused by H₂S?” had the lowest accuracy (98.7%). These problems basically include all the knowledge points of sensor construction and abnormal fault detection. These questions can make the questioner learn the basic problems of engineering very well, which is of great help to the emergencies that occur in the actual working environment.

Figure S6 provides a visualization of the offline question-and-answer task by the Ocean X GPT model. It shows in detail the original corpus of the model, the trained questions, and the results of the trained answers. It can be seen that the model roughly correctly learns the key information in the original corpus and can accurately answer the questions without grammatical errors. This part combines the abnormal gas detection and the dialog system well, making the model have the concept of embodied intelligence. The model can simulate the real human feedback to the questioner through the external environment.

To compare the accuracy of the offline question-and-answer task and assess model performance, we selected two other models (

Table 3. Accuracy of offline question-and-answer tasks using three models.

Model	Mean Accuracy	<40 Tokens	40–50 Tokens	>50 Tokens	Prompt or Not	I Do Not Know Assignment
LSTM-ecoder	96.6%	97.1%	96.5%	96.2%	✓	No
GPTDecoder	98.1%	98.3%	98.1%	97.9%	✓	No
Ocean X GPT	99.4%	99.7%	99.5%	98.9%	✓	No

). The comparative models were LSTM_Decoder and GPT Decoder (GPT Decoder). Ocean X GPT had the highest mean accuracy (99.4%), and LSTM_Decoder had the lowest accuracy. LSTM was not suitable as a model for discrete tasks. According to the number of tokens, the accuracy rate decreased by increasing the number of tokens, which was easy to understand. When the number of answering words increased, the required computing power increased, and it became more difficult to achieve the same accuracy rate.

3.4. Attention Visualization for Anomaly Detection in the Training Process

The training process included a thermodynamic diagram for the attention mechanism to detect the toxic gas concentration interval as well as an offline question-and-answer task (Figure 11). Figure 11a–d show the toxic gas concentration interval detection task, illustrating a thermodynamic diagram for the attention mechanism after training 50 epochs (MSE 0.07, MAE 0.06), 20 epochs (MSE 0.12, MAE 0.11), 10 epochs (MSE 0.38, MAE 0.32), and 5 epochs (MSE 0.68, MAE 0.41).

The Ocean X GPT task provided a thermodynamic diagram for the attention mechanism after training 500 epochs (CrossEntropy 0.02), 300 epochs (CrossEntropy 0.16), 100 epochs (CrossEntropy 0.84), and 50 epochs (CrossEntropy 1.39).

3.5. Comparison of Model Memory Cost

Table 4. ESS transformer model summary.

Model/Paper	Complexity	Decode	Class
Trans.-XL (Dai et al., 2019) [47]	O(n2)	$\surd$	RC
Sparse Trans. (Child et al., 2019) [48]	$\mathrm{O}(\mathrm{n}\sqrt{n}$)	$\surd$	FP
Reformer (Kitaev et al., 2020) [49]	O(nlogn)	$\surd$	LP
ESS transformer model	O(nlogn)	$X$	FP

Ps: Class abbreviations include FP = Fixed Patterns or Combinations of Fixed Patterns, LP = Learnable Pattern, RC = Recurrence.

compares similar efforts to limit time complexity. We employed three models for comparison, including Dai et al. (2019) [47], Child et al. (2019) [48], and Kitaev et al. (2020) [49]. The ESS transformer model limited the time complexity to O (nlog_n) by using fixed patterns or combinations of fixed patterns. Model performance was not lost, and the effect was elegant.

4. Conclusions

In this study, we proposed two models for industrial conditions. We employed the ESS transformer model, which was used for toxic gas concentration interval detection under marine conditions. Furthermore, we employed the Ocean X GPT model, which embedded the ESS transformer model for intelligent health management and lifetime prediction of the CH₄ sensor arrays. The ESS transformer model enhanced the performance of concentration interval detection under marine conditions (accuracy = 0.99). These results were superior to other similar models, including CNN + SVM and transformer encoder. The proposed model had a time complexity of only O (n log_n) compared with the time complexity of the original model, which was O (n²). The ESS transformer model offered a beneficial solution for concentration interval detection under marine conditions. It offered high accuracy, high speed, and low computational complexity. The Ocean X GPT model for offline question-and-answer tasks had a high mean accuracy (99.4%), which was superior to other related models, such as LSTM–AE and GPT decoder. It also featured an elegant concept using a question-and-answer system framework.