Modeling and Evaluation of Attention Mechanism Neural Network Based on Industrial Time Series Data

Abstract

Chemical process control systems are complex, and modeling the controlled object is the first task in automatic control and optimal design. Most chemical process modeling experiments require test signals to be applied to the process, which may lead to production interruptions or cause safety accidents. Therefore, this paper proposes an improved transformer model based on a self-attention mechanism for modeling industrial processes. Then, an evaluation mechanism based on root mean square error (RMSE) and Kullback–Leibler divergence (KLD) metrics is designed to obtain more appropriate model parameters. The Variational Auto-Encoder (VAE) network is used to compute the associated KLD. Finally, a real nonlinear dynamic process in the petrochemical industry is modeled and evaluated using the proposed methodology to predict the time series data of the process. This study demonstrates the validity of the proposed transformer model and illustrates the versatility of using an integrated modeling, evaluation, and prediction scheme for nonlinear dynamic processes in process industries. The scheme is of great importance for the field of industrial soft measurements as well as for deep learning-based time series prediction. In addition, the issue of a suitable time domain for the prediction is discussed.

1. Introduction

Models that describe the dynamic characteristics of processes are an essential component of advanced control systems in the refining industry [1]. The quality of the model is a determining factor in the efficacy of the control process [2]. Due to changes in operating conditions, random disturbances caused by environmental influences, complex and variable chemical reaction processes, and time delays caused by measurement processes, industrial processes exhibit various complex characteristics such as multivariable, nonlinearity, slow time-varying, and strong coupling [3,4]. However, identification experiments typically necessitate the cessation of industrial production and their execution under open-loop conditions [5]. At the same time, the process is manually fed with an excitation signal for a long enough period of time to excite the full dynamic characteristics of the system and obtain the data needed for identification [6]. These problems seriously affect industrial production and are difficult to put into use for a long period of time, with limitations for complex and variable operating conditions. Therefore, it is essential to improve the reliability and accuracy of modeling key process variables in industrial production processes.

The emergence and development of deep learning neural networks and the abundance of process data available in industrial production through distributed control systems (DCS) provide an effective method for solving complex modeling problems [7,8,9]. Some specular designed neural networks are adopted for modeling dynamic characteristics [10,11,12]. The advantages of deep networks in modeling complex industrial processes lie in the following: deep networks can capture complex nonlinear relationships within industrial processes, allowing for more accurate modeling compared to linear methods; by changing the number of neurons in the input and output layers, the deep network can be applied to different multivariate systems; and the deep networks have strong feature extraction capabilities and are less demanding in terms of data identifiability than traditional recognition methods.

Since its proposal in 2017, Transformer [13] has attracted widespread attention in the field of natural language processing [14] due to its unique architecture and excellent performance. Compared with basic deep learning models such as recurrent neural networks (RNN) [15] and convolutional neural networks (CNN) [16], Transformer adopts a new encoder–decoder architecture based on self-attention-based encoder–decoder architecture, which has the advantages of strong parallel computation capability, strong data feature-dependent capture capability, strong global information capture capability, high scalability and flexibility, and greatly improves the ability to extract features from long sequential data. Based on the above advantages, the task scenarios of Transformer are gradually expanding from the field of natural language processing to other fields, including image processing [17], speech recognition [18], and time series analysis [19]. Therefore, Transformer’s powerful sequence processing and feature extraction capabilities can meet the needs of industrial modeling.

However, in the field of industrial modeling, the existing Transformer [20,21,22] does not consider the autocorrelation property of the output data when designing the input and output data, and only considers the cross-correlation property between the temporal input and output data. By introducing the auto-correlation property of output data [23,24], the dimensionality of network input data can be reduced to a certain extent, i.e., less historical data are used for modeling, and the efficiency of network training can be improved to a certain extent. Therefore, in this paper, a modified transformer prediction model is proposed to achieve full dynamic modeling of nonlinear dynamic processes in the petrochemical industry. To be specific, the input–output (I/O) design should contain different time series of different variables, at the same time the design can simplify the decoder structure and thus the Transformer model structure.

Moreover, when neural network structures are determined, the relevant values of structure parameters are important for improving the modeling accuracy. Usually, classical evaluation indexes RMSE [25] and mean square error (MSE) [26] are used to observe the overall situations of predictive errors. However, the distributions of residuals are not analyzed sufficiently. In order to observe residual distributions, KLD [27,28,29] is used for computing the distances between different distributions [30,31]. An autoencoder [32,33], an unsupervised learning algorithm, compresses input data into a low-dimensional representation through an encoder, reconstructs the data through a decoder, and learns an effective data representation method. In terms of the above situations, the VAE [34] network is adopted for both reconstructing the residuals and computing the KLD values [35,36]. Hence, in this paper, a new evaluation mechanism based on both RMSE and KLD is exploited. Through the proposed evaluation mechanism, the optimal structure parameters of the proposed original modified transformer are determined, in order to harvest more accurate predictive performances.

To sum up, the main contributions of this paper are as follows:

(1)

Firstly, the traditional Transformer model is simplified and improved by ingenious I/O design, and a modified transformer model is established. The proposed model is applied to full dynamic modeling and prediction of nonlinear dynamic processes in the petrochemical industry for the first time. It can adaptively capture time series dependencies between input and output sequences.

(2)

Secondly, the evaluation mechanism is proposed. The mechanism uses RMSE for preliminary judgment at first and then uses KLD to further excavate residual characteristics. Finally, a group of optimal network structure parameters can be selected from the set of structure parameters. The network training and prediction under optimal structure parameters can ensure that the predictive effect of the model is relatively best.

(3)

Then, this paper combines the proposed model and evaluation mechanism to model, predict, and evaluate a real nonlinear dynamic process, and the effectiveness of our model is proved. Experimental results show that our model has better predictive performance than baseline methods.

(4)

Finally, in order to select the predictive time domain, a hybrid network is designed and consists of multiple subnets. Each subnet has different predictive output sequences. By observing the predictive effects of different subnets on the same output sequence, the suitable predictive time domain can be determined.

The rest of this paper is organized as follows. Section 2 describes the overall architecture of the modified transformer model, explains the I/O design of the model, interprets how to determine a predictive time domain, and describes relevant theory and process details. A training algorithm and evaluation mechanism of the proposed model are introduced, and the operation principle of the evaluation mechanism is analyzed in detail in Section 3. In Section 4, the proposed model and two baseline models are used to conduct experiments on a real nonlinear dynamic process. Finally, this paper gives conclusions of this study in Section 5.

2. Modified Transformer Model Incorporating Industrial Timing I/O Design

This paper proposes a Transformer prediction model that incorporates industrial timing I/O design for use in nonlinear dynamic process prediction scenarios. The model’s structure is illustrated in Figure 1. As illustrated in Figure 1, the Transformer model comprises four principal components: the input layer, position encoder, encoder and decoder, where $\mathit{x}$ and $\widehat{\mathit{y}}$ are used to represent input variables and model output variables, respectively, $n$ and $q$ are used to represent input order and predictive time domain, respectively. The primary distinctions between this model and the classical Transformer model are evident in the input layer and decoder, whereas the position encoder and encoder remain consistent with the classical Transformer model. The input layer streamlines the traditional Transformer model through the strategic integration of input/output (I/O) design.

2.1. Incorporating Industrial Timing I/O Design

The Transformer prediction model used in this paper, which incorporates industrial timing I/O design, describes the dynamic characteristics using the correlation between different moments of the input and output variables. Figure 2 illustrates the I/O design of the Transformer prediction model. For the input module, the dynamic order of the actual process determines the number of columns in the input matrix, while the number of state variables determines the number of rows in the input matrix. As for the output module, the internal dynamic characteristics of the actual system determine the prediction horizon, i.e., the number of columns in the output matrix, and similarly, the number of state variables determines the number of rows in the output matrix. The forward time steps $n$ and prediction horizon $q$ in Figure 2 are both related to the dynamic order of the actual system.

More specifically, in this paper, the prediction of data at one time in the future is called ‘single-step prediction’, and the prediction of data at multiple times in the future is called ‘multi-step prediction’. In the following, this paper gives a general definition of the prediction task for nonlinear dynamic processes. Its goal is to predict the value of the output sequence in the next period of time based on historical time series. As shown in Figure 1, for single-input and single-output nonlinear dynamic process (hereinafter called univariate nonlinear dynamic process), the input of the model at time $t$ is defined as a data information matrix containing the data at previous $n$ times, namely, ${\mathit{X}}_t=[{\mathit{x}}_{t-n},{\mathit{x}}_{t-n+1},\cdots ,{\mathit{x}}_t]$, where ${\mathit{x}}_t={[u_t,{\mathit{y}}_t]}^T$. This design means that the input sequence and the output sequence of processes are fused together as the input of the model. Hence, model inputs become more complex, contain more abundant dynamic characteristics, and make their generalization ability stronger. Single-step prediction is usually of little help to real early warning application, because it is difficult to predict what will happen after multiple steps. Hence, this paper defines the data information vector containing the next $q-1$ moments in the future as predictive output of the model at the $t-\mathrm{th}$ moment, namely, ${\widehat{\mathit{Y}}}_t=[{\widehat{\mathit{y}}}_{t+1},\cdots ,{\widehat{\mathit{y}}}_{t+q-1}]$. Equation (1) simply and clearly presents the prediction task of univariate nonlinear dynamic processes. For multiple-input and multiple-output nonlinear dynamic processes (hereinafter called multivariate nonlinear dynamic process), the input representation of the model at time t is the same as that of univariance. However, ${\mathit{X}}_t$ is expanded to ${\mathit{x}}_t={[u_{1,t},\cdots ,u_{r,t},{\mathit{y}}_{1,t},\cdots {\mathit{y}}_{l,t}]}^T$, where $r$ is the number of process input variables, and $l$ is the number of process output variables. The predictive output of the model at the $t-\mathrm{th}$ moment is extended to ${\widehat{\mathit{Y}}}_t=[{\widehat{\mathit{y}}}_{t+1},\cdots ,{\widehat{\mathit{y}}}_{t+q-1}]$, where ${\widehat{\mathit{y}}}_t=[{\widehat{\mathit{y}}}_{1,t},\cdots ,{\widehat{\mathit{y}}}_{l,t}]$. Equation (2) simply and clearly presents the prediction task of multivariate nonlinear dynamic processes.

$$\left[\mathit{x}=\left(\begin{array}{ccc}{u}_{t-n}& \cdots & u_t\\ y_{t-n}& \cdots & y_t\end{array}\right)\right]\underrightarrow{ \mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}\left(\begin{array}{ccc}{\widehat{y}}_{t+1}& \cdots & {\widehat{y}}_{t+q}\end{array}\right),$$

(1)

$$\left[\mathit{x}=\left(\begin{array}{ccc}{u}_{1,t-n}& \cdots & u_{1,t}\\ ⋮& \ddots & ⋮\\ u_{r,t-n}& \cdots & u_{r,t}\\ y_{1,t-n}& \cdots & y_{1,t}\\ ⋮& \ddots & ⋮\\ y_{l,t-n}& \cdots & y_{l,t}\end{array}\right)\right]\underrightarrow{ \mathrm{model} }\left(\begin{array}{ccc}{\widehat{y}}_{1,t+1}& \cdots & {\widehat{y}}_{1,t+q}\\ ⋮& \ddots & ⋮\\ {\widehat{y}}_{l,t+1}& \cdots & {\widehat{y}}_{l,t+q}\end{array}\right).$$

(2)

2.2. Modified Transformer Model Structure

2.2.1. Positional Encoding

The structural setup of the attention mechanism in the Transformer results in the inability of the model to learn the positional information in the input sequences, thus the input layer needs to encode the positions. For nonlinear dynamic processes in industrial processes, the position information in the sequence represents the moment information, and this information plays an extremely important role in the prediction effect of the industrial process. As shown in Equations (3) and (4).

$$P_{loc}(loc,2i)=\sin ({\displaystyle \frac{loc}{{10000}^{2i/d_{model}}}}),$$

(3)

$$P_{loc}(loc,2i+1)=\cos ({\displaystyle \frac{loc}{{10000}^{2i/d_{model}}}}),$$

(4)

where $loc$ is the position of the input sequence, $d_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathbf{l}}$ represents the dimension of the input sequence, $2i$ and $2i+1$ represent parity, and $i$ denotes the $ith$ dimension of the input sequence feature vector.

2.2.2. Multi-Head Self-Attention Mechanism

The core principle of the encoder is the self-attention mechanism, and its computational process is commonly used in the Query-Key-Value (QKV) mode, as shown in Equation (5).

$$SelfAttention(\mathit{Q},\mathit{K},\mathit{V})=SoftMa\mathit{x}({\displaystyle \frac{\mathit{Q}{\mathit{K}}^T}{\sqrt{d_k}}})\cdot \mathit{V},$$

(5)

where $\mathit{Q}$ denotes the query feature matrix, $\mathit{K}$ denotes the key feature matrix, and $\mathit{V}$ denotes the value feature matrix, which are obtained by multiplying the input feature matrix with the corresponding weight matrix, respectively; and $d_k$ represents the dimensions of $\mathit{Q}$, $\mathit{K}$, and $\mathit{V}$, and the SoftMax function is a commonly used activation function that converts any real vector into a probability distribution.

Based on the self-attention mechanism, the Transformer model uses a multi-head self-attention mechanism. As shown in Equations (6) and (7).

$$MultiHead(\mathit{Q},\mathit{K},\mathit{V})=Concat(hea{d}_1,\cdots ,hea{d}_{\mathit{m}})\cdot {\mathit{W}}^O,$$

(6)

$$hea{d}_i=SelfAttention(\mathit{Q}{\mathit{W}}_i^Q,\mathit{K}{\mathit{W}}_i^K,\mathit{V}{\mathit{W}}_i^V),$$

(7)

where ${\mathit{W}}_i^Q$, ${\mathit{W}}_i^K$, and ${\mathit{W}}_i^V$ are weight matrices of $\mathit{Q}$, $\mathit{K}$, and $\mathit{V}$, $i\in \left[1,\mathit{m}\right]$, Concat function stitches together matrices generated by multiple attention headers, ${\mathit{W}}^O$ denotes the output parameter matrix, and $\mathit{m}$ is the total number of attention heads.

2.2.3. Residual Linking and Normalization

Residual connection can solve the problem of network degradation by overlaying input and output. Normalization can not only eliminate the effects of gradient disappearance and gradient explosion but also accelerate convergence. The specific calculation equations are shown in Equations (8)–(11)

$${\mathit{X}}_{Mnorm,o}=LayerNorm({\mathit{x}}_{add})=LayerNorm(\mathit{X}+MultiHead(\mathit{Q},\mathit{K},\mathit{V})),$$

(8)

$$LayerNorm({\mathit{x}}_{add})={\displaystyle \frac{x_{add,az}-{\mu }_a}{\sqrt{{({\sigma }_a)}^2+\epsilon }}},$$

(9)

$${\mu }_a={\displaystyle \frac{1}{d_t}}{\displaystyle \sum _{z=1}^{d_t}{x}_{add,az}},$$

(10)

$${({\sigma }_a)}^2={\displaystyle \frac{1}{d_t}}{\displaystyle \sum _{z=1}^{d_t}{(x_{add,az}-{\mu }_a)}^2},$$

(11)

where ${\mathit{X}}_{Mnorm,o}^g$ is the output of residual connection and normalization layer behind the multi-head self-attention layer, LayerNorm is the normalization function, ${\mathit{x}}_{add,az}$ is the value of row $a$ and column $z$ of output matrix ${\mathit{x}}_{add}$ after residual connection, ${\mathit{\mu }}_a$ is the mean value of row $a$ of ${\mathit{x}}_{add}$, ${({\mathit{\sigma }}_a)}^2$ is the variance of row $a$ of ${\mathit{x}}_{add}$, and the smoothing parameter $\epsilon$ is used to prevent the denominator from being zero in Equation (9).

2.2.4. Feed-Forward Neural Network

The feed-forward neural network is another important component in the Transformer. In each Transformer block, the feed-forward neural network accepts the output from the multi-head self-attention mechanism as input and maps it into a new representation vector using two linear transformations and a nonlinear activation function. As shown in Equation (12).

$${\mathit{X}}_{fnn,o}=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}({\mathit{W}}_1{\mathit{X}}_{Mnorm,o}+{\mathit{b}}_1)\cdot {\mathit{W}}_2+{\mathit{b}}_2,$$

(12)

where ${\mathit{W}}_1$ and ${\mathit{W}}_2$ are trainable weights of the first and second layers, respectively, ${\mathit{b}}_1$ and ${\mathit{b}}_2$ are bias vectors, ReLU is the activation function, and ${\mathit{X}}_{fnn,o}$ is the output of the feed-forward neural network. At the same time, the dropout operation [13] is adopted in the feed-forward layer to prevent the overfitting of training.

2.2.5. Encoder and Simplified Decoder

The input sequence designed in this paper contains the data of the process input vector and the process output vector. Since the multi-head self-attention mechanism of the encoder can learn the dependence between vectors of the input sequence itself, hence, only one encoder can achieve the effect of a classical Transformer containing encoding and decoding structure. In order to avoid the redundancy of encoding–decoding the multi-head attention mechanism in the decoder and the multi-head self-attention mechanism in the encoder, a fully connected neural network with two hidden layers is used as the decoder to obtain predictive output $\widehat{\mathit{y}}$ of the modified transformer in this paper. This simplifies the structure of traditional transformers. As shown in Equations (13) and (14).

$${\mathit{X}}_{Fnorm,o}=LayerNorm({\mathit{X}}_{Mnorm,o}+{\mathit{X}}_{fnn,o}).$$

(13)

$$\widehat{\mathit{y}}=(\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}({\mathit{W}}_1{\mathit{X}}_{Fnorm,o}^N+{\mathit{b}}_1)){\mathit{W}}_2+{\mathit{b}}_2,$$

(14)

where ${\mathit{X}}_{Fnor\mathit{m},o}^N$ is the output matrix of the encoder.

2.3. Learning Rate Optimization Algorithm

In this paper, the Adaptive Moment Estimation with Weight-decay (AdamW) algorithm is used to train and optimize the model. AdamW was proposed by Ilya Loshchilov and Frank Hutter in 2018 [37]. Weight attenuation was added to the parameter update in order to solve the problem of the non-convergence or slow convergence of the Adaptive Moment Estimation (Adam) algorithm [38]. Llugsi et al. proved that the AdamW optimizer has higher predictive accuracy than the Adam optimizer in a time series prediction task [39]. It is also proved that AdamW has smaller oscillation in the process of convergence and can produce better generalization performance. The updated equations of AdamW are as Equations (15)–(20).

$$L\left({\theta }_t\right)={\displaystyle \frac{1}{2}}{\left({\theta }_t\right)}^2={\displaystyle \frac{1}{2}}{\left({\mathit{y}}_t-{\widehat{\mathit{y}}}_t\right)}^2,$$

(15)

$${\mathit{m}}_t={\beta }_1{\mathit{m}}_{t-1}+(1-{\beta }_1)\mathsf{\nabla }L({\theta }_{t-1}),$$

(16)

$${\upsilon }_t={\beta }_2{\upsilon }_{t-1}+(1-{\beta }_2){(\mathsf{\nabla }L({\theta }_{t-1}))}^2,$$

(17)

$${\widehat{\mathit{m}}}_t={\displaystyle \frac{{\mathit{m}}_t}{1-{\beta }_1^t}},$$

(18)

$${\widehat{\upsilon }}_t={\displaystyle \frac{{\upsilon }_t}{1-{\beta }_2^t}},$$

(19)

$${\theta }_{t+1}={\theta }_t-\alpha ({\displaystyle \frac{{\widehat{\mathit{m}}}_t}{\sqrt{{\widehat{\upsilon }}_t+\epsilon }}}+\omega {\theta }_t),$$

(20)

where ${\mathit{y}}_t$ is the actual output at time $t$; ${\widehat{\mathit{y}}}_t$ is the predicted output at time $t$, ${\mathit{m}}_t$ is the update of the 1-st moment estimate, ${\upsilon }_t$ is the update of estimate of the 2-nd moment, ${\beta }_1$ is the exponential decay rate of the 1-st moment (generally, ${\beta }_1=0.9$), ${\beta }_2$ is the exponential decay rate of the 2-nd moment (generally, ${\beta }_2=0.999$), $\omega$ is the weight attenuation factor (generally, $\omega =0.01$), $\mathsf{\nabla }L({\theta }_t)$ is the gradient of loss function $L({\theta }_t)$, ${\widehat{\mathit{m}}}_t$ and ${\widehat{\upsilon }}_t$ are the bias correction of ${\mathit{m}}_t$ and ${\upsilon }_t$ at initial value of 0, respectively.

3. Evaluation Mechanisms of Models and Methods of Parameter Integration in the Prediction Time Domain

3.1. Evaluation Mechanism

In order to make a predictive effect of the modified transformer on nonlinear dynamic processes as good as possible, it is necessary to select a group of optimal network structure parameters from the set of structure parameters. Under optimal structure parameters, the predictive error of the model can be relatively minimal, and the fitting degree of the predictive curve and real curve can be relatively high. In order to achieve the above objectives, this paper proposes an evaluation mechanism for determining the optimal parameters of a neural network structure. The mechanism takes into account two performance evaluation indexes: RMSE and KL dispersion. The latter is obtained using a variational autoencoder (VAE). The basic idea is as follows:

Firstly, the performance indexes RMSE, Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) are used to measure the deviation between the real value and the predictive value. The smaller their values, the better the prediction of the model. As shown in Equations (21)–(23).

$$RMSE=\sqrt{{\displaystyle \frac{1}{va}}\cdot {\displaystyle \frac{1}{q}}\cdot {\displaystyle \frac{1}{L}}{\displaystyle \sum _{k=1}^{va}{\displaystyle \sum _{i=1}^L{\displaystyle \sum _{j=1}^q{({\widehat{\mathit{y}}}_{i,j}^{(k)}-{\mathit{y}}_{i,j}^{(k)})}^2}}}},$$

(21)

$$MAE={\displaystyle \frac{1}{va}}\cdot {\displaystyle \frac{1}{q}}\cdot {\displaystyle \frac{1}{L}}{\displaystyle \sum _{k=1}^{va}{\displaystyle \sum _{i=1}^L{\displaystyle \sum _{j=1}^q\left|{\widehat{\mathit{y}}}_{i,j}^{(k)}-{\mathit{y}}_{i,j}^{(k)}\right|}}},$$

(22)

$$MAPE={\displaystyle \frac{100}{va}}\cdot {\displaystyle \frac{1}{q}}\cdot {\displaystyle \frac{1}{L}}{\displaystyle \sum _{k=1}^{va}{\displaystyle \sum _{i=1}^L{\displaystyle \sum _{j=1}^q\left|\frac{{\widehat{\mathit{y}}}_{i,j}^{(k)}-{\mathit{y}}_{i,j}^{(k)}}{{\mathit{y}}_{i,j}^{(k)}}\right|}}\%}.$$

(23)

where $L$ represents the length of the dataset, ${\widehat{\mathit{y}}}_{i,j}^{(k)}$ is the model predictive output of the $k-\mathrm{th}$ output variable, ${\mathit{y}}_{i,j}^{(k)}$ is the real output of the $k-\mathrm{th}$ output variable, $\mathit{v}\mathit{a}$ represents the number of process output variables, and $q$ represents the length of the predictive output sequence corresponding to each time.

RMSE is primarily used to measure the overall predictive accuracy of a model, reflecting the average deviation between predicted values and actual values. However, the model’s network structure may introduce specific types of errors that might not be fully captured by RMSE. Therefore, for selecting a relatively optimal set of network structure parameters within a defined parameter space, more detailed model analysis and evaluation are necessary, including further examination of residual distributions.

VAE was first proposed by Kingma and Welling [40], which consists of an encoder and a decoder. The encoder part takes the initial data input and compresses and encodes it to obtain a low-dimensional hidden variable. Specifically, the encoder takes a real sample $\mathit{\tau }$ as the input and outputs its mean ${\mathit{\mu }}_i(\mathit{\tau })$ and variance ${\mathit{\sigma }}_i(\mathit{\tau })$ in the hidden variable space, where $i=1,2,\cdots ,\mathit{m}$ and $\mathit{m}$ is the dimension of the hidden variable space. Suppose the data in the hidden variable space satisfies a particular distribution $N(0,\mathit{I})$ with a mean value of 0 and a variance of $\mathit{I}$. By using the re-parameterization method, the hidden variables composed of mean and variance are taken as the input of the decoder, and reconstructed samples are finally output. Through repeated iterations, the RMSE of a real sample and a reconstructed sample is expected to be small enough, and the mapping of the real sample on the latent variable space can be as close as possible to the assumed data distribution. The loss function of VAE includes KLD and RMSE. KLD can be expressed as Equation (24).

$$\mathrm{K}\mathrm{L}\mathrm{D}=-0.5\times {\displaystyle \sum _{i=1}^{\mathit{m}}(1+\log ({\mathit{\sigma }}_i{(\mathit{\tau })}^2)-{({\mathit{\mu }}_i(\mathit{\tau }))}^2-{({\mathit{\sigma }}_i(\mathit{\tau }))}^2)},$$

(24)

$$\mathrm{K}\mathrm{L}\mathrm{D}=-0.5\times {\displaystyle \frac{1}{va}}{\displaystyle \sum _{k=1}^{va}{\displaystyle \sum _{i=1}^{\mathit{m}}(1+\log ({\mathit{\sigma }}_i^{(k)}{(\mathit{\tau })}^2)-{({\mathit{\mu }}_i^{(k)}(\mathit{\tau }))}^2-{({\mathit{\sigma }}_i^{(k)}(\mathit{\tau }))}^2)}}.$$

(25)

In this paper, VAE was used to calculate the KLD between residual sequence and standard normal distribution, as shown in Equation (25). The residual distribution is tested based on KLD. Where ${\mathit{\mu }}_{i,j}^{(k)}(\mathit{\tau })$ and ${\mathit{\sigma }}_{i,j}^{(k)}(\mathit{\tau })$ are the mean and variance of the $k-\mathrm{th}$ output variable in the hidden variable space. In a statistical sense, KLD can be used to measure the degree of difference between two distributions. The process studied in this paper is generally in a random noise environment, and random noise has certain distribution characteristics. If only residual distribution caused by noise is compared with standard normal distribution, the difference between the two should be the smallest. However, if the residual also has the contribution of model error, the residual distribution will be more complex and diverse, and the difference from the standard normal distribution will be greater. Hence, we can derive the following:

• The larger the KLD → the larger the difference between the residual distribution and standard normal distribution → the stronger the non-randomness in the residual → the larger the output error → the larger the model error → the value of neural network structure parameters is inappropriate;
• The smaller the KLD → the smaller the difference between the residual distribution and standard normal distribution → the weaker the non-randomness in residual → the smaller the output error → the smaller the model error → the more appropriate the selection of network structure parameters.

To sum up, a group of structure parameters minimizing KLD is selected from the set of structure parameters corresponding to the smaller RMSE. This group of parameters is the optimal solution for the structure parameters of the neural network. In the experiment, this paper selects the first ten groups with better RMSE. However, the optimal structure parameters are the group corresponding to the minimum KLD among these ten groups. Based on this, the specific value vector of structure parameters is determined. In general, according to the above analysis, the specific process of the evaluation mechanism is to establish the value matrix of all structure parameters. Each row represents a complete value of structure parameters. Under this definition, the loop body traverses each row, trains, and tests the model $NN(•)$ in turn, in order to obtain residual sequence matrix. Firstly, the RMSE is obtained and sorted from small to large. Then, take out the smaller serial numbers of the first 10 groups. According to these serial numbers, an optimized residual sequence matrix is formed. Next, the KLD of the residual sequence under these serial numbers is calculated by the loop body. Finally, the serial numbers and the group of structure parameters corresponding to the minimum KLD are taken as the optimal group of structure parameters. It should be mentioned that RMSE and KLD are calculated after data normalization in this paper. Finally, the data are denormalized, and then the predictive and real curves are compared by drawing a graph. The following algorithmic pseudo-code (Algorithm 1) provides a more intuitive representation of the exact flow of the evaluation mechanism.

Algorithm 1. Algorithm for evaluation mechanism

Begin

1.   Initialize the network structure parameter vector ${\mathit{p}}_1\in {ℝ}^{L_1\times 1}$, ${\mathit{p}}_2\in {ℝ}^{L_2\times 1}$, ${\mathit{p}}_3\in {ℝ}^{L_3\times 1}$.

2.   Initialize $kl\_optimal=0$ and $seq\_optimal=0$.

3.   Generate the parameter matrix $\mathit{M}\in {ℝ}^{L\times 3}$ using ${\mathit{p}}_1$, ${\mathit{p}}_2$ and ${\mathit{p}}_3$. $L=L_1\cdot L_2\cdot L_3$.        // Overall structure of selected neural networks //

4.   class NN

5.     NNstructure

6.     train()

7.     test()

8.   End

9.   NN1 = NN()        // Calling different structure parameters for network training and prediction, and getting the corresponding predictive error and residual sequence //

10.  For $i=1:L$         // Normalization of input data //

11.    [data_normalize_train, data_normalize_test] = normalize(data)

12.    NNstructure = M(i, :)

13.    NN1.train(NNstructure, data_normalize_train)

14.    residual(i,:) = NN1.test(NNstructure, data_normalize_test)

15.    rmse(i) = RMSE(residual(i, :))

16.  End for         // Rank RMSE from small to large and get the corresponding parameter serial //

17.  [rmse, seq] = sort(rmse)         // Calculate the KLD of residual sequence corresponding to the top 10 RMSE after ranking by VAE //

18.  For $i=1:10$

19.    residual2(i, :) = residual(seq(i), :)

20.    [vae_kl(i, 1), ... , vae_kl(i, n)] = VAE_KL(residual2(i, :))

21.    kl(i) = average(vae_kl(i, j))        // j = 1, 2, …, n, n represents the number of output variables //

22.   End for          // The minimum value of KLD and its corresponding structure parameter serial number are obtained by bubble sorting //

23.  kl_optimal = kl(1)

24.  For $i=2:10$

25.    If kl(i) < kl_optimal

26.      kl_optimal = kl(i)

27.      seq_optimal = seq(i)

28.    End if

29.  End for

30.  print(seq_optimal, M(seq_optimal, :))        // The network is modeled and predicted again under the optimal structure parameters //

31.   NNstructure = M(seq_optimal, :)

32.  NN1.train(NNstructure, data_normalize_train)

33.  residual = NN1.test(NNstructure, data_normalize_test)        // Renormalized the predictive output, and draw the predictive curve and the real curve //

34.  yy = renormalize(y-residual)

35.  plot(yy)

End

3.2. Predictive Time-Domain Parameterisation Methods

After determining the optimal network based on the previous description, the following is to predict the key process variables according to the actual situation. When performing multi-step predictions, the prediction time domain is not chosen arbitrarily and needs to be analyzed specifically according to the actual situation. To address the problem of uncertainty in the prediction time domain, a hybrid network is designed in this paper, which consists of $q$ sub-networks. Each sub-network is of the same chosen model class, except that they have different prediction steps.

For the case where the prediction time domain is $q$, the hybrid network contains $q$ sub-networks capable of predicting the data ${\widehat{y}}_t,\dots ,{\widehat{y}}_{t+q-1}$ in real time as well as at $q-1$ moments in the future, respectively, and the $q-\mathrm{th}$ sub-network has a prediction step of $q$. For the case where the prediction time domain is 1, the hybrid network has only 1 sub-network, which can only predict the data ${\widehat{\mathit{y}}}_t$ corresponding to the real-time moment with a prediction step of 1. Observing the fitting degree between the predicted output sequence at the last prediction time step of each subnetwork and the target sequence. When the predictive trend of $\left(q+1\right)-\mathrm{th}$ subnet becomes chaotic, then the predictive time domain is determined to be $q$.

Remark 1. 

For the I/O of the neural network, input is the historical time series data matrix of different variables at the current and forward moments, which does not change. But the output is the predictive value of output variables for the next $q$ steps, and it has to change. In particular, $q$ changes gradually from 1 to large, and different values of $q$ correspond to different subnets. This paper build $NN(•)$ with such a hybrid structure as described above. The evaluation index RMSE of $NN(•)$ is still relatively accurate until $q=q_{\max }$. However, when $q=q_{\max }+1$, RMSE increases significantly. It means that the loss function value becomes larger, and the predictive performance becomes worse, that is, the predictive time domain is determined to be $q_{\max }$. In addition, it is worth noting that the predictive time domain value should be further determined after the overall modeling, evaluation, and prediction of  $NN(•)$ is completed, that is, after the optimal neural network is determined (including the value of optimal structure parameters and the specific form of optimal $NN(•))$ because it is closely related to the internal dynamic characteristics of the system or process itself, and it needs to be further effectively determined on the basis of the optimal network.

Modeling and evaluation are mainly used in the offline training and testing phase to determine the optimal neural network, while prediction is mainly used in the online phase to provide effective online prediction and estimation of key output indicators or assay indicators. In the implementation process of multi-step prediction, the ability to predict the next few moments is mainly determined by the prediction time domain, which determines the upper limit of the prediction step size, the flow chart on the adjustment of the prediction time domain parameters is shown in Figure 3a.

The online prediction stage can predict the future moments of data arriving at the prediction time domain, and it can also determine the future prediction moments or real-time moments according to the actual needs of the project and output the predicted values of the real-time moments and the future moments in their entirety. The specific online prediction stage flow chart is shown in Figure 3b, where “actual prediction time domain” means a prediction time domain determined according to the actual production needs within the ideal maximum prediction time domain. After obtaining the prediction time domain for the actual production process, the corresponding hybrid network structure is established, and all the internal characteristic parameters of the corresponding sub-networks are extracted from all the internal characteristic parameters of the hybrid network stored in the parameterization stage of the prediction time domain, so as to generate several optimal sub-networks, and then predict the real-time data and the data at different moments in the future based on the various optimal sub-networks, so as to realize the task of online prediction.

4. Integrated Modelling, Evaluation, and Prediction Experiment

In this section, a Transformer–VAE model is established, which integrates the proposed Transformer model for industrial time series I/O design with the evaluation mechanism introduced in Section 3. This integration allows for obtaining an optimal set of network structure parameters for network prediction, along with the corresponding predictive results. The structure is illustrated in Figure 4. Where solid black arrows indicate the traversal of all group structure parameters within the structural matrix. Through network training and prediction, the RMSE corresponding to each group of network structure parameters is obtained, and the top few groups with smaller RMSE are selected for calculating the KL divergence. The group of network structure parameters corresponding to the minimum KL divergence represents the optimal structural parameters. Dashed orange arrows signify that after obtaining the optimal structural parameters, the network is retrained and predicted using these optimal parameters to achieve more accurate final prediction results.

In this section of the experiment, this paper models, evaluates, and predicts a real nonlinear dynamic process in the petrochemical field based on a modified transformer model. Compared with two baseline models, the predictive performance and effectiveness of the proposed model are verified. In the end, the determination of the predictive time domain under different conditions is explored based on the proposed model. The experiment environments are as follows: MATLAB R2021a software is used to realize the baseline models, open-source machine learning library Scikit-learn and deep learning framework Torch are used to implement our model. The experimental hardware environment of the baseline models is configured with Intel(R) Core (TM) i7-8550U CPU @ 1.80 GHz 1.99 GHz (Intel Corporation, Santa Clara, CA, USA), and the experimental hardware environment of our model is configured with Intel(R) Xeon(R) Gold 5118 CPU @ 2.30 GHz (Intel Corporation, Santa Clara, CA, USA), each of the 4 GPUs is NVIDIA GeForce RTX 2080Ti (Nvidia Corporation, Santa Clara, CA, USA), and the memory is 96 G.

4.1. Selection of Research Subjects

In this example, this paper uses actual data of catalytic cracking fractionation system in Tianjin Dagang Oil Refinery to verify the predictive ability of modified transformer model for nonlinear dynamic process. The catalytic cracking fractionation system is mainly composed of the main fractionator, oil and gas condensation cooling system at the top of the tower, diesel stripping tower, recycle oil tank, and middle circulating reflux. The high-temperature reaction oil-gas mixture exiting from the top of the reactor is fed into the desuperheater section of the lower part of the main fractionator. After heat exchange and washing, the catalyst powder enters the fractionator body. The partial process flow of FCC fractionator C2201 in this plant is shown in Figure 5. Its main fractionator is equipped with top circulating reflux, middle circulating reflux, partial refining reflux, and oil slurry circulating reflux.

This paper mainly considers three closed-loop control loops TC2202, FC2207, and LC2209, which have mutual coupling and mutual influence. To be specific, the change in the branch valve lift controlled by TC2202 affects flow and temperature, and the change in the main pipeline valve lift controlled by FC2207 also affects flow and temperature. Secondly, whether the opening of the main or branch pipeline changes, the input flow changes. At the same time, the liquid level under the LC2209 will change. Hence, three closed-loop control loops are mutually influenced and coupled. This paper set three manipulated variables of the above three closed-loop control loops as the process input variables $u_1$, $u_2$, and $u_3$, and set three controlled variables as the process output variables ${\mathit{y}}_1$, ${\mathit{y}}_2$, and ${\mathit{y}}_3$. The sampling interval is 1 min. This paper collected 2823 data of each input and output variable for three days from 12 August 2021 10:43 to 12 October 2021 09:44 as the dataset. The dataset is partitioned into training, validation, and test data by a ratio of 3:1:1 in this experiment. Then, based on the proposed model and evaluation mechanism, the multivariate nonlinear dynamic process with three inputs and three outputs is modeled, evaluated, and predicted.

In this experiment, to evaluate the forecasting performance of our proposed method, two baseline models are compared, including RNN and Gated Recurrent Unit (GRU). RNN is a classical deep neural network and also a memory model where connections between computational units form a directed circle. Unlike feed-forward networks, RNNs can use their internal memory to process arbitrary sequences of inputs, thus providing dynamic memory. GRU is a variation in Long Short-Term Memory (LSTM). It can not only solve the problem of gradient disappearance in RNNs but also simplify the network structure of LSTM and improve the convergence speed [41].

4.2. Determination of Optimal Structural Parameters of Transformer

In terms of model parameter settings, the optimal structural parameters of the relevant models in this section are derived according to the experimental procedure for optimal structural parameter determination in Section 3.

For the Transformer model incorporating industrial timing I/O design, the batch size is set to 100, dropout is set to 0.1, the learning rate is set to 0.009, and the AdamW function is used as the model optimizer and the loss function is the same as that of the baseline model for evaluating the model. In addition, the number of forward moments $n$ in the Transformer model designed in this paper is set to 30. The number of encoder layers ($n_e$), the number of attention heads ($n_h$), and the number of hidden layer neurons ($n_n$) in the feedforward neural network are set as variable parameters as the network structure parameters. The range of initialized network structure parameters in this subsection of the experiment: $n_e=\{1,2,3,4,5,6\}$, $n_h=\{2,4,6,10,15,20\}$, $n_n=\{50,60,70,80,90\}$.Then, this subsection determines the optimal network structure parameters based on the Transformer–VAE model that makes the designed Transformer predictions the best for subsequent experiments.

Firstly, we perform a single-step prediction of the validation set under 180 groups of different structure parameters and obtain the first ten groups of parameter combinations with smaller RMSE through the evaluation mechanism. In order to minimize the interference of model errors on predictive results, we further calculate the KLD of the residual sequence corresponding to each group of parameters through the VAE network.

Table 1. Evaluation results of evaluation mechanism.

$({\mathit{n}}_{\mathit{e}}$$,{\mathit{n}}_{\mathit{n}}$$,{\mathit{n}}_{\mathit{h}}$)	RMSE	KLD
(1,80,4)	0.01357	0.2730
(1,90,4)	0.01364	0.2673
(1,60,4)	0.01455	0.2801
(1,90,2)	0.01473	0.2571
(1,50,2)	0.01483	0.2699
(2,80,2)	0.01499	0.2753
(2,70,2)	0.01501	0.2740
(2,50,2)	0.01545	0.2777
(1,50,4)	0.01620	0.2804
(1,50,6)	0.01631	0.2913

shows the predictive results of the modified transformer under these ten parameter combinations. Through the performance index RMSE, it can be seen that the proposed model has high predictive accuracy under these ten parameter combinations. The optimal parameter combination is (1,80,4). Its RMSE is 0.01357. However, in order to reduce the influence of model error on predictive results as much as possible, residual characteristics are further excavated by calculating KLD. Finally, the optimal solution of neural network structure parameters is obtained as (1,90,2). Hence, we set $n_e$, $n_h$ and $n_n$ to 1, 90, and 2, respectively, for the follow-up experiments of this example. It should be mentioned that RMSE and KLD are calculated after data normalization in this paper, and so are the performance indexes used in the following experiments. Subsequently, the data need to be denormalized.

4.3. Comparison Experiment

In this paper, single-step prediction of a multivariate nonlinear dynamic process is performed using the proposed model and three reference models. The number of forward propagation units of the reference model GRU is set to 30, the number of backward updating units to 11, the dimensionality of the state variables within the module to 30, the RMSE is used as the loss function, and the Adam algorithm is chosen to optimize the network. The reference model RNN contains a hidden recurrent layer consisting of 30 neural units, with a backward update unit count of 30, and the rest of the setup is consistent with GRU. The reference model Transformer uses a traditional data input–output structure, and the rest of the parameters are consistent with the modified Transformer model proposed in this paper. The training process is conducted for 200 epochs.

Table 2. Performance comparisons of different models.

Models	RMSE	MAE	MAPE (%)
RNN	0.1238	0.1125	71.4814
GRU	0.0786	0.0671	60.8175
Transformer	0.0468	0.0264	67.5926
Modified transformer	0.0136	0.0096	23.1028

shows the single-step predictive results of four models on the test set. As shown in Table 2, the experimental results show that compared with the three baseline models, the modified transformer has a smaller predictive error and higher predictive accuracy. More particularly, the RMSE, MAE, and MAPE of the modified Transformer model designed in this paper are reduced by 70.9%, 63.6%, and 65.8%, respectively, over Transformer, 82.7%, 85.7%, and 62.0% over GRU, and 89%, 91.5%, and 67.7% over RNN.

Figure 6 visually presents the comparisons between predictive curves and real curves of three models in the case of single-step prediction, where legend $t$ represents the prediction at the current moment in time. We collect 100 moments of data ranging from 12 October 2021 00:30 to 12 October 2021 02:09 in the test set. It can be seen from Figure 6 that the predictive value based on the modified transformer can well fit the trend of the actual value and has higher modeling accuracy.

4.4. Predictive Time-Domain Parameterisation of Modify Transformer

This subsection performs multistep prediction based on a Transformer model incorporating industrial timing I/O design in the context of a catalytic cracking fractionation system. The objective is to explore the laws for predicting time-domain parameters for multivariate nonlinear dynamic processes.

As shown in Figure 7, the comparisons of predictive errors are produced by the designed hybrid network on the test set, where $t+p$ is used to indicate the future $p-\mathrm{th}$ moments. This hybrid network includes four subnets. This means that the $j-\mathrm{th}$ subnet can provide the comparative errors at the future $j-\mathrm{th}$ step, which are described in sequence in Figure 7a–d. As shown in Figure 8, this hybrid network includes five subnets. This means that the $j-\mathrm{th}$ subnet can provide the comparative errors at the future $j-\mathrm{th}$ step, which are described in sequence in Figure 8a–e. It can be seen from Figure 8 that when the predictive step increases to 5, the predictive trend begins to become chaotic.

Hence, it is more appropriate to choose 4 as the predictive time domain in this example. The performance indexes in

Table 3. Performance comparisons of modified transformer under different predictive time domains.

Indexes	t + 1	t + 2	t + 3	t + 4	t + 5
RMSE	0.01683	0.02261	0.03023	0.03838	0.04940
MAE	0.01195	0.01517	0.01881	0.02385	0.03063
MAPE (%)	28.64307	36.51211	45.86090	58.12561	75.83628

can also illustrate the above conclusion. Table 3 shows the performance indexes of modified transformers under different predictive time domains. It can also be seen that with the increase in the predictive step, the predictive effect of our model is becoming worse and worse, and the predictive accuracy is becoming lower and lower. The longer the predictive step is, the harder it is to predict the corresponding moment in the future. More specifically, when the predictive step is 4, RMSE, MAE, and MAPE are 2.28, 2, and 2.03 times for those when the predictive step is 1 ($\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=0.01683, \mathrm{M}\mathrm{A}\mathrm{E}=0.01195,$$\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}=28.64307$). The experimental results of Figure 7 and Figure 8 and Table 3 show that a modified transformer can accurately predict multivariate nonlinear dynamic processes in multiple steps when the predictive time domain is selected properly.

5. Discussion

In this paper, we propose a Transformer model that incorporates industrial timing I/O design. On this basis, we conduct a comparative experiment on a real multivariate nonlinear dynamic process through a modified transformer model and three baseline models. These results corroborate the conclusions of a large number of previous studies [19] that the Transformer model can be used in time series forecasting.

It can be concluded from Table 2 and Figure 6 that the method proposed in this paper is superior in the task of predicting nonlinear dynamic processes. This result is consistent with the findings of Reza et al. [42]. This is because the ability of GRU and RNN to extract information features is relatively poor, and it is not easy to accurately predict long-time series [15,41]. The Transformer, which is designed by integrating industrial time series I/O, can fully capture the impact of the input sequence on the output sequence. In other words, it can not only effectively learn hidden nonlinear correlation characteristics of multivariate time series data, but also learn different effects of different input sequences on output sequence at different times. From Table 2, it can be seen that GRU has a better prediction than RNN. This is consistent with the statement that GRU has a longer memory than RNN, and GRU solves the problem of vanishing gradient to some extent [43].

The present paper proposes an integrated approach to the modeling, evaluation, and prediction of Transformer–VAE. The optimal parameter solution of the modified transformer model is obtained through an evaluation mechanism.

In addition, we carry out an exploratory experiment on the predictive time domain. It can be seen from Figure 7 that with the continuous increase in the predictive step, the fitting degree of the predictive curve and the real curve is becoming worse and worse, and the tracking effect is also becoming worse and worse. However, under each future step of this time domain, the predictive curve can track the trend of the actual curve well, so there is no way to determine the predictive time domain of the process. As shown in Figure 8, the predictive trend begins to become chaotic. It means that the predictive data cannot track the trend of the actual data and are difficult to predict more accurately. This also reflects a practical problem of multi-step prediction. The predictive time domain needs to be selected according to the actual situation. This selection depends on the internal dynamics of the system or process itself, rather than the fact that a modified transformer model can well predict the data at any time in the future.

The prediction model proposed in this paper can provide a valuable reference for industrial early warning and other applications based on multivariate time series data.

6. Conclusions

This paper presents an integrated approach to modeling, evaluation, and prediction using Transformer–VAE. The research object is an industrial catalytic cracking fractionation system. The comparative experiments and analyses demonstrate that the proposed evaluation system is capable of identifying the optimal network structure parameters. Furthermore, the Transformer model, which integrates the industrial time-sequence I/O design exhibits enhanced prediction accuracy, while the prediction time domain is investigated through the hybrid network to ascertain the optimal prediction time domain range. The Transformer model, which integrates industrial time series I/O, offers a valuable reference point for industrial early warning based on multivariate time series data and other applications. However, this paper still has areas that need to be studied in depth, the subsequent research is described below: this paper lacks interpretability in the prediction of nonlinear dynamic processes using deep learning models, visualizations can then be used to improve the interpretability of the model; this paper does not consider the fitness of the optimal network structure of the model to the system when the system undergoes dynamic changes. In the future, an adaptive adjustment mechanism for the network structure could be designed to accommodate changing industrial process conditions or significant fluctuations in process loads. In addition, this paper is limited regarding the prediction time domain and relies heavily on experimental determinations. Subsequent studies will address the relationship between the prediction time domain and industrial process modeling.