A Novel Reconstruction Method for Irregularly Sampled Observation Sequences for Digital Twin

Abstract

Various uncertainties such as communication delay, packet loss and disconnection in the Industrial Internet, as well as the asynchronous sampling of sensors, can cause irregularity, sparsity, and misalignment of sampling sequences, and thereby seriously affect the training and prediction performance of a digital twin model. Sequence reconstruction is an effective way to deal with the above problems, but if the measurement data become sparse or contain significant noise due to packet loss and electromagnetic interference, existing methods struggle to achieve ideal results. Therefore, a novel variational autoencoder model based on a parallel reference network and neural controlled differential equation (PRN-NCDE) is proposed in this article to solve the problem of reconstructing irregular series under sparse measurements and high noise levels. First, a multi-channel self-attention module is established, which can not only analyze the position and feature information of the sampled data to improve the reconstruction accuracy under sparse measurements, but also effectively tackle the misalignment and irregularity of the observation sequence through multi-channel and mask mechanisms. Second, to improve the accuracy of sequence reconstruction under large noise levels, a PRN is established to obtain reference features, which are weighted and fused with the features of observed data. Third, we use the NCDE to construct a decoder that can combine the control input of the system to predict the output values to solve the problem of sequence reconstruction in a controlled system. Finally, a weighted loss function is constructed to better train the network parameters of the model. This article takes the furnace of the boiler system in a coal-fired power plant as the test object to verify the effectiveness and fitting accuracy of the proposed PRN-NCDE model compared to the existing methods for a controlled system under sparse measurements and large noise levels. Simulation results show that the proposed PRN-NCDE model can improve the estimation accuracy by more than 50% and 70% compared with the recurrent neural network-NCDE (RNN-NCDE) under different sampling numbers and noise levels, and by more than 80% and 60% compared with the recurrent neural network-NODE (RNN-NODE).

1. Introduction

As one of the key technologies for the digital transformation of traditional industries, digital twin can create virtual mappings of physical entities, which enables real-time reflection of the entire lifecycle of physical systems and facilitates their optimization and monitoring [1]. Maintaining consistency between the twin model and the physical system is crucial for the successful implementation of digital twins’ functionalities [2]. However, several factors can degrade the training and prediction performance of twin models. First, network communication often has irregularities such as delays and packet loss, leading to irregular sampling intervals [3]. Second, sensors in physical systems have varying sampling periods, causing misalignment and sparsity in multivariate sequences. These issues make the sampled sequences unsuitable for model training and produce negative impacts on prediction performance [4].

Sequence reconstruction is an effective means for these problems in digital twin systems. It involves rebuilding irregular observation sequences using function fitting or generative neural networks to obtain regular sequence data. Traditional methods include polynomial fitting [5] and maximum likelihood estimation [6]. Polynomial fitting approximates complex functions by learning suitable polynomial coefficients but requires predefining the polynomial degree [7,8]. A higher predefined polynomial degree may lead to overfitting and a lower one may have negative impact on the fitting performance. For complex and ever-changing observation sequences, it is often difficult to make an assumption on its degree. Moreover, polynomial fitting is very sensitive to noise, resulting in its inapplicability for noise-corrupted observation sequences [9]. Based on the curve fitting method in statistics, maximum likelihood estimation determines model parameters by calculating likelihood function values but shares similar limitations with polynomial fitting, such as narrow applicability and the need to assume a model form. Narrow applicability means this method can only produce satisfactory results under some certain distributions [10]. And an assumed model form will prevent the method from dealing well with complex sequences influenced by control inputs. Fixed-time discretization is another conventional approach that divides continuous-time observations into fixed-length, non-overlapping data windows [11]. Although this method is easy to follow and utilize, it may induce empty windows and data aggregation issues when window sizes are increased [12]. Therefore, window size is a key parameter for this method. However, determining an appropriate window size for complex sequences is often difficult.

Alternative solutions involve developing neural network models that can directly use irregular sequences as inputs, including interpolation-based and neural ordinary differential equation (NODE)-based methods. Interpolation-based methods use past and future observations for interpolation. Yoon et al. [13] studied a multi-directional recurrent neural network (RNN) that can use past and future observation data flows at a given time step to realize interpolation, thus improving estimation performance in missing measurement scenarios. Shukla et al. [14] developed an interpolation prediction network, which consists of several semi-parametric radial basis function (RBF) interpolation layers. This interpolation network can interpolate multivariate and irregularly time series against a set of reference time points. In [15,16,17], attention mechanism is used for time encoding and several multi-time attention-based RNNs are proposed for irregular sequence modeling. However, these interpolation-based methods face variable uncertainty when observation intervals change significantly [18]. Exponential decayed RNN autoencoder frameworks for sequence reconstruction were studied in [19,20], but Mozer et al. [21] found these networks have no obvious improvements in prediction accuracy compared to the standard RNN autoencoders. In 2018, Chen et al. introduced the NODE network [22], offering new solutions for irregular sequence reconstruction. Subsequently, Rubanova et al. [23] developed a latent NODE model, which combines RNN and NODE in a variational autoencoder (VAE) framework, outperforming the exponential decay RNNs. The GRU-ODE-Bayes model proposed by Brouwer et al. [24] achieved good results on sparse data. Huang et al. [25] presented LG-ODE, which is a latent ordinary differential equation VAE for multi-agent systems with known graph structure. LG-ODE can learn the embedding of high-dimensional trajectories and deduce the latent system dynamics simultaneously. In practical applications, most systems have state trajectories influenced not only by initial values but also by control inputs. NODEs cannot account for control inputs’ impact on output sequences, limiting their application in controlled system sequence reconstruction. Moreover, when the measurement data are sparse or noisy due to network issues and electromagnetic interference, existing methods struggle to handle digital twin sequence reconstruction tasks.

To address these challenges, this article proposes a VAE model based on a parallel reference network (PRN) and neural controlled differential equation (NCDE) [26] for noisy multivariate irregular sequence reconstruction in controlled systems. First, we establish a multi-channel self-attention (MCSA) module to analyze the position information and correlations of sampled data in the target sequence, improving reconstruction accuracy under sparse measurements while effectively handling misalignment and irregularity through multi-channel and masking mechanisms. Second, to enhance reconstruction accuracy under high noise levels, we construct a PRN to obtain reference features from the prediction results of digital twin model and fuse them with actual data features. We also calculate feature weights based on the noise level of the observation sequence to determine the proportion of actual and reference information in the fused features. Third, we use NCDE to build a decoder that can predict observations at any time by incorporating control inputs to solve the sequence reconstruction problem in controlled systems. Finally, we develop a weighted loss function based on feature weights to better train the model’s network parameters. Simulation experiments demonstrate the effectiveness and fitting accuracy of the proposed model for controlled systems under sparse measurements and high noise levels compared to the existing methods.

Overall, the main contributions of this article are threefold:

(1)

We propose a PRN-NCDE-based VAE model that improves sequence reconstruction accuracy for controlled systems under sparse measurements and high noise levels.

(2)

We develop an MCSA module that can not only analyze data position and correlations to enhance reconstruction performance under sparse measurements, but also effectively handle misaligned and irregular observation sequences.

(3)

To improve reconstruction accuracy under high noise levels, we establish a PRN to obtain reference features and calculate feature weights based on the noise level of observation data for weighted fusion of latent features.

The rest of this article is organized as follows. Section 2 formally describes the irregular sequence reconstruction problem. Section 3 presents the overall framework of the PRN-NCDE model and analyzes the MCSA module, PRN, NCDE network, and weighted loss function. Section 4 validates effectiveness of the proposed model compared to the existing methods through a boiler system under different sampling numbers and high noise levels. Conclusions and future work are discussed in Section 5.

2. Problem Formulation

In this section, we first give formal descriptions of multivariate regular and irregular observation sequences.

Definition 1.

Consider an observation dataset $𝒟={\left\{\left(y_n,t_n\right)\right\}}_{n=0}^N$, where $y_n\in {ℝ}^d$ is the observation vector at sampling time $t_n$, and $N+1$ is the total number of samples. Let NaN denote the missing data in the observation vector. And when an observation vector has missing data, its dimension satisfies $\dim \left(y_n\right)<d$. Then, if dataset $𝒟$ meets the following two conditions:

(1)

For any two consecutive observation vectors $y_n$ and $y_{n+1}$ in $𝒟$, the sampling interval $\Delta t_n=t_{n+1}-t_n$ is a constant $C$, i.e., $t_{n+1}-t_n\equiv C$ for $\forall n\in \left[0,N\right]$.

(2)

No observation vector $y_n$ in $𝒟$ contains missing data NaN, i.e., $\dim \left(y_n\right)=d$ for $\forall n\in \left[0,N\right]$.

Then, $𝒟$ forms a regular sampled time series. Conversely, if $𝒟$ does not satisfy both conditions, it forms an irregular sampled time series.

Condition (1) in Definition 1 describes the regularity of the sampling sequence distribution on the time dimension, that is, whether a fixed sampling interval is followed. Condition (2) describes whether the sampling time of every variable in the multivariate observation sequence is aligned. Only when $𝒟$ satisfies both conditions, which means that the sampling times of all the variables are aligned and there is an unified and fixed sampling interval, it can be regarded as a regular multivariate observation sequence. If either of the conditions is not met, it is considered as an irregular multivariate observation sequence. Overall, a regular multivariate sampling sequence needs to meet both conditions: a fixed sampling interval and aligned sampling times.

Due to various uncertainties in communication networks and non-synchronous sampling of sensors, twin systems usually receive non-aligned, irregular sampling sequences. Consider the nonlinear discrete-time system corrupted by measurement noise as follows:

$$\left\{\begin{array}{l}{x}_{n+1}=f\left(x_n,u_n\right)\\ y_{n+1}=h\left(x_{n+1},u_n\right)+v_n\end{array}\right.$$

(1)

where $x_n\in {ℝ}^s$, $u_n\in {ℝ}^c$, and $y_n\in {ℝ}^d$ are the state vector, control vector, and observation vector of the system at sampling time step $t_n$, respectively. $f:{ℝ}^s\times {ℝ}^c\to {ℝ}^s$ and $h:{ℝ}^s\times {ℝ}^c\to {ℝ}^d$ are the nonlinear state transition function and measurement function, respectively. For system (1), we make the following assumptions:

Assumption 1.

The process noise of the system is ignored, and it is considered that system (1) only contains measurement noise $v_n$, which follows a Gaussian distribution with mean $\mathbf{0}$ and covariance $R$.

Assumption 2.

Due to the uncertainties of network communication and non-synchronous sampling of sensors, the time series formed by the observations $y_n$ of system (1) is no longer regular. Instead, it is a non-aligned irregular time series with non-aligned sampling times and non-fixed sampling intervals.

Based on the above descriptions, the main objective of this article is as follows: Given a set of noisy, non-aligned and irregular observation sequences ${\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N$ from a controlled system, we want to develop a VAE network model that can maximize the evidence lower bound (ELBO) given by

$$\underset{W_{ec},W_{dc}}{\max }{𝔼}_{x_0\sim p_e\left(x_0|{\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N\right)}\left[\log p_d\left({\widehat{y}}_0^{irr},\cdots ,{\widehat{y}}_N^{irr}|x_0\right)\right]-\mathrm{KL}\left[p_e\left(x_0|{\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N\right)\left|\right|p\left(x_0\right)\right]$$

(2)

to train the encoder network weights $W_{ec}$ and decoder network weights $W_{dc}$, as shown in Figure 1. ${\widehat{y}}_n^{irr}$ is the estimated observation vector from the neural network model, corresponding to the original observation vector $y_n^{irr}$; $p_e$ is the conditional distribution that the encoder network needs to approximate; $p_d$ is the conditional distribution that the decoder needs to approximate; $p\left(x_0\right)$ is the prior distribution. Once the optimal sequence reconstruction model is trained, it can predict observations at any desired time and produce the regular sequence needed ${\left\{\left(y_n^r,t_n^{\prime }\right)\right\}}_{n=0}^{N^{\prime }}$ by the twin system. Here, $y_n^r$ is the estimated observation vector at desired time $t_n^{\prime }$; $N^{\prime }+1$ is the number of samples in the regular sequence.

The next section will focus on the reconstruction method for noisy irregular sequences in controlled systems and propose a VAE network model based on PRN-NCDE to obtain regular sequences with higher fitting accuracy.

3. Irregularly Sampled Observation Sequence Reconstruction Method Based on PRN-NCDE

This section constructs a deep generative network model based on PRN-NCDE within the VAE framework, which mainly consists of an encoder and a decoder. First, the overall network framework is established, and the workflow of the model and the functions of each module are described. Then, the establishment of the MCSA module in the encoder, the PRN and the calculation method of latent feature weights, as well as the construction method of the NCDE network in the decoder, are analyzed in detail, and the corresponding weighted loss function is established. Finally, the approximation ability of the reconstruction model for the observation sequence is discussed. We refer the readers to [27,28], and [26,29] for detailed theories and implementations of VAE, MCSA and NCDE.

3.1. Overall Network Framework of PRN-NCDE

The overall network framework of the sequence reconstruction model, as shown in Figure 2, consists primarily of an encoder and a decoder. The encoder comprises two identical network structures that map the actual observational sequence and the reference observational sequence into latent features, subsequently merging them through weighted fusion based on the noise level of the actual observational data. The decoder, consisting of an NCDE network and an output layer, utilizes these fused latent features to reconstruct the desired regular sequence.

Initially, the input of the encoder is made up by the actual irregular observation sequence and the reference observation sequence, each passing through the identical neural network architecture to produce the actual and reference latent features, respectively. In order to enhance reconstruction accuracy under sparse measurement conditions, a multi-head self-attention mechanism is adopted to process both sequences, focusing on the position information of the measurement data and capturing the correlations among them. The multi-channel self-attention mechanism specifically addresses the irregularity and non-alignment issues in actual observation sequences.

Taking the coal-fired power plant boiler system as an illustrative example, multivariate irregular observational sequences are first expanded to equal-length sequences based on desired sampling intervals to deal with irregular sampling intervals. Furthermore, discrepancies in sensor sampling times and frequencies lead to non-alignment among variables such as gas density, gas temperature, and gas oxygen content, causing inconsistent positions of missing values across different variables. Therefore, the expanded sequence is segmented along variable dimensions and input into corresponding masked multi-head self-attention modules, where masks indicate missing values, preventing negative impacts during network training and prediction. Subsequently, the output sequence from the self-attention module undergo group normalization and reversal before being fed into an LSTM network for reverse-time inference from time $t_{N^{\prime }}$ to $t_0$, obtaining the hidden state at the initial time step $t_0$. A fully connected layer maps this hidden state to the mean and variance of the latent feature distribution, and by combining with noise sampled from a standard Gaussian distribution can generate latent features.

Similarly, reference observation sequences generated by a digital twin model undergo processing through an identical PRN structure to obtain reference latent features. Incorporating PRN to acquire the reference latent features provides prior information, thereby enhancing the reconstruction model’s fitting performance under higher noise conditions. Higher noise levels would increase the noise information in the latent features from the actual observation sequences, significantly impairing the decoder’s effectiveness. Hence, weights assigned to latent features are set as a nonlinear function of observation sequence noise variance. With noise level increasing, we should reduce the weight of latent features of the actual observation sequence and rely more on the reference latent features. Noise variance is approximated from the difference between actual and reference sequences. The resulting weighted fusion feature $x_0^{fus}$ is subsequently used as the decoder input.

The decoder can be viewed as the inverse process of the encoder, reconstructing observation data from latent features through neural networks. Compared with NODE models that rely solely on the initial value $x_0^{fus}$, the NCDE-based decoder, which takes the influence of the control inputs into consideration during sequence reconstruction, can deal with the sequence reconstruction of a controlled system better. The NCDE network includes multi-channel fully connected layers and activation function layers linked to corresponding control derivative channels to perform element-wise multiplication with control derivatives. Summation across channels produces the NCDE output. The number of channels in the NCDE network is determined by the dimensionality of control variables $c$. Ultimately, the NCDE-generated predictions are mapped via the output layer to yield the desired regular data sequence.

3.2. Multi-Channel Self-Attention Module

The multi-channel self-attention (MCSA) module comprises multiple masked multi-head self-attention modules designed to handle the misalignment and irregularities inherent in multivariate observation sequences. Initially, the observation sequence of each dimension within the coal-fired power plant is expanded to a fixed-length sequence at a desired sampling interval. Consequently, for the irregular observation vector $y_n^{irr}$, certain dimensional observations will be missing. These missing observations can be filled using a constant value $\tau$, represented as follows:

$$y_n^c=y_n^{irr}\odot m+\tau \cdot \stackrel{\text{–}}{m}$$

(3)

where $y_n^c\in {ℝ}^d$ is the observation vector after filling. $m\in {\left\{0,1\right\}}^d$ is an indicator function taking values of either 0 or 1, with 1 denoting the presence of an observation and 0 denoting its absence. $\stackrel{\text{–}}{m}$ is the vector formed by inverting elements of $m$, and $\odot$ represents element-wise multiplication.

Therefore, the irregular observation sequence can be expanded into a regular sequence with the desired length $N^{\prime }+1$, represented as follows:

$$Y=\left(y_0^c,\cdots ,y_n^c,\cdots ,y_{N^{\prime }}^c\right)$$

(4)

where $Y\in {ℝ}^{d\times \left(N^{\prime }+1\right)}$ represents the expanded multivariate sequence containing the filled values $\tau$.

Additionally, due to the non-alignment of multivariate observation sequence in the boiler system, the filled positions of missing values are not entirely consistent. Thus, the number of MCSA self-attention channels should correspond to the dimensionality of observational variable $d$, and the sequence $Y$ must be divided according to variable dimensions (i.e., the rows of $Y$), and input into their corresponding self-attention module channels. Taking observational variables in the boiler system as an example, division by variable dimension can be expressed as

$${\left({\rho }_b,T_{gs},\cdots ,O_{cp}\right)}^{\mathrm{T}}={\mathrm{Block}}_{\mathrm{Row}}\left(Y=\left(y_0^c,\cdots ,y_n^c,\cdots ,y_{N^{\prime }}^c\right)\right)$$

(5)

where ${\rho }_b,T_{gs},O_{cp}\in {ℝ}^{1\times \left(N^{\prime }+1\right)}$ represent the time series with length $N^{\prime }+1$ of gas density, gas temperature, and gas oxygen content. ${\mathrm{Block}}_{\mathrm{Row}}$ represents row-wise partitioning of the matrix. Since the filled positions for each variable are inconsistent, the divided sequences are separately input into corresponding multi-head self-attention modules, represented as follows:

$${\left({\mathit{s}}_1^{\mathbf{\prime }},{\mathit{s}}_2^{\mathbf{\prime }},\cdots ,{\mathit{s}}_d^{\mathbf{\prime }}\right)}^{\mathrm{T}}=\mathrm{MCSA}{\left(M{H}_1\left({\rho }_b\right),M{H}_2\left(T_{gs}\right),\cdots ,M{H}_d\left(O_{cp}\right)\right)}^{\mathrm{T}}$$

(6)

with

$$M{H}_i\left(s_i\right)=W_o^i\left(satt\left(Q_{i,1},K_{i,1},V_{i,1}\right),\cdots ,satt\left(Q_{i,H},K_{i,H},V_{i,H}\right)\right)$$

$$\forall h\in \left\{1,\cdots ,H\right\},Q_{i,h}=W_q^{i,h}{s}_i,K_{i,h}=W_k^{i,h}{s}_i,V_{i,h}=W_v^{i,h}{s}_i$$

where $s_i$ represents the time series of the i-th variable, ${\mathit{s}}_i^{\prime }$ is the output row vector of MCSA, $M{H}_i$ represents the multi-head attention operation of the i-th channel, $W_o^i\in {ℝ}^{1\times H\cdot v}$ is the output projection matrix of the i-th channels and $satt\left(Q_{i,h},K_{i,h},V_{i,h}\right)$ is the self-attention operation of the h-th head of the i-th channel. $W_q^{i,h}\in {ℝ}^{k\times 1}$, $W_k^{i,h}\in {ℝ}^{k\times 1}$, and $W_v^{i,h}\in {ℝ}^{v\times 1}$ are the corresponding projection matrices.

The output results of each self-attention channel are concatenated along the dimension direction and written as $Y^{\prime }=\mathrm{concat}\left({\mathit{s}}_1^{\prime },{\mathit{s}}_2^{\prime },\cdots ,{\mathit{s}}_d^{\prime }\right)$, where $Y^{\prime }\in {ℝ}^{d\times \left(N^{\prime }+1\right)}$ represents the concatenated matrix and $\mathrm{concat}$ represents the concatenation operation.

3.3. Weighted Fusion of Latent Features and Weight Calculation

Sensor data in twin systems can have significant measurement noise covariance due to electromagnetic interference. Existing methods struggle with reconstructing irregular observation sequences with high noise levels. Large measurement noise will heavily contaminate the actual observation data, which means that the real distribution of the measurement information cannot be well learned. An effective means for dealing with this issue is to incorporate reference latent features into the model training and prediction process, and fuse them with latent features from noisy actual observations, thus reducing the impact of noise on reconstruction results.

Therefore, we utilize the prediction results of a high-fidelity digital twin model as a reference observation sequence, and construct a PRN with the same network structure to obtain reference latent features. We can assume that the actual and reference observation sequences are mapped to latent states $x_0^{obs}$ and $x_0^{ref}$, respectively, through identical neural network structures, with their posterior distributions following normal distributions given by

$$\begin{array}{l}{p}_e\left(x_0^{obs}|{\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N\right)=𝒩\left(x_0^{obs}|{\mu }_0^{obs},{\sigma }_0^{2,obs}\right)\\ p_e\left(x_0^{ref}|{\left\{\left(y_n^{ref},t_n\right)\right\}}_{n=0}^N\right)=𝒩\left(x_0^{ref}|{\mu }_0^{ref},{\sigma }_0^{2,ref}\right)\end{array}$$

(7)

with

$$\begin{array}{l}{\mu }_0^{obs}=\mathrm{fnn}\left({ɀ}_0^{obs}\right)\\ {\sigma }_0^{2,obs}=\exp \left(\mathrm{fnn}\left({ɀ}_0^{obs}\right)\right)\\ {\mu }_0^{ref}=\mathrm{fnn}\left({ɀ}_0^{ref}\right)\\ {\sigma }_0^{2,ref}=\exp \left(\mathrm{fnn}\left({ɀ}_0^{ref}\right)\right)\end{array}$$

where ${\mu }_0^{obs}$ and ${\mu }_0^{ref}$ denote the mean values of $x_0^{obs}$ and $x_0^{ref}$, ${\sigma }_0^{2,obs}$ and ${\sigma }_0^{2,ref}$ are the corresponding covariances. ${ɀ}_0^{obs}$ and ${ɀ}_0^{ref}$ are the initial hidden states from two LSTM networks. ${\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N$ and ${\left\{\left(y_n^{ref},t_n\right)\right\}}_{n=0}^N$ represent the actual irregular observation sequence and the reference observation sequence generated by the model, respectively. $\mathrm{fnn}\left(•\right)$ is the feedforward neural network transforming the initial hidden state ${ɀ}_0$ into ${\mu }_0$ and ${\sigma }_0^2$, and $\exp \left(•\right)$ denotes the exponential operation which can ensure the covariance matrix is positive definite.

By using the reparameterization trick, $x_0^{obs}$ and $x_0^{ref}$ can be written as

$$\begin{array}{l}{x}_0^{obs}={\mu }_0^{obs}+{\sigma }_0^{obs}\odot \varsigma \\ x_0^{ref}={\mu }_0^{ref}+{\sigma }_0^{ref}\odot \varsigma \end{array}$$

(8)

where $\varsigma \sim 𝒩\left(\mathbf{0},\mathbf{I}\right)$ is sampled from the standard normal distribution.

Then, we need to determine reasonable weight allocation criteria for the weighted fusion of latent features. By introducing reference latent features, the reconstruction performance under large noise conditions can be improved. Therefore, the noise level of the actual observation sequence can be used for weight allocation. When noise is small, the weight of the actual observation sequence should be increased to capture more of its feature information. Conversely, when noise is large, the weight of the reference latent features should be increased to avoid introducing excessive measurement noise. Assuming the measurement noise follows a Gaussian distribution with mean $\mathbf{0}$ and covariance $R$, we estimate the noise covariance using the reference observation sequence:

$$\widehat{R}={\displaystyle \frac{1}{N}}{\displaystyle {\sum }_{n=0}^N\left({\tilde{y}}_n^{irr}-{\tilde{y}}_n^{ref}\right){\left({\tilde{y}}_n^{irr}-{\tilde{y}}_n^{ref}\right)}^T}$$

(9)

where ${\tilde{y}}_n^{irr}$ and ${\tilde{y}}_n^{ref}$ are the normalized actual and reference observations.

Let ${\widehat{R}}_{\max }=\max \left\{\mathrm{diag}\left(\widehat{R}\right)\right\}$ be the largest diagonal element of $\widehat{R}$. The reference feature weight is then calculated by

$$\left\{\begin{array}{cc}\begin{array}{l}Q=s_1\cdot {\widehat{R}}_{\max }\\ Q=s_2\cdot {\widehat{R}}_{\max }+c\\ Q=q_2\end{array}& \begin{array}{l}{\widehat{R}}_{\max }<r_1\\ r_1\le {\widehat{R}}_{\max }\le r_2\\ {\widehat{R}}_{\max }>r_2\end{array}\end{array}\right.$$

(10)

where $s_1=q_1/r_1$, $s_2=\left(q_2-q_1\right)/\left(r_2-r_1\right)$, $c=q_1-s_2\cdot r_1$. $r_1$, $r_2$, $q_1$ and $q_2$ are the threshold parameters needed to be set properly. If the noise level is smaller than $r_1$, there is no need to increase the weight of the reference observation feature and the model will primarily utilize the actual observation features to realize sequence reconstruction. While if the noise level is larger than $r_1$, the weight of the reference feature should be increased so that the negative impact of noise on reconstruction accuracy could be reduced. In addition, in order to guarantee that the fused feature always contains some feature information of the actual data, the maximum reference feature weight is set to $q_2<1$.

Finally, the fused feature can be expressed as follows:

$$x_0^{fus}=\left(1-Q\right)\cdot x_0^{obs}+Q\cdot x_0^{ref}$$

(11)

3.4. NCDE-Based Decoder

The decoder predicts the desired regular data sequence from the fused initial latent feature $x_0^{fus}$. Generally, most real systems are controlled, e.g., the coal-fired power plants. However, the prediction results of the NODE method depend only on the initial hidden state without considering the impact of control inputs on state changes. The NCDE proposed by Kidger et al. [28] takes the impact of control inputs into consideration during the prediction process. The solution of the NCDE is defined as follows:

$$\begin{array}{cc}\hfill x_t& =x_0+{\displaystyle {\int }_{t_0}^t{f}_{cd}\left(x_w,w;W_{cd}\right)\mathrm{d}{U}_w}\hfill \\ & =x_0+{\displaystyle {\int }_{t_0}^t{f}_{cd}\left(x_w,w;W_{cd}\right)\frac{\mathrm{d}{U}_w\left(w\right)}{\mathrm{d}w}\mathrm{d}w}\hfill \end{array}$$

(12)

where $x_t\in {ℝ}^r$ is the hidden state at time $t\in \left(t_0,t_{N^{\prime }}\right]$, $U_w:\left[t_0,t_{N^{\prime }}\right]\to {ℝ}^c$ is a thrice-spline curve based on the control sequence ${\left\{u_n^r\right\}}_{n=0}^{N^{\prime }}$ over the interval $\left[t_0,t_{N^{\prime }}\right]$, and $f_{cd}\left(x_t,t;W_{cd}\right):{ℝ}^r\to {ℝ}^{r\times c}$ is a neural network with learnable parameters $W_{cd}$.

Consider a generative model defined by the NCDE network with the initial latent feature $x_0^{fus}$ and desired sampling times ${\left\{t_n^{\prime }\right\}}_{n=0}^{N^{\prime }}$. The initial latent feature $x_0^{fus}$ follows a normal distribution given by

$$x_0^{fus}\sim p_e\left(x_0^{fus}\right)=𝒩\left(x_0^{fus}|{\mu }_0^{fus},{\sigma }_0^{2,fus}\right)$$

(13)

where

$$\begin{array}{l}{\mu }_0^{fus}=\left(1-Q\right)\cdot {\mu }_0^{obs}+Q\cdot {\mu }_0^{ref}\\ {\sigma }_0^{2,fus}={\left(1-Q\right)}^2\cdot {\sigma }_0^{2,obs}+Q^2\cdot {\sigma }_0^{2,ref}\end{array}$$

Using Formula (12), we can compute the latent features at all desired time points. The neural network $f_{cd}$ consists of multiple channels of multilayer feedforward networks, with the number of channels determined by the system’s control dimension $c$. Taking the boiler system as an example, since it has two control inputs (coal feed and secondary air flow), $f_{cd}$ has two channels as follows:

$$\left(h_1,h_2\right)=f_{cd}\left(x_t,t;W_{cd}\right)=\mathrm{mlfnn}\left(x_t,x_t\right)$$

(14)

where $\mathrm{mlfnn}\left(•\right)$ denotes a multilayer feedforward network cascaded with fully connected layers and LeakyReLU activation functions, and $h_1,h_2\in {ℝ}^{r\times 1}$ are the outputs of the $\mathrm{mlfnn}\left(•\right)$ for each channel. Multiplying $h_1$ and $h_2$ with their respective control derivatives and summing the results yields the latent feature of the NCDE network at the desired time $t_{n+1}^{\prime }$ given by

$${\left.x_t\right|}_{t=t_{n+1}^{\prime }}=h_1\cdot {\left.{\displaystyle \frac{d{U}^1\left(t\right)}{dt}}\right|}_{t=t_n^{\prime }}+h_2\cdot {\left.{\displaystyle \frac{d{U}^2\left(t\right)}{dt}}\right|}_{t=t_n^{\prime }}$$

(15)

where $U^1$ and $U^2$ are the cubic spline curves formed by the sequences of control input 1 (coal feed) and control input 2 (secondary air flow), respectively.

After obtaining the latent features at all desired time points, the output layer generates the desired observation sequence as follows:

$$\left(y_0^r,\cdots ,y_{N^{\prime }}^r\right)=\mathrm{fnn}\left(\left(x_0,\cdots ,x_{N^{\prime }}\right)\right)$$

(16)

where $y_n^r\sim p_d\left(y_n^r|x_n\right),n=0,1,\cdots ,N^{\prime }$. $\left(y_0^r,\cdots ,y_{N^{\prime }}^r\right)$ is the desired regular observation sequence generated from the latent feature sequence $\left(x_0,\cdots ,x_{N^{\prime }}\right)$ at sampling time $\left(t_0^{\prime },\cdots ,t_{N^{\prime }}^{\prime }\right)$.

3.5. Weighted Loss Function of the PRN-NCDE Model

Through reparameterization, parameters can be learned by using the gradient descent method. According to [30], the objective loss function in Equation (2) can be simplified as

$$\begin{array}{l}𝒥\left(W_{ec},W_{dc}|{\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N\right)\\ ={\displaystyle {\sum }_{n=0}^N-\frac{1}{2}{‖y_n^{irr}-{\widehat{y}}_n^{irr}‖}^2-\frac{1}{2}\lambda \left(tr\left({\sigma }_0^2\right)+{\mu }_0^T{\mu }_0-n-\log \left(\left|{\sigma }_0^2\right|\right)\right)}\end{array}$$

(17)

where $y_n^{irr}$ and ${\widehat{y}}_n^{irr}$ are the encoder input and decoder output at the corresponding time, $\lambda$ is a hyperparameter controlling the variance and $n$ is the dimension of the latent feature $x_0$. ${\mu }_0$ and ${\sigma }_0^2$ are the mean and covariance of $x_0$, respectively. $tr\left(\cdot \right)$ denotes the trace operator.

This objective function measures the reconstruction accuracy of the decoder output ${\widehat{y}}_n^{irr}$ relative to the encoder input $y_n^{irr}$. When the actual observation sequence ${\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N$ is not corrupted by noise or the noise level is low, this function enables the model to learn the data distribution effectively. However, if the actual observational data contain high levels of noise, achieving the desired training performance using this objective function is difficult.

To address this challenge, we propose a weighted loss function for the PRN-NCDE model, which is defined as follows:

$$\begin{array}{l}𝒥\left(W_{ec},W_{dc}|{\left\{\left(y_n^{irr},t_n\right)\right\}}_{n=0}^N,{\left\{\left(y_n^{ref},t_n\right)\right\}}_{n=0}^N\right)\\ ={\displaystyle {\sum }_{n=0}^N-\frac{1}{2}{‖y_n^{irr}-{\widehat{y}}_n^{irr}‖}_{\left(1-Q\right)}^2}-{\displaystyle \frac{1}{2}}{‖y_n^{ref}-{\widehat{y}}_n^{irr}‖}_Q^2\\ -{\displaystyle \frac{\lambda }{2}}\left(tr\left({\sigma }_0^{2,fus}\right)+{\left({\mu }_0^{fus}\right)}^T{\mu }_0^{fus}-n-\log \left(\left|{\sigma }_0^{2,fus}\right|\right)\right)\end{array}$$

(18)

where $Q$ is the reference latent feature weight calculated in Section 3.3.

Equation (18) integrates data from the reference observation sequence. It uses the reference feature weight, which is calculated based on the noise level in the actual observation data, to determine the importance of the actual and reference data distributions for parameter training. When noise is low, the loss function will rely more on the actual observation data for training. Conversely, the reference observation data will be used to mitigate the negative impact of noise on model learning.

4. Simulation Experiments and Analysis

This section selects the furnace of the boiler system as the test object to evaluate the reconstruction performance of the developed model. We use the nominal data of a 600 MW coal-fired power plant under 100% rated conditions to train the sequence reconstruction model and validate its effectiveness. Under the rated conditions, it is required that the coal flow rate is about 74.5 kg/s and the excess air coefficient is 1.2; thus, the gas oxygen content in the boiler system can be maintained around 3.2% and the boiler system can operate stably. The reference observation sequence is generated by the digital twin model of the furnace, which is defined by the following equations [31,32,33]:

$$\left\{\begin{array}{l}x\left(k+1\right)=Ax\left(k\right)+Bu\left(k\right)\\ y\left(k+1\right)=Cx\left(k+1\right)\end{array}\right.$$

(19)

where $x$, $u$, and $y$ represent the system’s state, input, and observation vectors, respectively, and they are defined as follows:

$$\begin{array}{l}x={\left[\begin{array}{cccc}{V}_{gs}& {\rho }_b& T_{gs}& O_{cp}\end{array}\right]}^{\mathrm{T}}\\ u={\left[\begin{array}{cc}{W}_{cf}& V_{sa}\end{array}\right]}^{\mathrm{T}}\\ y={\left[\begin{array}{cccc}{V}_{gs}& T_{gs}& P_b& O_{cp}\end{array}\right]}^{\mathrm{T}}\end{array}$$

The matrices $A$, $B$, and $C$ are given by:

$$A=\left[\begin{array}{cccc}{a}_{11}& 0& 0& 0\\ 0& a_{22}& 0& 0\\ 0& 0& a_{33}& 0\\ a_{41}& 0& 0& a_{44}\end{array}\right] B=\left[\begin{array}{cc}0& b_{12}\\ b_{21}& b_{22}\\ b_{31}& b_{32}\\ 0& 0\end{array}\right] C=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 0& 1& 0\\ 0& c_{32}& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

where

$$\left\{\begin{array}{l}{a}_{11}=1-T_a\\ a_{22}=1-V_{gs}/V_b\\ a_{33}=1-{\displaystyle \frac{V_{gs}}{V_b}}-K_{sl}{\displaystyle \frac{\left(T_{gs}^4-T_{sv}^4\right)}{\left(C_{gs}{\rho }_b{V}_b{T}_{gs}\right)}}\\ a_{41}=21\left(V_{sa}-V_0\cdot W_{cf}\right)/\left(V_{sa}\cdot V_b\right)\\ a_{44}=1-V_{gs}/V_b\end{array}\right. \left\{\begin{array}{l}{b}_{12}=T_a\\ b_{21}=1/V_b\\ b_{22}={\rho }_a/V_b\\ b_{31}=Q_{net,ar}/\left(C_{gs}{\rho }_b{V}_b\right)\\ b_{32}=H_a{\rho }_a/\left(C_{gs}{\rho }_b{V}_b\right)\end{array}\right. c_{32}=R_{gs}{T}_{gs}$$

The physical meanings of the parameters and the value ranges of the input and output quantities are shown in

Table 1. Physical meaning of furnace parameters.

Parameter		Unit	Physical Meaning	Value
Input and Output Parameters	$W_{cf}$	$kg/s$	Coal flow rate	[54, 80]
	$V_{sa}$	${\mathrm{m}}^3/\mathrm{s}$	Secondary air flow rate	[400, 600]
	$V_{gs}$	${\mathrm{m}}^3/\mathrm{s}$	Gas flow rate	[400, 650]
	${\rho }_b$	${\mathrm{kg}/\mathrm{m}}^3$	Gas density	[1.3, 1.4]
	$T_{gs}$	$°\mathrm{C}$	Gas temperature in furnace	[1000, 1100]
	$P_b$	$\mathrm{kpa}$	Furnace pressure	[101, 102]
	$O_{cp}$	$\%$	Gas oxygen content	[2.6, 3.8]
Constant Parameters	$T_a$	/	Inertial time constant	0.9
	$C_{gs}$	$\mathrm{kJ}/\left(\mathrm{kg}\cdot °\mathrm{C}\right)$	Specific heat capacity of gas	2.03
	$H_a$	$\mathrm{kJ}/\mathrm{kg}$	Air enthalpy	934.1
	$V_b$	${\mathrm{m}}^3$	Furnace volume	16,855
	$V_0$	${\mathrm{m}}^3/\mathrm{kg}$	Theoretical air consumption	5.834
	$T_{sv}$	$°\mathrm{C}$	Temperature of water wall	326.66
	${\rho }_a$	${\mathrm{kg}/\mathrm{m}}^3$	Air density	1.273
	$R_{gs}$	/	Universal gas constant	$6.52\times {10}^{-2}$

. To validate the model’s performance, we compare the proposed model with the RNN-NODE method from [22] and the improved RNN-NCDE method. The network architecture parameters for each model are listed in

Table 2. Network architecture parameters of each reconstruction model.

Sequence Reconstruction Model	Network Architecture Parameters
PRN-NCDE	(1) MCSA module: number of channels $d=4$; number of attention heads $M=4$; fill value $\tau =1\times {10}^{-7}$; (2) Number of LSTM network neurons: 80; (3) Number of encoder fully connected layer neurons: 100; (4) Latent feature dimension n = 10; (5) NCDE module (two-channel feedforward network): [100]-[100]-[100]-[10].
RNN-NCDE	(1) Number of LSTM network neurons: 80; (2) Number of encoder fully connected layer neurons: 100; (3) Latent feature dimension n = 10; (4) NCDE module (two-channel feedforward network): [100]-[100]-[100]-[10].
RNN-NODE	(1) Number of LSTM network neurons: 80; (2) Number of encoder fully connected layer neurons: 100; (3) Latent feature dimension n = 10; (4) NCDE module:100-100-100-10.

.

The model training process uses the following settings: number of iterations—200; initial learning rate—$2\times {10}^{-3}$; minimum learning rate—$2\times {10}^{-4}$; decay rate—0.999; hyperparameter—$\lambda =0.005$. The thresholds for the reference latent feature weights are set as $r_1=5\times {10}^{-4}$, $r_2=5\times {10}^{-3}$, $q_1=5\times {10}^{-6}$, and $q_2=0.9$. The desired regular sequence length is set to $N^{\prime }=500$, with a sampling interval of $1s$. The control input $u$ is normalized to the range $\left[0,1\right]$, and the observation $y$ is normalized to the range $\left[-1,1\right]$. The experimental analysis focuses on two aspects: (1) performance under different sampling numbers; (2) performance under high noise levels.

The relevant information about the parameters of the experimental environment, including CPU, GPU, memory, Matlab and deep learning toolbox version, are detailed in

Table 3. Experimental environment configuration.

Hardware and Software	Details
CPU	Intel(R) Core(TM) i9-10900 CPU @ 2.8 GHz
GPU	NVIDIA GeForce GTX 1660 SUPER
Random Access Memory	32 GB
Matlab Version	2024a
Deep Learning Toolbox	24.1

.

4.1. Analysis of Model Reconstruction Results Under Different Sampling Numbers

The dataset for model training and validation consists of 150 irregular observation sequences, including 5 different random sampling sequences with a sampling number of $N$ under 30 different working conditions. This section compares the reconstruction results of models under sampling numbers of $N=60$, $N=40$, and $N=20$. The noise covariance of the actual observation sequence is set to

$$\begin{array}{cc}\hfill R=& \mathrm{diag}\left(\begin{array}{cccc}{R}_1& R_2& R_3& R_4\end{array}\right)\hfill \\ \hfill =& \mathrm{diag}\left(\begin{array}{cccc}5\times {10}^{-4}& 1\times {10}^{-2}& 2\times {10}^{-2}& 5\times {10}^{-3}\end{array}\right)\hfill \end{array}$$

Figure 3 and Figure 4 show the reconstruction results of each model for the normalized values of each observation variable under different sampling numbers. Since RNN-NODE does not consider the impact of control inputs on outputs, its predictions rely solely on the initial latent state, resulting in significant estimation errors. After improvement with NCDE, the model can account for control sequences, enhancing reconstruction accuracy. As shown in

Table 4. RMSE results of different models under different sampling numbers.

Sequence Reconstruction Model	${\mathbf{RMSE}}_{\mathit{N}=60}$	${\mathbf{RMSE}}_{\mathit{N}=40}$	${\mathbf{RMSE}}_{\mathit{N}=20}$
PRN-NCDE	0.0213	0.0295	0.0322
RNN-NCDE	0.0463	0.0711	0.0888
RNN-NODE	0.1375	0.2069	0.1910

, RNN-NCDE reduces the estimation error by 66.3%, 65.6%, and 53.5% compared to RNN-NODE when the sampling numbers are 60, 40, and 20, respectively. Although RNN-NCDE improves the reconstruction results of RNN-NODE, its performance degrades noticeably under sparse measurements, as evidenced by a near doubling of prediction errors when the sampling number decreases from 60 to 20.

The proposed PRN-NCDE model, by focusing on the correlation between sampling points and their position information, not only further improves fitting accuracy under each sampling number but also mitigates the impact of sparse measurements on reconstruction results. The RMSE results indicate that PRN-NCDE enhances estimation accuracy by 54%, 58.5%, and 63.7% compared to RNN-NCDE when the sampling numbers are 60, 40, and 20, respectively. Moreover, when the sampling number decreases from 60 to 20, PRN-NCDE’s prediction error increases by only 50%, remaining superior to the other two methods. Therefore, the PRN-NCDE model is more effective in handling irregular sequence reconstruction for controlled systems and can significantly reduce the impact of sparse measurements on reconstruction results.

4.2. Reconstruction Performance Analysis Under High Noise Levels

Similarly, a dataset of 150 irregular sequences with three noise levels is used to compare reconstruction performance. The sampling number for each noise level is set to $N=60$, with noise covariances as follows:

• Noise Level 1:

$$R^1=\mathrm{diag}\begin{array}{cccc}(1\times {10}^{-3}& 0.05& 0.1& 0.02)\end{array}$$

• Noise Level 2:

$$R^2=\mathrm{diag}\begin{array}{cccc}(2\times {10}^{-3}& 0.1& 0.2& 0.04)\end{array}$$

• Noise Level 3:

$$R^3=\mathrm{diag}(\begin{array}{cccc}3\times {10}^{-3}& 0.15& 0.3& 0.06)\end{array}$$

Figure 5 and Figure 6 show the reconstruction results of each model under high noise levels. Both RNN-NCDE and RNN-NODE fail to effectively recover the true values from noisy sequences, with significant drops in accuracy.

The results in

Table 5. RMSE results of different models under high noise levels.

Sequence Reconstruction Model	${\mathbf{RMSE}}_{{\mathit{R}}^1}$	${\mathbf{RMSE}}_{{\mathit{R}}^2}$	${\mathbf{RMSE}}_{{\mathit{R}}^3}$
PRN-NCDE	0.0158	0.0155	0.0182
RNN-NCDE	0.0599	0.0581	0.0771
RNN-NODE	0.0471	0.0479	0.0529

show that RNN-NCDE has larger fitting errors than RNN-NODE under all noise levels, indicating that NCDE does not improve fitting accuracy in noisy conditions. As can be seen from the above, these two methods cannot handle the sequence reconstruction problem under large noise levels effectively.

The proposed PRN-NCDE model uses a parallel reference network to obtain reference latent features and calculates weights based on the noise level of the actual observation sequence. This weighted fusion of latent features improves reconstruction accuracy under high noise levels. The results show that PRN-NCDE can accurately recover the true value trends, with prediction errors reduced by over 70% and 60% compared to RNN-NCDE and RNN-NODE, respectively.

4.3. Discussion

For a well-trained sequence reconstruction model, the parameters related to the latent feature weight, including $r_1$, $r_2$, $q_1$ and $q_2$, will decide the reconstruction results. For the simulation experiments in this paper, the value of $r_1$ is set based on the largest measurement noise variance. While guaranteeing the noise in a reasonable range, the features of the real observation data are more utilized for sequence reconstruction. If the noise level is smaller than $r_1$, there is no need to increase the weight of the reference observation feature and the model will primarily utilize the actual observation features to realize sequence reconstruction. While if the noise level is larger than $r_1$ due to electromagnetic interference or sensor failure, the weight of the reference feature should be increased so that the negative impact of noise on reconstruction accuracy could be reduced. In addition, in order to guarantee that the fused feature always contains some feature information of the actual data, the maximum reference feature weight is set to $q_2<1$.

In conclusion, for the reconstruction of noisy, irregular multivariate sequences in controlled systems, the PRN-NCDE model outperforms existing methods. Simulation results confirm its effectiveness in improving reconstruction accuracy under sparse measurements and high noise levels, making it suitable for digital twin systems in complex coal-fired power plant environments.

5. Conclusions

The uncertainty and randomness in network communication and the non-synchronous sampling of sensors can cause irregularities, sparsity, and misalignment in measurement data. Existing methods struggle to achieve ideal reconstruction under sparse measurements and high noise, making them unsuitable for sequence reconstruction tasks in digital twin systems. To address these issues, this article establishes a variational autoencoder model based on parallel reference networks and neural controlled differential equations, which can effectively handle the reconstruction of noisy, irregular multivariate sequences in digital twin systems.

Firstly, a multi-channel self-attention module is incorporated into the encoder. This module not only analyzes the position information of sampled data in the sequence and focuses on the correlation between data to enhance reconstruction accuracy under sparse measurements but also handles the misalignment and irregularity of observation sequences. Secondly, a parallel reference network is established. The reference sequence provided by the digital twin model is input and mapped to reference latent features, which are then fused with the latent features of the actual observation sequence in a weighted manner to address the reconstruction of observation sequences under high noise conditions. Thirdly, a decoder is constructed using an NCDE network, which takes into account the effect of control inputs on outputs to improve the sequence reconstruction performance of controlled systems. Finally, a weighted loss function for training the PRN- NCDE model is formed based on the calculated feature weights, which helps to better train the network parameters of the model.

Simulation results show that the proposed PRN-NCDE model improves the estimation accuracy by more than 50% and 70%, respectively, compared with RNN-NCDE under different sampling numbers and noise levels, and by more than 80% and 60%, respectively, compared with RNN-NODE. It can recover the changing trend of observation sequence more accurately. Therefore, the proposed method can effectively improve the reconstruction accuracy of observation sequences under sparse measurements and large noise conditions, and is suitable for irregular sequence reconstruction tasks in digital twin systems with uncertain network transmission and non-synchronous sensor sampling.

In future work, three aspects could be further investigated. First, the structure of the deep learning network can be improved to realize better abilities of feature extraction and prediction accuracy, especially under small sampling numbers and high noise levels. Second, the situation when outliers exist in the measurements should be considered to enhance the robustness of the reconstruction method. Third, we assume that the measurement noise follows Gaussian distribution. Non-Gaussian distribution should be considered for modelling measurement noise and the reconstruction performance under this situation could be another interesting research direction.