TiTAD: Time-Invariant Transformer for Multivariate Time Series Anomaly Detection

Abstract

Anomaly detection in multivariate time series data is critical for industrial sectors such as manufacturing and aerospace. While existing methods have achieved notable success in specific scenarios, they often narrowly focus on either the temporal or spatial dimensions while overlooking their complex interdependencies. Furthermore, these approaches tend to neglect the time-invariant characteristics that are crucial for accurately capturing the spatio-temporal dynamics of the time series. To address these limitations, this paper introduces the Time-invariant Transformer for Multivariate Time Series Anomaly Detection (TiTAD), a novel framework that synergizes temporal invariance with spatio-temporal modeling. TiTAD leverages the Time-invariant Transformer, a component that excels at extracting both spatio-temporal and time-invariant features by incorporating an augmented memory mechanism. This mechanism enhances anomaly identification robustness through synergistic integration of heterogeneous feature sets. Additionally, TiTAD mitigates the Transformer’s tendency to lose temporal sequence information through the use of the Gated Recurrent Unit (GRU), thereby further enhancing the model’s capability to discern spatio-temporal patterns. The inclusion of a Feature Fusion module within TiTAD serves to refine the extracted features by adjusting their weights and minimizing redundancy, ensuring that the most relevant information is utilized for prediction and anomaly detection. Empirical evaluation on three industrial-scale benchmarks (SWaT, WADI, and SMD) demonstrates TiTAD’s superior performance compared to other methods.

1. Introduction

With the advancement of sensor technology, sensor deployment in cyber–physical systems—including vehicles, water treatment plants, and space probes—is experiencing rapid growth. These sensors generate vast amounts of multivariate time series data while continuously monitoring the operational health of these systems [1]. The effective utilization of these time series data has become increasingly important for ensuring the seamless operation of various facilities, particularly critical infrastructures such as transportation, power, and communications [2,3,4,5]. The primary focus is on time series anomaly detection, which aims to identify and alert security teams to anomalies within these multivariate time series datasets. By doing so, potential irregularities can be addressed promptly, thereby minimizing potential risks and losses.

Multivariate time series exhibit dynamic characteristics over distinct time periods, encompassing spatial features such as evolving variable correlations and temporal features like inherent noise fluctuations [6]. Existing approaches [7,8,9,10,11,12,13,14] predominantly leverage Recurrent Neural Networks (RNNs) to capture the temporal features of multivariate time series. Another major category of methods [15,16,17,18,19] adopts Convolutional Neural Networks (CNNs) or Graph Neural Networks (GNNs) to capture the spatial features among variables within multivariate time series. Multivariate time series data generated by real-world systems exhibit complex patterns and large volumes, making the process of labeling anomalies costly [20,21]. Consequently, the field of anomaly detection predominantly relies on unsupervised methods [22]. Unsupervised methods typically employ prediction or reconstruction models, which improve the accuracy of predicting or reconstructing the time series to learn the standard patterns during the training process [23,24]. Since anomalous data patterns deviate from the standard patterns, the models generate higher bias in predicting or reconstructing anomalies, allowing these anomalies to be inferred from these biases.

While these methods have demonstrated remarkable performance in anomaly detection, they predominantly concentrate on either temporal or spatial dimensions of the data, failing to adequately capture the complex spatio-temporal interactions inherent in multivariate time series. Such data naturally embody nonlinear spatial dependencies between variables and intricate temporal dependencies within variables. The unilateral modeling approach inevitably leads to incomplete pattern learning, causing failures in capturing either spatial structural anomalies or temporal contextual deviations, thereby constraining detection capabilities [25]. Moreover, these methods often overlook the importance of time-invariant properties when extracting spatio-temporal features. Notably, multivariate time series preserve essential time-invariant attributes including stable inter-variable correlations and persistent periodic patterns or trend components. Neglecting these fundamental characteristics may severely compromise model effectiveness. In this work, ”invariant” refers to the robustness of our model to specific transformations (e.g., temporal shifts or scaling) within the multivariate time series. This does not assume stationarity of the underlying process, as trends, cycles, or seasonalities may still exist. Following [1], we define anomalies as deviations from invariant relational patterns across variables, rather than static statistical properties.

Recent advances have demonstrated the successful application of Transformers to multivariate time series analysis, harnessing their exceptional global representation and long-range relationship modeling capabilities [26]. These architectures have shown the capacity to extract time-invariant features across both spatial and temporal dimensions. The latest methods leverage Transformers to decode temporal patterns within individual variables, achieving significantly improved performance [12,13]. However, current methods fail to explicitly model the time-invariant characteristics of time series, limiting the full utilization of these properties. Additionally, Transformers tend to lose the temporal order that is critical to time series data, which further hinders their ability to extract intrinsic temporal features of each variable [27,28].

To address these limitations, this paper proposes the Time-invariant Transformer for Multivariate Time Series Anomaly Detection (TiTAD). TiTAD harnesses a Time-invariant Transformer to holistically model the spatio-temporal dependencies in multivariate time series. We capitalize on the Transformer’s dual strengths in long-range dependency modeling and relational representation to extract spatio-temporal features. Meanwhile, TiTAD integrates the Gated Recurrent Unit (GRU) [29] to counteract the Transformer’s temporal order degradation, thereby refining spatio-temporal feature extraction. The Time-invariant Transformer further incorporates an augmented memory mechanism comprising learnable embedding vectors and an attention module. During training, embeddings are continuously optimized to fit the time-invariant characteristics per variable, while the attention mechanism precisely quantifies inter-variable relationships. Through the coordinated operation of these components, the mechanism discovers latent time-invariant attributes in the data. TiTAD strategically combines time-invariant and spatio-temporal features to amplify anomaly discrimination. Crucially, time-invariant features achieve superior prediction fidelity for normal data with stable patterns, while failing to generalize anomalous instances with irregular patterns. This amplifies the disparity between normal and abnormal prediction errors, sharpening anomaly detectability. The architecture culminates in a Feature Fusion (FF) module employing NeXtVLAD [30], which dynamically recalibrates time-invariant feature weights while eliminating redundancy. Fused features are processed through transformation layers for prediction-based anomaly inference.

The main contributions of this study are summarized below:

• We propose the Time-invariant Transformer, which includes an augmented memory mechanism to extract spatio-temporal features inherent in time series. Specifically, it integrates the spatio-temporal features extracted by the Transformer and the time-invariant features obtained through the augmented memory mechanism. The different effects of time-invariant features on the prediction of anomalous and normal data can highlight the distinguishability of anomalous data, thereby improving the performance of anomaly detection.
• We utilize GRU, which focuses on extracting intra-variable dynamic features from a temporal perspective, to compensate for the Transformer’s loss of temporal order. This ensures that the model retains the critical temporal information necessary for accurate anomaly detection.
• We propose a novel Feature Fusion module with adaptive gating mechanisms that dynamically balances spatio-temporal and time-invariant feature contributions. This module helps balance the contributions of spatio-temporal and time-invariant features, ensuring that the model’s overall performance is robust and reliable.
• We conduct extensive empirical research on three widely used datasets from real-world sources, demonstrating the superiority of TiTAD. Our experiments also showcase the model’s capability in anomaly diagnosis, highlighting its practical value in real-world applications.

The remainder of this paper is organized as follows: Section 2 reviews related work in multivariate time series anomaly detection. Section 3 details the proposed TiTAD framework, including the Time-invariant Transformer architecture, enhanced temporal modeling, and Feature Fusion module. Section 4 presents experimental results on three industrial datasets, with comparisons to state-of-the-art baselines. Section 5 discusses the implications and limitations of our approach. Finally, Section 6 concludes the paper.

2. Related Work

Anomalies exhibit multifaceted data patterns, whereas normal data demonstrate relatively stable characteristics. Therefore, existing anomaly detection methods often learn standard patterns from normal data and then perform prediction or reconstruction from learned standard patterns, using the prediction or reconstruction error as an important basis for anomaly inference. These methods are thus categorized into prediction-based and reconstruction-based models.

Prediction Methods: LSTM-NDT [7] uses Long Short-Term Memory (LSTM) and nonparametric dynamic thresholding to achieve superior prediction performance while preserving interpretability in the system. When model forecasts are obtained, it also provides an unsupervised thresholding method to evaluate residuals. Deep Autoencoding Gaussian Mixture Model (DAGMM) [10] consists of an estimation network and a compression network. The compression network includes an autoencoder for feature extraction of the input data, while the estimation network uses a Gaussian Mixture Model to predict the next data point and uses the prediction error as an important basis for anomaly inference. The Graph Deviation Network (GDN) [17] is a prediction model that integrates graph structure learning and Graph Neural Networks (GNNs) to extract spatial features and generate anomaly scores using prediction errors.

Reconstruction Methods: UnSupervised Anomaly Detection (USAD) [31] leverages adversarial autoencoders for unsupervised learning, enabling efficient anomaly identification through adversarial training. OmniAnomaly [9] employs stochastic variable connections and planar normalization flow to characterize normal time series patterns, detecting anomalies via reconstruction probabilities. It detects anomalies using reconstruction probabilities. LSTM-VAE [32] integrates LSTM and the Variational Autoencoder (VAE) by substituting the VAE’s linear layer with LSTM to better capture the temporal dynamics. The Multi-Scale Convolutional Recurrent Encoder–Decoder (MSCRED) [33] creates multi-scale signature matrices representing temporal states, using convolutional encoders to encode inter-sensor correlations and ConvLSTM networks to model temporal dependencies. While advancing the field, these methods lack the capacity for comprehensive modeling of temporal feature interactions and neglect the intrinsic time-invariant properties of time series, ultimately limiting anomaly detection efficacy.

Transformer Models: AnomalyTransformer [12] adopts Transformer architectures for temporal feature extraction. It computes association discrepancy via the Transformer’s attention weights and combines this metric with reconstruction errors to derive anomaly scores. TranAD [13] establishes a Transformer-based framework for anomaly detection and diagnosis, enabling few-shot learning through model-agnostic meta-learning. GTA [11] integrates structure learning with graph convolution and Transformer-based temporal modeling. Its graph structure learning employs Gumbel softmax sampling to learn bidirectional sensors while developing influence propagation convolution to model anomaly information flow between nodes. Though achieving progress, these methods exhibit limitations in spatio-temporal feature extraction and fail to explicitly model time-invariant time series properties. Additionally, Transformers suffer from temporal order degradation in sequential data.

Prior works assume temporal stationarity or overlook persistent patterns. TiTAD explicitly disentangles invariant contextual features (learned via contrastive phase alignment) from time-variant residuals. Most methods prioritize either spatial or temporal features. TiTAD introduces attention gates that dynamically weight spatial and temporal embeddings based on their anomaly saliency. The Transformers’ permutation invariance weakens sequential causality. Our TiTAD enforces order-aware attention distributions, preserving causal dependencies critical for anomaly localization. By unifying these innovations, TiTAD achieves outstanding performance while addressing foundational gaps in anomaly detection.

3. Method

The multivariate time series consists of the observations of multiple variables over some time, denoted by $S\in {\mathbb{R}}^{T\times N}$, where T represents the time length and N represents the number of variables. Specifically, $S=\left[o_1,o_2,\dots ,o_T\right]$, where $t\in \left\{1,2,\dots ,T\right\}$ and $o_t$ represents the observations of N variables at time point t: $o_t=\left[v_1,v_2,\dots ,v_i\right]$, with $i\in \left\{1,2,\dots ,N\right\}$ and $v_i$ representing the observations of the ith variable at time point t. Given that multivariate time series can be long, we usually divide them into sliding windows as inputs to the model.

In the foundational challenges of time series anomaly detection, there are three critical pain points: (1) temporal dependency modeling, where the GRU layer is tailored to capture short-term local patterns (e.g., sensor drifts) while the Transformer attends to global context (e.g., diurnal cycles), explicitly addressing the multi-scale nature of industrial time series data; (2) non-stationarity mitigation, achieved through adversarial training to disentangle time-invariant features (e.g., equipment degradation signatures) from transient noise, as motivated by the failure of traditional thresholding under concept drift; and (3) anomaly sparsity and temporal locality, handled via the hybrid loss function that prioritizes reconstruction fidelity around abrupt deviations. We further contextualized these choices with domain-specific temporal constraints, such as the use of exponential smoothing in preprocessing to suppress high-frequency instrumentation noise without blurring step-change anomalies, and the dynamic thresholding mechanism that adapts to seasonal baseline shifts. Anomaly detection in multivariate time series focuses on continuously monitoring the time series and promptly detecting and reporting anomalous events.

Given a multivariate time series $S\in {\mathbb{R}}^{T\times N}$, it outputs a set of anomaly labels denoted by $L=\left\{l_1,l_2,\dots ,l_t\right\}$, with $t\in \left\{1,2,\dots ,T\right\}$ and $l_t\in \left\{0,1\right\}$, where 0 means that S is not anomalous in the time point t and 1 means that S is anomalous in the time point t. Due to the complexity and variability of anomalies, which differ from normal data patterns, the model generates higher prediction deviations for anomalies. The prediction deviations are further amplified by the distinct effects of time-invariant features on normal versus anomalous data [34]. The model is based on a fixed window length w of historical data $x_t=\left[o_{t-w+1},o_{t-w+2},\dots ,o_{t-1}\right]$, where $o_t\in {\mathbb{R}}^{1\times N}$ to predict the observation $\widetilde{x_t}\in {\mathbb{R}}^N$ in the time point t.

This study proposes TiTAD, a novel prediction framework that capitalizes on the Transformer’s attention mechanism to explicitly model complex variable interactions and efficiently extract spatio-temporal features. A critical innovation lies in the Transformer’s augmented memory mechanism, specifically engineered to learn time-invariant characteristics. The integration of GRUs preserves temporal order information, thereby enhancing intrinsic temporal feature extraction. TiTAD addresses feature redundancy by incorporating a Feature Fusion module that dynamically calibrates time-invariant feature weights. The FF module’s output undergoes linear transformation for temporal prediction, while the framework culminates in anomaly detection through prediction error analysis. Figure 1 illustrates TiTAD’s architectural overview.

3.1. Time-Invariant Transformer

The prediction models learn standard patterns from multivariate time series by improving prediction accuracy during training. The prediction accuracy of a time t depends strongly on the spatio-temporal properties of the corresponding sliding window, and these properties change dynamically. To help the model explore standard patterns, we use the Transformer to mine the dynamic spatio-temporal features within the sliding windows. During the anomaly inference phase, the prediction model generates higher prediction errors when predicting anomalous points that deviate from the standard patterns. In particular, we improve the discrimination of anomalies by leveraging time-invariant features. To ensure robustness against temporal distribution shifts, we formally define time-invariant features. A feature $f\in \mathcal{F}$ is considered time-invariant if it satisfies the conditions as defined in Equation (1).

$$\begin{array}{ll}\hfill \left(\mathrm{C}1\right)& {\displaystyle \frac{df\left(X_t\right)}{dt}}\approx 0\text{(Continuous systems)},\hfill \\ \hfill \left(\mathrm{C}2\right)& \rho (f,t)<{\delta }_{\rho }\text{(Discrete systems, Spearman’s rank test)},\hfill \end{array}$$

(1)

where ${\delta }_{\rho }=0.1$ denotes the correlation threshold and $X_t$ represents input data at time t. These features, including spectral energy ratios and conserved physical quantities, are selected based on signal stationarity and domain-specific invariance principles. Since the time-invariant features can complement the spatio-temporal features to enhance the prediction precision of the normal data while the anomalies have a lower correlation with the spatio-temporal features, it does not help or even harm the prediction accuracy of the anomalies, making the prediction deviation of the anomalies more outstanding and easier to detect.

3.1.1. Memory-Augmented Transformer

The integration of time-invariant features induces structural differentiation in prediction error distributions. We formalize the error amplification mechanism through error decomposition in Equation (2).

$$\epsilon ={\epsilon }_s+\lambda {\epsilon }_t,$$

(2)

where $\epsilon \triangleq \parallel \widehat{y}{-y\parallel }_2$ denotes the total prediction error, ${\epsilon }_s$ represents spatio-temporal components, and ${\epsilon }_t$ quantifies the mismatch in time-invariant features. The coupling coefficient $\lambda$ governs interaction intensity between modalities. For a normal sample $\mathcal{N}$, temporal consistency in invariant features ($C_t^{\mathcal{N}}>\tau$) is induced by Equation (3).

$${\epsilon }_t^{\mathcal{N}}\propto exp(-\alpha C_t^{\mathcal{N}})\to 0(\alpha >0),$$

(3)

yielding ${\epsilon }^{\mathcal{N}}\approx {\epsilon }_s^{\mathcal{N}}$. For an anomalous sample $\mathcal{A}$, temporal inconsistency ($C_t^{\mathcal{A}}<\tau$) triggers error amplification, as defined in Equation (4).

$${\epsilon }^{\mathcal{A}}={\epsilon }_s^{\mathcal{A}}+\lambda \underset{\mathrm{Invariance}\mathrm{Violation}}{\underbrace{{\Delta }_t^{\mathcal{A}}}},$$

(4)

where ${\Delta }_t^{\mathcal{A}}\triangleq {\parallel f_t\left(x_{\mathcal{A}}\right)-\mathbb{E}\left[f_t\left(x_{\mathcal{N}}\right)\right]\parallel }_2$ measures the deviation from invariant patterns.

The characteristics of each variable and the relationships among variables are dynamic, but there are also some time-invariant characteristics and relationships. We use embedding vectors denoted by $E\in {\mathbb{R}}^{N\times d}$ to capture these time-invariant features, where d represents the embedding vector dimension. These vectors are continuously updated through the training process. The attention mechanism is a powerful tool for learning relationships between items and has been extensively applied in various domains, including link prediction and natural language processing. In this study, we apply the attention mechanism on the embedding vectors to extract the time-invariant relationships among variables, as defined in Equation (5).

$$\begin{array}{c}{Q}^{\prime }=E{W}_q,\hfill \\ K^{\prime }=E{W}_k,\hfill \\ \omega ={\displaystyle \frac{Q^{\prime }{K}^{\prime }}{\sqrt{d_k^{\prime }}}},\hfill \end{array}$$

(5)

where $Q^{\prime }$ and $K^{\prime }$ denote query and key, respectively, $W_q$ and $W_k$ are learnable parameters, and $d_k^{\prime }$ represents the query and key dimension.

The attention score $\omega \in {\mathbb{R}}^{N\times N}$ generated by the attention mechanism represents the relationship among each variable, and the relationship may be strong or weak. The weak correlations may be due to errors, and they are probably closer to noise. To reduce the influence of this noise, we introduce a threshold for $\omega$. If the attention score between two variables is greater than the threshold, the relationship between them is stable. Conversely, the relationship between them is unstable. Finally, we obtain an adjacency matrix $G\in {\mathbb{R}}^{N\times N}$. We set the threshold to 0 for better training. E and G represent the time-invariant feature from different perspectives. E captures the long-term behavioral characteristics of its corresponding variables in T, and G captures the stable inter-variable relationships throughout T.

3.1.2. Enhanced Temporal Modeling

Figure 2 depicts the Transformer’s architecture. Given the dynamic evolution of variable characteristics and inter-variable relationships, we employ the Transformer to compute attention scores through variable similarity comparisons, enabling effective spatio-temporal feature aggregation within temporal windows. To amplify prediction deviations between normal and anomalous data, we implement dual time-invariant mechanisms. First, E is dynamically updated during training to capture stable behavioral patterns (periodicity/trends) per variable. E and $X_t$ are jointly processed to incorporate time-invariant temporal context in predictions. Second, we multiply the attention scores generated by the Transformer element-wise with G, which represents the long-term stable correlation between variables. This operation ensures predictions consider both transient and time-invariant correlations, effectively overcoming limitations caused by local temporal smoothness while maintaining sensitivity to persistent system-wide dependencies.

The time-invariant features exhibit selective efficacy analogous to ”One man’s meat is another one’s poison”. For normal data, these features suppress noise and provide long-term contextual cues, thereby improving prediction accuracy. Conversely, their mismatch with anomalous patterns exacerbates prediction errors, enhancing anomaly discriminability. Anomalous data, characterized by sporadic and statistically divergent temporal patterns, inherently conflict with time-invariant feature representations. This incongruity manifests in the prediction phase and the detection phase. In the prediction phase, time-invariant features, optimized for stable normal patterns, fail to guide predictions for anomaly-driven irregularities. This results in amplified reconstruction/prediction errors due to the mismatch between invariant feature priors and anomalous dynamics. In the detection phase, the exaggerated prediction errors act as discriminative signals, creating a stronger separation between normal and anomalous distributions in the error space. This discriminative effect lowers the decision boundary for anomaly identification, effectively transforming the model’s prediction weakness into a detection strength. Our Transformer’s attention scores are calculated by Equation (6).

$$\begin{array}{cc}& Q=(X_t+E)W_Q,\hfill \\ & K=(X_t+E)W_K,\hfill \\ & V=(X_t+E),\hfill \\ & \alpha =\psi \left({\displaystyle \frac{QK}{\sqrt{d_k}}}\odot G\right).\hfill \end{array}$$

(6)

Let Q, K, $V\in {\mathbb{R}}^{d\times d}$ denote query, key, value, and normalization operations, respectively. $W_Q$ and $W_K\in {\mathbb{R}}^{d\times d}$ represent the learnable matrix, and $d_k$ is the dimension of query, key, and value. $\alpha \in {\mathbb{R}}^{N\times N}$ represents the relationship between the variables. Finally, we aggregate the spatial features $S_t\in {\mathbb{R}}^{N\times d}$ through Equation (7).

$$\begin{array}{c}\hfill S_t=\alpha V.\end{array}$$

(7)

3.1.3. Feature Interaction Mechanism

Among existing methods, Transformers have been predominantly employed for temporal feature extraction due to their exceptional capability in capturing inter-temporal relationships, analogous to word interaction modeling in NLP. However, the native attention mechanism inherently lacks inherent sequential preservation—a critical property for time series analysis [27]. GRU, as a streamlined RNN variant, matches LSTM’s performance with reduced parameterization while effectively retaining temporal order information and preserving long-term dependencies. We therefore integrate GRUs to complement the Transformers’ temporal modeling capabilities through sequential information retention. Specifically, the GRU acts as a local temporal encoder, processing sequential data stepwise to retain fine-grained short-term dynamics. Its hidden states preserve local temporal order, providing context-rich inputs to the Transformer. The Transformer leverages global self-attention to model long-range dependencies and variable correlations, enhanced by time-invariant embeddings that stabilize relationships across time. When the GRU detects a short-term spike in a sensor, the Transformer evaluates whether this anomaly contradicts the long-term behavioral patterns of other variables (e.g., stable correlations learned via time-invariant embeddings). This collaboration distinguishes true faults from noise, as the GRU’s local sensitivity and the Transformer’s global reasoning mutually validate anomalies.

Especially, we first calculate the update gate state denoted by $Z_t\in {\mathbb{R}}^{N\times h}$ and the reset gate state denoted by $R_t\in {\mathbb{R}}^{N\times h}$, where h represents the dimension of the GRU. The reset gate is applied to determine the amount of information from the previous hidden state $H_{t-1}$ that should be retained when constructing the candidate hidden state ${\tilde{H}}_t$. The update gate decides the amount of information from $H_{t-1}$ that should be retained and the amount of information from ${\tilde{H}}_t$ that should be accepted when constructing the current hidden state $H_t$, as defined in Equation (8).

$$\begin{array}{cc}\hfill & R_t=\sigma (X_t{W}_{xr}+H_{t-1}{W}_{hr}+b_r),\hfill \\ \hfill & Z_t=\sigma (X_t{W}_{xz}+H_{t-1}{W}_{hz}+b_z),\hfill \end{array}$$

(8)

where $W_{xr},W_{xz}\in {\mathbb{R}}^{d\times h}$, $W_{hr},W_{hz}\in {\mathbb{R}}^{h\times h}$ represent the weight matrix, $b_r,b_z\in {\mathbb{R}}^{1\times h}$ represent the bias matrix, and $\sigma$ is the sigmoid operation. After that, we use $R_t$ to compute the candidate hidden state denoted by ${\tilde{H}}_t\in {\mathbb{R}}^{N\times h}$, as defined in Equation (9).

$$\begin{array}{cc}\hfill & {\tilde{H}}_t=\tanh (X_t{W}_{xh}+(R_t\odot H_{t-1})W_{hh}+b_h),\hfill \end{array}$$

(9)

where $W_{xh}\in {\mathbb{R}}^{d\times h}$, $W_{hh}\in {\mathbb{R}}^{h\times h}$ represent the weight matrix, and $b_h\in {\mathbb{R}}^{1\times h}$ represents the bias matrix. Then, we combine ${\tilde{H}}_t$ and the update gate $Z_t$ to calculate $H_t\in {\mathbb{R}}^{N\times h}$. Finally, $H_t$ will take as final output $T_t\in {\mathbb{R}}^{N\times h}$, as defined in Equation (10).

$$\begin{array}{cc}\hfill & H_t=Z_t\odot H_{t-1}+(1-Z_t)\odot {\tilde{H}}_t,\hfill \\ \hfill & T_t=H_t.\hfill \end{array}$$

(10)

3.2. Feature Fusion

The FF module incorporates NeXtVLAD, originally developed for video processing. To mitigate model overemphasis on time-invariant features, this module recalibrates spatio-temporal feature weights for optimal balance. Video data, conceptualized as multivariate time series, exhibit inherent spatio-temporal correlations. Each pixel represents a variable, while frame sequences capture temporal observations [35]. NeXtVLAD’s success in video feature fusion stems from its ability to aggregate intra-frame (spatial) and inter-frame (temporal) dependencies [30,36]. As a VLAD extension [37], NeXtVLAD reduces parameterization while enhancing performance over NetVLAD [38]. Its proficiency in spatio-temporal dependency aggregation makes it ideal for fusing temporal patterns and invariant characteristics in our framework.

Figure 3 shows the architecture of NeXtVLAD. Before Feature Fusion, our pipeline conducts feature decomposition followed by clustering operations. The process computes residuals between decomposed features and cluster centroids based on grouping results. This methodology achieves dual objectives: mitigating noise contamination through efficient feature representation while amplifying spatio-temporal feature saliency to counteract the disproportionate emphasis on time-invariant characteristics. Specifically, we first feed the spatio-temporal features $(T_t\mid S_t)\in {\mathbb{R}}^{N\times 2d}$ to the linear layer in order to expand its dimension to $x\in {\mathbb{R}}^{N\times 2\lambda d}$. Then, we divide x into G groups $\overline{x}\in {\mathbb{R}}^{N\times G\times {\displaystyle \frac{2\lambda d}{G}}}$; this step can be considered as decomposing x into G low-dimensional feature vectors $\overline{x}$. And then we cluster $\overline{x}$, the cluster centers are $u\in {\mathbb{R}}^{K\times {\displaystyle \frac{2\lambda d}{G}}}$, it is derived from the model training, and it fits time-invariant features. Next, we compute the probability ${\alpha }_{gk}$ that each group of features ${\overline{x}}_n^g$ belongs between the clustering centers $u_k$; the probability ${\alpha }_{gk}$ will be computed by the linear layer and softmax, as defined in Equation (11).

$$\begin{array}{cc}& \mathrm{softmax}{\left(z\right)}_i={\displaystyle \frac{exp\left(z_i\right)}{{\sum }_{j=1}^{K}exp\left(z_j\right)}},\mathrm{for}i=1,\dots ,K,\hfill \\ & {\alpha }_{gk}\left(x_n\right)=\mathrm{softmax}(w_{kg}^T{x}_n+b_{kg}),\hfill \\ & {\alpha }_{gk}\left(x_n\right)={\displaystyle \frac{\exp (w_{kg}^T{x}_n+b_{kg})}{{\sum }_{k^{\prime }=1}^K\exp (w_{k^{\prime }g}^T{x}_n+b_{k^{\prime }g})}},\hfill \\ & n\in \left\{1,\dots ,N\right\},k\in \left\{1,\dots ,K\right\},g\in \left\{1,\dots ,G\right\}.\hfill \end{array}$$

(11)

We then aggregate the feature residuals of each group of features $\overline{x}$ with clustering centers u with probability ${\alpha }_{gk}$ to eliminate some redundant information and noise; the residuals represent dynamic features, as defined in Equation (12).

$$\begin{array}{c}\hfill {\tilde{z}}_{kg}=\sum _{n=1}^N{\alpha }_{gk}\left(x_n\right)({\overline{x}}_{ng}-{\mu }_k).\end{array}$$

(12)

Next, we use a linear and a sigmoid operation (denoted by $\sigma$) to calculate the weights ${\alpha }_g$ for each set of features $\overline{x}$ and construct the global features $z_k$ of the input data based on ${\alpha }_g$, as defined in Equation (13).

$$\begin{array}{cc}& {\alpha }_g\left(x_n\right)=\sigma (w_g^N{x}_n+b_g),\hfill \\ & z_k=\sum _{g=1}^G{\alpha }_g\left(x_n\right)\sum _{n=1}^N{\alpha }_{gk}\left(x_n\right)({\overline{x}}_{ng}-{\mu }_k).\hfill \end{array}$$

(13)

Finally, a linear layer is applied for mining high-level features as well as performing dimensional transformations for the prediction task. The final output is represented as $A\in {\mathbb{R}}^{N\times d}$.

3.3. Optimization and Anomaly Inference

3.3.1. Prediction

The model utilizes the standard patterns learned during training to generate predictions for test data. Owing to complex and variable data patterns of anomalous data, which are different from those of normal data, and the different effects of time-invariant features on anomalous data and normal data, prediction deviation becomes pronounced for anomalies, enabling detection through error analysis. We process the features fused by FF through a linear layer, yielding a prediction $\widetilde{x_t}\in {\mathbb{R}}^N$.

3.3.2. Loss Function

Compared to MAE, MSE exhibits greater sensitivity to outliers, prioritizing them at the expense of normal data accuracy. However, as anomaly detection models learn the standard patterns in the training, and the training set does not tend to contain anomalies while outliers are often anomalies, the impact of outliers is small. Therefore, we adopt Mean Squared Error (MSE) for the loss function to accelerate model convergence, which calculates the mean squared error of the predictions ${\tilde{x}}_t$ and ground truth $x_t$ and takes the square. This is a common choice for regression problems, as defined in Equation (14).

$$\begin{array}{c}\hfill Loss={\displaystyle \frac{1}{N}}{∥{\tilde{x}}_t-x_t∥}_2^2.\end{array}$$

(14)

3.3.3. Anomaly Inference

We compute anomaly scores per variable based on the basis of the distance of the predicted value from the ground truth value, then aggregate variable-wise scores into temporal point scores for holistic anomaly detection. These variable-level scores provide diagnostically valuable insights, facilitating root cause localization while enhancing model interpretability. To calculate the anomaly scores, we measure the Mean Absolute Error (MAE) between ${\tilde{x}}_{ti}$ and $x_{ti}$ of ith variable at t. The formula for MAE is computed using Equation (15).

$$\begin{array}{c}\hfill s_{ti}=\left|{\tilde{x}}_{ti}-x_{ti}\right|.\end{array}$$

(15)

Due to the varying difficulty of predicting each variable, such as with discrete and continuous data, certain variables may have a significantly higher average prediction error than others. To neutralize this bias in the final anomaly scoring, we implement the normalization strategy employed by GDN. We first compute the median (${\mu }_i^{\prime }$) and the interquartile range (IOR) (${\sigma }_i^{\prime }$) for the time series of the ith variable. We then smooth anomaly scores by the simple moving average, as defined in Equation (16).

$$\hfill S_{ti}={\displaystyle \frac{s_{ti}-{\mu }_i^{\prime }}{{\sigma }_i^{\prime }}}.$$

(16)

We base the prediction deviation to compute the anomaly scores for each variable at each time point and choose the maximum values as the total score for the whole time point, as defined in Equation (17).

$$\begin{array}{c}\hfill A_t=\max \left(S_t\right).\end{array}$$

(17)

Finally, we set a threshold for the anomaly score; time steps that have scores above the threshold are detected as anomalous, and time steps that have scores below the threshold are detected as normal. Threshold selection critically influences anomaly inference [34], with numerous methodologies existing for this purpose (e.g., Peaks-Over-Threshold (POT) [39]). The optimal threshold is determined through parametric optimization of POT. For simplicity, we exhaustively evaluate candidate thresholds via $F1$ score-driven optimization to identify the most discriminative value.

4. Experiment

In the experimental section, we evaluate our method using three publicly available anomaly detection datasets. These datasets are selected to encompass a variety of real-world scenarios, ensuring a thorough assessment of the model’s performance. Our experiments compare TiTAD with several state-of-the-art baseline methods, and the results consistently show that our model outperforms these baselines across all datasets.

4.1. Experimental Setup

4.1.1. Datasets

In this study, we evaluate our method using three publicly available anomaly detection datasets: Secure Water Treatment (SWaT), Water Distribution (WADI), and Server Machine Dataset (SMD).

• Secure Water Treatment [40,41]: The SWaT dataset originates from an operational water treatment facility, capturing eleven days of continuous multivariate operational data encompassing actuator states, sensor measurements, and network telemetry. This dataset includes seven days of data collected during non-attack operations and four days of data collected under attack conditions. The seven days of operational data without attacks are used for training, while the four days of operational data under attack are used for testing.
• Water Distribution [42]: The WADI dataset consists of sixteen days of actuator and sensor data from a city Water Distribution System (WDS). It includes two weeks of data collected during operation without attacks and two days of data collected under attack conditions. The two weeks of operation data without attacks are used for training, and the two days of operation data in the presence of attack behavior are used for testing.
• Server Machine Dataset [9]: The SMD dataset is provided by a large internet company and is composed of monitoring data from twenty-eight server machines over a period of five weeks, all with the same variables. The SMD dataset has twenty-eight sub-datasets, each of which needs to be independently trained and tested.

We use these three datasets to evaluate our method, and their statistics are shown in

Table 1. The statistical profile of the datasets in the experiment.

Datasets	Train	Test	Dimensions	Entities	Anomaly Ratio (%)	Year
SWaT	496,800	449,919	51	1	11.98%	2016
WADI	1,048,571	172,801	123	1	5.99%	2017
SMD	708,405	708,420	38	28	4.16%	2019

. The training sets of these datasets are exclusive of anomalies. For SWaT and WADI, we adopt a downsampling strategy to speed up training. For SMD, we train and test the twenty-eight sub-datasets separately and then calculate the average of the results. Since TranAD and AnomalyTransformer [12] use different evaluation methods for SMD, we also use their respective evaluation methods to ensure fair comparisons.

4.1.2. Data Preprocessing

To balance certain variables that have larger orders of magnitude with other variables that do not have larger orders of magnitude, we use StandardScaler to normalize each variable so that its variance is 1 and its mean is 0. The formula is defined in Equation (18).

$$\begin{array}{c}\hfill x={\displaystyle \frac{x-\mu }{\sigma }}.\end{array}$$

(18)

Specifically, we first normalize each variable by computing its mean $\mu$ and standard deviation $\sigma$ in the training set, then use the $\mu$ and $\sigma$ to normalize the test sets. While z-score normalization assumes stationarity, our sliding window design inherently mitigates moderate local trends by normalizing relative variations within each window. For datasets with strong global trends, we recommend preliminary detrending (e.g., differencing). The current approach remains valid for our benchmark datasets, where operational phase alignment ensures quasi-stationarity. Since multivariate time series are long, we divide them into sliding windows that can overlap as inputs. $x_t=\left[o_{t-w+1},o_{t-w+2},\dots ,o_{t-1}\right]$, where $o_t\in {\mathbb{R}}^{1\times N}$, w represents the sliding window length, and N represents the number of variables. To extract high-dimensional features from the input data, we employ the projection layer in advance, as defined in Equation (19).

$$\begin{array}{c}\hfill X_t=x_t^T{W}_{line},\end{array}$$

(19)

where $W_{line}\in {\mathbb{R}}^{w\times d}$ represents the trainable weight matrix and $X_t\in {\mathbb{R}}^{N\times d}$ represents the extracted high-dimensional features. The projection layer serves as a critical preprocessing step to transform raw sensor readings into a latent space that better captures discriminative spatio-temporal patterns. Specifically, this layer applies a learnable linear transformation to each sliding window of normalized input data, effectively compressing the temporal dimension while preserving cross-variable correlations. It acts as a learnable feature enhancer that transforms raw sensor readings into discriminative representations.

4.1.3. Implementation Details

We used PyTorch [43] version 1.12.1, PyTorch Geometric Library [44] 1.5.0, and CUDA 11.6, and all our experiments were performed on an NVIDIA RTX 4080. For all the experiments, we fixed the size of the time window to 64, the groups’ number G to 16, the number of clustering centers K to 4, and $\lambda$ to 2. For the WADI dataset, the embedding vector dimensionality is fixed at 128, for the SWaT dataset, the embedding vector dimensionality is fixed at 512, and for the SMD dataset, the SMD consists of 28 sub-datasets, which are not presented owing to the limited space. The linear layer and GRU dimensionality are fixed equal to the embedding vector dimensionality. The number of layers for both Transformer and GRU is set to 3. We adopted the Adam [45] optimizer and fixed the learning rate at 0.001. The batch size is set to 256. The number of epochs is set to 70 for the SWaT and the WADI datasets, and 250 for the SMD dataset. For better training, we randomly divided 30% of the training data for validation, whereas the WADI dataset is randomly divided into 20% for validation. To avoid overfitting, we applied the early-stopping criterion in training the model, and we stopped training when the accuracy started to drop continuously after validation.

4.1.4. Baselines

We select DAGMM [10], LSTM-VAE [32], OmniAnomaly [9], MSCRED [33], USAD [31], THOC [46], MTAD-GAT [47], CAE-M [48], GDN [17], TranAD [13], AnomalyTransformer [12], and GRN [18] as our baseline methods. The MSCRED captures multi-scale temporal patterns and inter-variable dependencies using a hybrid CNN-RNN architecture. The Temporal Hierarchical One-Class network (THOC) leverages hierarchical temporal representations for one-class anomaly detection. The Deep Convolutional Autoencoder Memory network (CAE-M) enhances autoencoder robustness by integrating a memory module for prototypical normal patterns. The Multivariate Time Series Anomaly Detection with Graph Attention Network (MTAD-GAT) combines graph attention layers with temporal convolutions to explicitly model variable interactions. In particular, we note that 15 variables are removed during the training of GDN on the WADI dataset, so we re-evaluated GDN on the WADI dataset. The original papers of TranAD and AnomalyTransformer only provide the result with point adjustment, so we re-evaluated these methods without point adjustment.

4.1.5. Evaluation Metrics

We employ the $F1$ score, a commonly applied evaluation metric in multivariate time series anomaly detection, as our evaluation metric. The $F1$ score comprehensively indicates the precision and recall of anomaly detection. The $F1$ Score is computed as defined in Equation (20).

$$\begin{array}{cc}\hfill & F1={\displaystyle \frac{2\times Prec\times Rec}{Prec+Rec}},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & Prec={\displaystyle \frac{TP}{TP+FP}},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill & Rec={\displaystyle \frac{TP}{TP+FN}},\hfill \end{array}$$

(22)

where $TN$, $TP$, $FN$, and $FP$ are the counts of true negatives, the counts of true positives, the counts of false negatives, and the counts of false positives, separately.

4.2. Main Results

4.2.1. Performance Comparison

To ensure reproducibility, we provide complete implementation details including hyperparameter configurations and training protocols. All experiments were conducted under identical hardware/software conditions to ensure statistical reliability.

Table 2. The $F1$ scores comparison of our model with the baseline. In addition, as a reference, we also list the precision and recall for each dataset, except for the SMD which consists of multiple sub-datasets. The best results are in bold and the next best results are underlined.

Datasets	SWaT			WADI			SMD
Metric	F1	Prec	Rec	F1	Prec	Rec	F1
DAGMM	0.550	0.469	0.665	0.121	0.065	0.913	0.238
LSTM-VAE	0.775	0.989	0.637	0.227	0.994	0.128	0.435
OmniAnomaly	0.782	0.982	0.649	0.223	0.994	0.129	0.474
MSCRED	0.662	-	-	0.087	-	-	0.097
USAD	0.791	0.985	0.661	0.232	0.994	0.131	0.426
THOC	0.612	-	-	0.130	-	-	0.168
MTAD-GAT	0.810	0.971	0.695	0.416	0.281	0.801	-
CAE-M	0.810	0.969	0.695	0.411	0.278	0.791	-
GDN	0.808	0.993	0.681	0.426	0.291	0.793	0.529
TranAD	0.607	0.996	0.436	0.251	0.217	0.298	0.288
AnomalyTransformer	0.246	0.444	0.170	0.054	0.037	0.100	0.021
GRN	0.749	0.998	0.590	0.482	0.358	0.739	-
Ours	0.842	0.966	0.745	0.512	0.468	0.565	0.557

presents the anomaly detection capabilities of TiTAD compared to various baseline models. Specifically, TranAD and AnomalyTransformer omit the evaluation of SMD’s 28 sub-datasets individually: TranAD tests only four sub-datasets (machine-1-1/2-1/3-2/3-7), while AnomalyTransformer concatenates all sub-datasets pre-evaluation. Our replication of their methodologies yields $F1$ scores of 0.437 (TranAD) and 0.2286 (AnomalyTransformer) on SMD. GDN achieves 0.569 $F1$ on WADI’s 15-variable subset, whereas TiTAD surpasses this with 0.658 $F1$. TiTAD consistently outperforms all baselines across the three benchmark datasets. Effective anomaly detection necessitates robust spatio-temporal feature extraction. LSTM-VAE, OmniAnomaly, USAD, and AnomalyTransformer underperform due to inadequate inter-variable relationship modeling. DAGMM’s neglect of temporal dynamics severely compromises its effectiveness. GDN’s graph-based spatial modeling achieves notable performance, particularly on high-dimensional WADI, yet its disregard for temporal patterns limits its overall efficacy. MSCRED’s sequential processing of spatial and temporal features fails to achieve effective spatio-temporal fusion. Critically, existing methods overlook time-invariant feature utilization for anomaly amplification. TiTAD’s dual exploitation of spatio-temporal dynamics and time-invariant characteristics drives its superior detection performance.

4.2.2. Anomaly Diagnosis

TiTAD provides anomaly diagnostic capabilities, evaluated using widely used metrics HitRate@P% [9] and NDCG@P% [49]. HitRate@P% is defined as ${\displaystyle \frac{Hit@\lfloor P\%\times \left|G{T}_t\right|\rfloor }{\left|G{T}_t\right|}}$, where $\left|G{T}_t\right|$ denotes the number of ground truth dimensions (i.e., labeled anomalous features) and $P\%\times \left|G{T}_t\right|$ represents the number of top candidates (i.e., highest-ranked predicted anomalies). The HitRate@P% measures the proportion of ground truth dimensions found within the top candidates that the model inferred. Unlike HitRate@P%, Normalized Discounted Cumulative Gain (NDCG) rewards models for ranking true anomalies higher within the same top-k candidates. To explore TiTAD’s diagnostic performance, we used the SMD dataset, which provides ground truth dimensions and published baselines. As shown in

Table 3. Anomaly diagnosis. The best results are in bold and the next best results are underlined.

Datasets	SMD
Metric	HitRate@100%	HitRate@150%	NDCG@100%	NDCG@150%
MERLIN	0.5907	0.6177	0.4150	0.4912
LSTM-NDT	0.3808	0.5225	0.3603	0.4451
DAGMM	0.4927	0.6091	0.5169	0.5845
OmniAnomaly	0.4567	0.5652	0.4545	0.5125
MSCRED	0.4272	0.5180	0.4609	0.5164
MAD-GAN	0.4630	0.5785	0.4681	0.5522
USAD	0.4925	0.6055	0.5179	0.5781
MTAD-GAT	0.3493	0.4777	0.3759	0.4530
CAE-M	0.4707	0.5878	0.5474	0.6178
GDN	0.3143	0.4386	0.2980	0.3724
TranAD	0.4981	0.6401	0.4941	0.6178
Ours	0.5328	0.6460	0.5703	0.6379

, our model achieves sub-optimal HitRate@100% and optimal HitRate@150%. In particular, our model outperforms others in NDCG@P%, which further demonstrates superior robustness to overprediction and ranking accuracy.

4.3. Model Analysis

4.3.1. Ablation Analysis

“Ablation analysis” refers to systematically removing or modifying components of a model to quantify their individual impact on performance. This method is a cornerstone of interpretable model design in complex systems. We conducted an ablation analysis to isolate the contributions of the GRU module, Transformer module, and fusion layer.

Table 4. Ablation results. The best results are in bold.

Datasets	SWaT	WADI	SMD
w/o:TiF	0.785	0.438	0.521
w/o:Transformer	0.745	0.389	0.517
w/o:GRU	0.528	0.408	0.510
w/o:FF	0.777	0.354	0.521
Ours	0.842	0.512	0.557

demonstrates the critical importance of each component. The original Transformer variant (“w/o:TiF”) shows performance degradation ($F1$: 0.785 vs. 0.842 on SWaT) due to its inability to leverage time-invariant features, particularly crucial for detecting subtle anomalies through long-term pattern analysis. The Transformer captures global dependencies via self-attention. Removing spatial modeling capability (“w/o:Transformer”) reduces $F1$ by 11.5% on SWaT, confirming the necessity of inter-variable relationship analysis for distinguishing local fluctuations from system-level anomalies. The GRU removal (“w/o:GRU”) causes the most severe performance drop ($F1$: 0.528 vs. 0.842 on SWaT), highlighting the essential role of sequential processing in capturing temporal dynamics. The GRU specializes in modeling local temporal dependencies through its gated recurrent structure. The GRU’s output (local features) serves as input to the Transformer, which then refines these features by incorporating global context. Simplified Feature Fusion (“w/o:FF”) degrades performance by 30.9% on WADI, emphasizing the need for sophisticated integration of spatio-temporal and invariant features. The memory-augmented time-invariant module stabilizes training by providing persistent normal patterns to both GRU and Transformer. These results (as shown in Table 4), combined with theoretical insights about GRU’s gate mechanisms filtering irrelevant temporal contexts for the Transformer’s global modeling, collectively suggest that their interaction balances local–global trade offs rather than merely stacking functions. These findings collectively validate our architectural design choices, demonstrating that the synergistic combination of temporal invariance modeling, spatial attention, sequential processing, and adaptive fusion is fundamental to TiTAD’s superior anomaly detection capability.

4.3.2. Point-Adjustment Analysis

If a time point within a continuous abnormal segment is detected, all anomalies within that segment are considered to be correctly identified. This approach is justified by the observation that an anomalous time point will trigger an alert and subsequently draw attention to the entire segment. Therefore, detecting a single point of anomaly effectively implies the recognition of the whole abnormal segment. This strategy is a standard practice in time series anomaly detection frameworks [9,12,13,46]. It considers an entire anomaly segment to be correctly detected if any abnormal point within that segment is identified. This methodology enables certain algorithms to report inflated $F1$ scores exceeding 0.90. However, recent studies [50,51,52] demonstrate that point adjustment significantly inflates the actual performance of existing methods. As evidenced in

Table 5. The $F1$ score comparison under the point-adjustment strategy. The best results are in bold.

Datasets	SWaT	WADI	SMD
DAGMM	0.853	0.209	0.723
LSTM-VAE	0.805	0.380	0.808
OmniAnomaly	0.866	0.417	0.944
MSCRED	0.868	0.346	0.389
USAD	0.846	0.429	0.938
THOC	0.880	0.506	0.541
GDN	0.935	0.855	0.716
TranAD	0.815	0.495	0.961
AnomalyTransformer	0.941	0.698	0.923
Random Anomaly Score	0.969	0.965	0.804

, randomly generated scores achieve superior performance to state-of-the-art methods under the point adjustment. This phenomenon arises because longer anomaly segments statistically increase the probability of random hits [53]. More critically, we identify a negligible correlation between raw and adjusted scores. To ensure rigorous evaluation, we exclusively report non-adjusted $F1$ scores.

To exemplify how point adjustment artificially inflates performance metrics and induces methodological bias, we analyze the inherent flaws of this strategy using AnomalyTransformer as an example. Figure 4 reveals frequent false positives in normal regions of AnomalyTransformer on the SWaT dataset. For comparison, Figure 5 presents the anomaly inference results of our proposed method on the same dataset. Figure 4a illustrates the anomaly scores generated by AnomalyTransformer on the SWaT dataset. Our analysis (Figure 5a) reveals partial detection capability where specific anomaly segments exhibit elevated scores, yet non-salient scores for numerous anomalies impede their distinction from normal data. Figure 4b and Figure 5b display the anomaly label distribution, showing a homogeneous distribution with frequent label occurrences in anomaly segments. This pattern induces excessive false positives, resulting in significantly sub-optimal detection accuracy. Figure 4c and Figure 5c illustrate the point-adjusted results. The presence of sporadic labels within anomaly segments enables artificial true positive inflation while retaining unchanged false positive/negative counts [50]. Crucially, AnomalyTransformer’s threshold selection (upper quartile of scores) reduces total anomaly labels, partially mitigating false positive impacts and enabling artificial $F1$ score inflation.

Our analysis demonstrates that point-adjusted scores constitute statistically biased estimators, failing to reflect true algorithmic performance. This adjustment induces algorithmic bias toward generating anomaly distributions with homogeneous spatial patterns and suppressed anomaly counts. Such methodological artifacts fundamentally explain the observed discrepancy between adjusted and raw $F1$ scores, where artificially inflated $F1$ metrics emerge from statistical gaming rather than detection capability. In operational contexts, these uniform anomaly labels prove analogous to stochastic sampling protocols—theoretically valid for security audits yet deficient in practical detection scenarios. Operational anomaly detection necessitates capturing genuine anomaly patterns with authentic temporal frequencies and spatial distributions, rather than optimizing for statistical artifacts.

4.3.3. Sensitivity Analysis

Figure 6 illustrates the impact of different window sizes on anomaly detection performance. Empirical analysis reveals a positive correlation between window size and $F1$ scores, as larger windows capture enhanced temporal context with extended spatio-temporal dependencies. Inadequate temporal context (window size < 32) induces homogeneous prediction errors across normal and anomalous data. Consequently, both normal and anomalous data exhibit large prediction biases, making it difficult for the model to distinguish between them, which leads to poor performance. Optimal window ranges (64–128) enable effective synergy between temporal dynamics and time-invariant patterns. Normal data typically have more stable patterns and stronger correlations with time-invariant features, providing periodicity, and trend information that improves prediction accuracy. In contrast, anomalous data exhibit rare and unstable patterns with weaker correlations to time-invariant features, limiting or even deteriorating their prediction accuracy. This discrepancy enhances the distinction between normal and anomalous data, contributing to improved anomaly detection performance. Excessive window sizes (>128) introduce feature distribution skew between invariant and dynamic patterns, compromising $F1$ scores. Therefore, we chose a window size of 64 as the optimal compromise.

4.3.4. Distinguishability Analysis

“Distinguishability” refers to the capability of the model to differentiate between normal and abnormal time points in time series data based on their association patterns. Normal points typically exhibit broad temporal associations (e.g., long-range dependencies, periodic patterns) that reflect stable system behavior. Abnormal points, due to their rarity and contextual isolation, tend to form adjacent-concentrated associations (focusing on neighboring points). To examine whether time-invariant features enhance anomaly discriminability, we conducted a comparative analysis of prediction deviation-based anomaly scores across three categories: anomalies, normal points, and all points. We quantified relative prominence metrics through normalized scoring as shown in

Table 6. The average anomaly scores of all points, normal points, anomalies, and the outstanding degree of normal and anomalies relative to all points and anomalies relative to normal. The best results are in bold.

Datasets	SWaT		WADI		SMD
Method	w/o:TiF	Our	w/o:TiF	Our	w/o:TiF	Our
All points	10.1357	9.0329	0.7799	0.7773	0.5847	0.6040
Normal points	3.0915	2.1087	0.134	0.1238	0.3798	0.3706
Abnormal points	60.7057	58.7411	11.1879	11.3072	7.5069	8.3227
Normal/All	0.305	0.2334	0.1718	0.1593	0.6495	0.6135
Anomaly/All	5.9892	6.503	14.3441	14.5455	12.8369	13.7774
Anomaly/Normal	19.6359	27.8556	83.466	91.2767	19.7615	22.4550

. The average anomaly scores for all points, normal points, and abnormal points were scaled from 1 to 100. Specifically, the outstanding degree (Normal/All) represents the prominence of normal points relative to all points, (Anomaly/All) represents the prominence of anomalies relative to all points, and (Anomaly/Normal) represents the prominence of anomalies relative to normal points. The average anomaly scores of normal points across all three datasets decreased after incorporating time-invariant features, indicating that these features aid in predicting normal points more accurately, thereby reducing their anomaly scores. In contrast, the average anomaly scores of anomalies either decreased minimally or increased, suggesting that anomalies are less associated with time-invariant features. This divergence substantiates our hypothesis: while time-invariant features optimize normal pattern modeling, they deliberately decouple from anomalous patterns, creating amplified contrast between normal and anomaly predictions.

4.3.5. Anomaly Threshold Determination

The anomaly threshold is dynamically optimized through a two-stage statistical process to ensure robustness against label sparsity and temporal noise. The thresholding interfaces with the error smoothing layer (exponential moving average) to suppress transient noise, ensuring that thresholding responds to sustained anomalies rather than sporadic fluctuations. In deployment, thresholding is periodically updated using an incremental algorithm, enabling adaptation to concept drift without full retraining. We acknowledge that fully adaptive thresholding (e.g., online Bayesian updates) could enhance generalizability, which will be explored in future work.

5. Discussion and Limitations

The development and evaluation of TiTAD yield foundational insights for time series anomaly detection. The hybrid GRU–Transformer architecture demonstrates superior efficacy in balancing local–global temporal dependencies compared to monolithic designs. Unlike AnomalyTransformer, TiTAD mitigates temporal order degradation via GRU, improving the detection of transient anomalies. In comparison to USAD, time-invariant features in TiTAD reduce false positives caused by normal pattern shifts. TiTAD underperforms compared with GDN in detecting interaction-driven anomalies. TiTAD may exhibit limitations in environments with abrupt concept drift and could miss subtle anomalies induced by latent variable interactions. These limitations delineate clear pathways for improvement. Future work should investigate dynamic feature adaptation mechanisms, such as online meta-learning, to handle non-stationary environments without retraining. Architectural innovations like neural architecture search could automate the GRU–Transformer balance per application domain, while graph-enhanced fusion modules may better capture multivariate causality.

6. Conclusions

This paper presents the Time-invariant Transformer for Multivariate Time Series Anomaly Detection (TiTAD), a novel unsupervised framework that synergizes Transformer architectures with temporal invariance modeling. TiTAD employs hybrid Transformer and GRU architectures to learn spatio-temporal features in time series data through a Feature Fusion module. The integrated features are subsequently fed into a linear layer for forecasting, with the forecasting results utilized to infer anomalies. Specifically, TiTAD innovatively integrates time-invariant features into the Transformer architecture and introduces a Feature Fusion module for enhanced integration. By leveraging the distinct impacts of time-invariant features on the prediction accuracy of normal versus anomalous data, TiTAD improves anomaly detection performance. Our experimental results demonstrate that our model outperforms baseline methods across all three real-world datasets.