A Novel Data-Driven Approach with a Long Short-Term Memory Autoencoder Model with a Multihead Self-Attention Deep Learning Model for Wind Turbine Converter Fault Detection

Abstract

The imminent depletion of oil resources and increasing environmental pollution have driven the use of clean energy, particularly wind energy. However, wind turbines (WTs) face significant challenges, such as critical component failures, which can cause unexpected shutdowns and affect energy production. To address this challenge, we analyzed the Supervisory Control and Data Acquisition (SCADA) data to identify significant differences between the relationship of variables based on data reconstruction errors between actual and predicted values. This study proposes a hybrid short- and long-term memory autoencoder model with multihead self-attention (LSTM-MA-AE) for WT converter fault detection. The proposed model identifies anomalies in the data by comparing the reconstruction errors of the variables involved. However, more is needed. To address this model limitation, we developed a fault prediction system that employs an adaptive threshold with an Exponentially Weighted Moving Average (EWMA) and a fixed threshold. This system analyzes the anomalies of several variables and generates fault warnings in advance time. Thus, we propose an outlier detection method through data preprocessing and unsupervised learning, using SCADA data collected from a wind farm located in complex terrain, including real faults in the converter. The LSTM-MA-AE is shown to be able to predict the converter failure 3.3 months in advance, and with an F1 greater than 90% in the tests performed. The results provide evidence of the potential of the proposed model to improve converter fault diagnosis with SCADA data in complex environments, highlighting its ability to increase the reliability and efficiency of WTs.

1. Introduction

The world is currently facing the depletion of traditional energy resources such as oil and coal, and the challenges of climate change. At the same time, global energy demand continues to grow to meet economic and social needs. In response, clean energy is emerging as an alternative solution, releasing less carbon dioxide and offering unlimited energy sources to drive sustainable development [1]. The initiative to achieve carbon neutrality by 2050, proposed in the Paris Agreement, has increased interest and investment in the renewable energy sector.

Wind energy, in particular, represents a viable alternative to fossil fuel power generation, using natural sources to produce electricity. The cost of producing energy with this technology is decreasing yearly, with this trend being expected to continue [2]. Moreover, the continued development of wind turbines (WTs), driven by the integration of advanced technologies, is expected to significantly increase their efficiency and generating capacity.

However, wind power generation faces challenges regarding the availability, reliability, and lifetime of WTs. For this reason, manufacturers have focused their efforts on prolonging the lifetime of electrical and mechanical systems, which has resulted in reduced failures during operation and consistent production of high-quality power [3].

Unexpected failures in these systems can negatively affect availability and production rate. Components such as blades [4], generators [5], power converters [3], and gearboxes [6] are especially vulnerable to failure due to harsh environmental and operating conditions, leading to extended downtime for maintenance. If unaddressed, these challenges could significantly impact the renewable energy industry.

Fault detection is a critical problem with two main aspects: accuracy and cost [7]. The high vibration and noise levels of generators complicate the accurate measurement of faults. In addition, the diversity of faults depending on environmental conditions requires an accurate classification of each situation [8]. Implementing multiple sensors and complex algorithms is costly, especially for large-scale wind farms, and challenging to apply in smaller installations. As WTs increase in size and number, the need for cost reduction in monitoring and repair becomes more urgent [9,10].

One of the main WT components is the converter. The converters in WTs play a crucial role in facilitating the control and efficient use of energy, especially in complex terrain and with lower maintenance requirements [11]. The converter is a critical component in the operation of the WT, as it converts the variable frequency and voltage output from the generator into a stable, grid-compatible alternating current (AC) with a consistent frequency and voltage, ensuring that the electrical power generated is suitable for transmission and use [12]. According to [13,14], converters are one of the components that fail most in WTs, responsible for approximately 23% of the failures; the authors indicate that humidity, contamination, and other factors play an essential role in the incidence of these failures. Additionally, with the increasing development of direct-drive WTs, the converter is one of the most failure-prone components. This aspect becomes especially critical in complex terrains, where environmental and operating conditions are significantly more challenging than in less demanding terrains [15].

This study proposes an improved approach to fault detection using Supervisory Control and Data Acquisition (SCADA) system data. The goal is to detect faults in the converter of a WT located in complex terrain. This approach aims to optimize reliability and reduce the costs associated with operation and maintenance (O&M). The ultimate objective is to ensure the greater long-term viability and sustainability of wind energy.

Two common approaches to detecting WT faults are classification algorithms (supervised) and reconstruction models (unsupervised) [16]. Classification algorithms train machine learning (ML) models, such as decision trees and k-nearest neighbors, using labeled data and testing them on new data. However, labeling data can be a complex and expensive process. An example of recent work in this area is that by Khan in 2022 [17], who proposed a new classifier called AdaBoost, K-nearest neighbors, and logistic regression-based stacking ensemble (AKL-SE) to classify faults in WT monitoring systems, obtaining promising results.

The aim of reconstruction models, in contrast, is to understand the time series and reconstruct the variables to detect anomalies, using the reconstruction error as a measure to identify outliers. A notable example of this approach is the research by Xiang et al. [18], which proposes a method for WT fault detection. This method, which combines a convolutional neural network (CNN) with a long- and short-term memory (LSTM) network based on an attention mechanism (AM), is designed to alert on generator and gearbox faults, making it highly relevant to the field of WT systems.

In addition, deep learning (DL) networks have also been used in this area. Liu et al. [19] propose a new Deep Residual Network (DRN) for WT fault detection. In their method, raw data from the SCADA system are directly used as inputs to the DRN, which incorporates a Convolutional Residual Building Block (CRBB) with convolutional layers and compression and excitation units. This approach performs fault recognition and classification only when faults occur or are imminent, although, in practice, it is preferable to provide early warnings before a fault occurs.

A notable limitation is that most studies focus on detecting failures in mechanical or electrical components [20,21,22], while only some address failures in electronic components using SCADA data [23,24]. In this context, one of the studies exploring converters is that by Xiao et al. [25], who propose an improved structure called attention octave convolution (AOctConv), applied to the ResNet50 backbone (called AOC–ResNet50) for detecting faults in WT converters using SCADA data with 10 min intervals. In line with this research, we study this component for its recurrent failures, as demonstrated in Section 2.3.

Meanwhile, hybrid models are more robust than their base models, as evidenced in recent studies [26,27]. However, a limitation of these approaches is that they tend to detect alarms and do not attempt to predict failures.

An emerging approach in this context is the use of attention mechanisms, as presented in the study by Wang et al. [28]. The authors propose a novel method using an integrated AM with a multivariate query pattern for anomaly detection and underlying cause identification. The proposed anomaly detection model comprises multiple cascaded encoders and decoders with a multihead self-attention mechanism to extract correlations between variables. Inspired by this approach, we propose integrating a similar mechanism into our architecture.

1.1. Contribution of the Present Work

The aim of implementing an autoencoder (AE) model is to reconstruct the input data. However, the literature has shown that hybrid models yield superior performance. For this reason, this work implements LSTM and Multihead Attention (MA) layers in the AE architecture, specifically in the encoding and decoding layers, to capture more complex temporal relationships.

Specifically, we propose an architecture for fault prediction in WT converters using SCADA data, with the objective of improving reliability and configurability through fault prediction. This study implements an unsupervised learning approach using an LSTM-AE with multihead attention (LSTM-MA-AE), which incorporates temporal features from SCADA data.

This model is evaluated on a real dataset from a wind farm located in a complex terrain in southern Ecuador. We evaluate it with different architecture configurations to measure its performance in anomaly detection.

Therefore, the main contributions of this research are:

• Development of a DL model that integrates autoencoders, LSTM networks, and MA for fault detection in WT converters.
• Converter fault detection using SCADA data from a wind farm in complex terrain.
• A system of prediction of converter failures in real WT operating conditions using unsupervised learning.

1.2. Background and Motivations

In [29], reports, several methods for fault detection in WTs, which can be classified into model-based, signal-processing, and data-based approaches. Furthermore, ref. [11] presents a review of the literature on converter faults; the authors include model-based, signal-based, and data-based methods and conclude that the data-based method has a high fault diagnosis capability.

Furthermore, in [25], a study is presented for detecting faults in WT converters from SCADA data using CNN; the effectiveness of the research was verified through a comparative study. In addition, ref. [30] is a study detailing the most common WT converter faults, and presents a Transfer Learning (TL)-based method; the results demonstrate the accuracy and efficiency of the TL method in diagnosing WT faults.

A strategy using wavelet transform, feature analysis, and a Back Propagation Neural Network (BPNN) to accurately identify open-circuit faults in WTs converters, is presented in [31]; the results show that the proposed strategy can successfully classify converter faults. In addition, ref. [32] describes a data-based approach to detect WT converter faults using an LSTM; the results show that the proposed method has powerful data processing capabilities and higher diagnostic accuracy. Similarly, early anomaly detection and root cause analysis in WTs using SCADA data are proposed in [33]. For this purpose, a hybrid model using a LSTM-based asymmetric variational autoencoding Gaussian mixture model (LSTM-AVAGMM) is employed. The robustness and competitiveness of the model are demonstrated in two case studies.

Moreover, a hybrid DL model combining recurrent neural networks (RNN) and LSTM for WT condition monitoring is presented in [34]. LSTM-AE is employed for data processing and feature extraction in the proposed model. SCADA data and simulated data are then used to provide a complete learning model of the WT behavior. The experimental results unequivocally demonstrate that the proposed model surpasses existing ML algorithms in terms of fault prediction accuracy.

Likewise, the approach proposed by [35] is based on a supervised implementation of the variational AE model, which allows for the projection of the WT system in a low-dimensional representation space for early fault prediction. Another similar work is presented in [36], where a deep EA (DAE), enhanced by fault cases, is developed for anomaly detection in WT. With the help of fault cases, the DAE can capture normal operation data patterns and acquire deep embedding features. Experimental results show that the method outperformed current AE-based methods in WT anomaly detection using multiple evaluation metrics.

In [37], a technique is presented to monitor the aging of insulated-gate bipolar transistor (IGBT) modules in offshore WT converters using SCADA data and a hybrid AE and attention-based LSTM (AT-LSTM) model. The AT-LSTM model is used to learn from SCADA data and establish a temperature prediction. AE is used to detect anomalies. Experimental results validate the effectiveness of the proposed model.

Meanwhile, in a previous study, we determined that the scientific literature on using SCADA data to predict failures in WT converters located in complex terrain is limited [24]. The Villonaco Wind Farm (VWF), located in Ecuador in a mountainous area at approximately 2700 masl, presents a challenging climate and irregular terrain, entailing unique characteristics that distinguish it from other wind farms [38]. These unique conditions highlight the need for advanced fault detection approaches and underlie the motivation for this study.

In modern WTs, the converter controls the speed and torque of the generator in addition to the power transfer to the grid [39]. Moreover, the study of the converter is a relevant research topic currently because a failure in this component can cause an unexpected WT shutdown, resulting in a decrease in energy production. This situation is even more critical in wind farms located in complex terrain, which can lead to long downtimes of the WT.

In addition, early fault detection is crucial for WT maintenance, as it saves significant time and costs. Implementing hybrid models and effectively using SCADA data can improve the reliability and efficiency of predictive maintenance, leading to a reduction in wind farm O&M costs.

Finally,

Table 1. Summary of key studies on WT fault detection.

Ref. & Results	Learning Algorithms	Tools/Methods	Main Component	Results & Limitations
[31], 2019	Classification	BPNN	Converter	Accuracy = 99.98% in identifying converter faults. There is limited discussion on the scalability of the proposed strategy to handle a wider variety of fault types. Does not use SCADA data.
[39], 2020	Prediction	K-means, LSTM, FP-Growth, Wavelet packet transform (WPT)	Converter	The proposed strategy shows promise for assessing converter reliability and fits well with SCADA data. However, it does not show an early prediction of failure or a considerable time window before failure.
[25], 2021	Detection	AOC–ResNet50	Converter	With accuracy = 96.11%, the proposed model outperformed other models in detecting failures. However, it does not show the variables that have a high abnormality rate and does not test the model in large dataset periods.
[30], 2022	Transfer Learning	Balanced Joint Adaptive Network (BJAN), LSTM	WT	BJAN achieves accuracy = 99.3% in WT fault diagnosis. The study also addresses the detection of faults in the converter but does not present the time advance to failure and does not distinguish between faults and alarms.
[36], 2022	Detection	DAE	Blade	It demonstrates high performance with an F1 > 89.00%. The study focuses on a mechanical component of the WT and does not clearly describe the advance time to failure.
[35], 2023	Detection	Variational AE	WT	The paper focuses on detecting and diagnosing WT mechanical component failures. However, it does not present model evaluation metrics.
[37], 2023	Prediction	AE, AT-LSTM	IGBT	The results show that the model can dynamically reconstruct the variable related to the target component. However, it does not present a prediction system to determine when the failure will occur.
[24], 2024	Detection	Transformers	IGBT	The proposed models can predict failures in the target component. The study uses SCADA data from a high-altitude wind farm. Due to the model architecture, the fault detection system is complex.
[34], 2024	Prediction	RNN, LSTM	WT	Experimental results show that the model reconstructs and fits the normal data well. However, it does not detect faults in a specific target component.
[33], 2024	Detection	LSTM-AVAGMM	Gearbox	The model’s accuracy of 96.44% shows its efficiency in detecting failures in a WT mechanical component. It does not present a failure prediction system.

summarizes key studies on failure prediction, from which some conclusions can be drawn. Most studies do not indicate how far in advance the models predict the failure before it occurs. Furthermore, there is a small body of scientific research focused on predicting WT converter failures, while more studies are needed to address the prediction of failures in WTs located in complex terrains, such as the wind farm studied in this work. Therefore, our motivation is to help fill the knowledge gap mentioned above.

The remainder of the paper is structured as follows: Section 2 describes the methodological process, the model architecture, the failure prediction system, and the evaluation methods. Section 3 details the results of the evaluation of the fault prediction model. Finally, Section 4 offers conclusions and suggestions for future research.

2. Materials and Methods

Figure 1 shows the methodological process for failure prediction employed in this study. The figure describes the process from raw data acquisition to feature cleaning and selection, proposed model development, and fault prediction. The raw data undergoes a data cleaning and filtering process, with variables related to the target component then being selected. These variables are the input to the LSTM-MA-AE model, which reconstructs the output signal for each variable. Subsequently, the predicted signals enter a fault prediction system where they are compared with each actual signal, and the reconstruction error is used to calculate the abnormality score. This system generates an alarm when a significant discrepancy is detected, allowing it to predict and alert of the possibility of a failure.

2.1. Description of the Data Set and Study Area

This paper uses SCADA data from the VWF, located in Ecuador, between UTM coordinates 693,030 E 9,558,392 N and 693,526 E 9,556,476 N. The VWF is located in a mountainous area at approximately 2720 m.a.s.l., and includes 11 GOLDWIND GW70/1500 WTs of 1.5 MW nominal power with Direct Drive technology [10,15]. Based on the available information, there are few wind farms located at that altitude or higher.

The SCADA operational data and fault records correspond to WT2 of the VWF, recorded between 1 January 2014, and 31 October 2021, representing 386,288 records and 69 variables.

2.2. Data Processing

For data processing, raw files in .txt and .tmp format were initially taken from the SCADA system, processed, and concatenated into a unified file. The steps performed for data cleaning and filtering to ensure data integrity are described below.

• Variables with null values exceeding 10,000 records were removed from the dataset, as these could potentially skew the analysis.
• Variables that, based on domain knowledge, do not contribute significant features to the analysis were eliminated.
• Null values per complete row were imputed, representing a maximum of 7 non-significant records in the WT2 variables.
• Data were filtered to include only those with an active power greater than 0 and less than 1600, ensuring the WT is within the guaranteed operating range.
• The operating mode of the WT must be 5, which indicates, in this case, that it is not in power limitation or maintenance.
• The wind speed range was set between 3 m/s and 25 m/s, as these are the start and cut-off values of the GOLDWIND GW70/1500 WT, ensuring the data reflects typical operating conditions.
• Temperature outliers were eliminated: specifically, IGBT temperatures higher than 120 °C, considering the average is 60 °C.
• Temperatures should be greater than 0 °C, since no sub-zero temperatures have been recorded in the study area, which has an average historical temperature of approximately 15 °C.

This approach ensures that the data used for analysis and model training are consistent and high-quality, which is essential for accurate and reliable results.

After data processing, the final set was reduced to 317,323 records and 55 variables. That is, 17.85% of the data were eliminated.

2.3. Target Component

The converter failure analysis was performed following the recommendations of VWF operating technicians. A review of the maintenance records and SCADA alarms confirmed that the converter has the highest failure rate at 89.3% (see Figure 2a). This percentage differs from the 23% mentioned in [14] for converter failures, since the cited article includes failures of other components of the WT, such as the generator, gearbox, pitch system, among others. In our case, the WT is relatively new, so no failures in these other components have been recorded, resulting in a higher percentage of failures attributed to the converter.

Additionally, SCADA system alarms on the converter account for 86.4% of the total, significantly outnumbering those on the pitch system (see Figure 2b). In this study, only the converter and pitch components were analyzed because both required replacement as critical parts of the WT.

In addition, it is worth noting the conceptual difference between failures and alarms. A failure involves the shutdown of the WT, and generally requires the replacement of the affected component, while an alarm may cause a temporary shutdown of the wind turbine, but does not necessarily lead to the replacement of the component. Failures can stop production and cause significant losses. It is thus essential to predict when a failure will occur, allowing preventive measures to be taken to minimize the impact on production.

2.4. Feature Selection

As mentioned above, the SCADA dataset used in this study consists of 55 variables. The variables were reduced to prevent the model from becoming too complex and to thus improve the learning of the features related to the target component. For this purpose, the variables were selected based on Pearson’s correlation.

Figure 3 depicts a correlation matrix for several variables related to the operation of the wind turbines and their components, particularly the converter and the IGBT. This matrix shows the strength and direction of the relationships between pairs of variables, represented by colors, where red indicates a high positive correlation, and blue has a high negative correlation.

The variable igbt_temperature_max shows a strong positive correlation with several other variables, such as wind_speed_avg, grid_active_power_avg, generator_speed_avg, and ambient_temperature_avg. This suggests that the IGBT temperature is closely linked to the overall WT operation and environmental conditions.

Three converter faults were identified in the preliminary analysis, which prevented us from observing significant correlations between the IGBT failure variable and the other variables. To address this challenge, the 10,000 data prior to each failure were labeled under the assumption that the component degrades over time and indications of failure may exist in such data. This allowed the data set to be balanced and correlations to be observed. This procedure is similar to that employed in [24]. In addition, although the correlations observed are small, they provide insight into the variables that could be involved in the faults.

Despite the low direct correlation between IGBT failures and the other variables, including tags in the pre-failure data provides valuable insight into understanding conditions that could contribute to converter failures.

In summary, this study chose variables based on their correlation with the target variable, correlation with converter failures, correlation with converter alarms, and the authors’ domain knowledge. Thus, nine variables related to the target component were selected, and are shown in

Table 2. Variables selected for converter failure analysis.

N°	Variables	Description
1	grid_active_power_avg	Indicates the performance of the WT and the operating conditions of the converter.
2	generator_speed_avg	This is closely linked to the performance of the WT and can influence the converter’s behavior.
3	ambient_temperature_avg	Ambient temperature affects several components of the WT, including the converter.
4	grid_U1_avg	The average voltage in the grid can influence converter failures.
5	grid_I1_avg	This a key indicator of the operational status of the WT, and can influence converter failures.
6	topbox_temperature_max	This may indicate thermal problems affecting the converter.
7	igbt_temperature_max	Directly related to the target component.
8	rectifier_temperature_max	Critical converter status indicator.
9	step_up_igbt_temperature_max	Monitors the IGBT temperature.

.

This selection ensures that the model effectively captures the operating conditions of the target component, anticipates possible failures, and minimizes the detection of failures in other components, improving the WTs reliability and efficiency.

2.5. Data Splitting

In this study, our dataset consists of 317,323 records and 9 variables after initial filtering and feature selection. During the time series analysis of the variables, we identified a problem with sensor reconfiguration in April 2017, which changed the measurement range in some cases. Therefore, we only considered data from after this date because including earlier data could have a negative impact on model training. After removing the affected data, we retained 176,707 records and 9 variables, which accounts for 55.69% of the original data.

For data splitting, a period of approximately one year of normal operation, between April 2017 and April 2018, was selected for training, which allows capturing seasonal patterns and short and long term variations. Subsequently, the testing phase was performed using the data labeled with failures.

It is important to note that no failure data were included in the training, as the model focuses on reconstructing the input depending on the relationships with other variables. The objective is that the model attempts to reconstruct the expected behavior, with a high discrepancy potentially indicating an anomaly.

2.6. Data Scaling

To improve the performance of a DL model, it is essential to standardize or normalize the data before feeding it into the model. These preprocessing techniques help the model learn more efficiently and effectively.

Standardization transforms each input variable $\mathbf{x}$ into a distribution with mean zero and standard deviation one, as shown in Equation (1). In this equation, $\tilde{x}$ represents the mean of the variable $\tilde{x}$, and $\sigma$ represents the standard deviation of $\tilde{x}$.

$$x={\displaystyle \frac{\tilde{x}-\mu }{\sigma }}$$

(1)

where $\mu$ is the mean of the original data.

This technique ensures that all features have a comparable scale, which can significantly improve the convergence and performance of the DL model. In this study, the StandardScaler function from the Scikit-learn library was used to perform data standardization.

After the model’s predictions were obtained, it was necessary to reverse the standardization process to interpret the results in their original scale. This step, known as back-standardization or rescaling, is crucial for understanding the model’s outputs in a practical context. This rescaling was implemented using the inverse transform function of the StandardScaler from the Scikit-learn library.

2.7. Long Short-Term Memory (LSTM) Network

LSTM networks can improve the fusion of temporal features from the state of different parts. This study combines our model with LSTM to extract temporal features from WT data. The LSTM contains an input gate, an output gate, and a forget gate. Its structure is shown in Figure 4.

The information to be forgotten is controlled by forget gate $f_t$, defined as:

$$f_t=\sigma (W_f{x}_t+W_h{h}_{t-1}+b_f)$$

(2)

where $\sigma$ is the sigmoid function, and $W_f$ and $b_f$ represent the weight matrix and the bias of the forget gate, respectively. In the next step, input gate $i_t$ receives $h_{t-1}$ and $x_t$ to determine the new data that should be stored in the cell state. At the same time, a vector of candidate values ${\tilde{C}}_t$ is created by a hyperbolic tangent layer (tanh), which returns a value between −1 and 1. The previous cell state $C_{t-1}$ is then updated to the new cell state $C_t$, as described in Equations (3)–(5) [40].

$$i_t=\sigma (W_i{x}_t+W_h{h}_{t-1}+b_i)$$

(3)

$${\tilde{C}}_t=tanh(W_c{x}_t+W_h{h}_{t-1}+b_c)$$

(4)

$$C_t=f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t$$

(5)

Finally, output gate $o_t$ determines the hidden state $h_t$ as follows:

$$o_t=\sigma (W_o{x}_t+W_h{h}_{t-1}+b_o)$$

(6)

$$h_t=o_t\odot tanh\left(C_t\right)$$

(7)

where $W_o$ and $b_o$ are the weight matrix and the output gate bias, respectively. Output $h_t$ is obtained according to the updated state of cell $C_t$.

2.8. Autoencoder (AE)

The AE is an unsupervised deep neural network that reconstructs the input data with minimal error based on the encoded data. The input data are encoded in this network by mapping it to a low-dimensional space [41]. It then attempts to minimize the loss between the input and decoded data. As illustrated in Figure 5, the AE generally consists of two parts: an encoder and a decoder.

The encoder transforms the input data $\mathbf{x}$ into the encoded representation $\mathbf{z}$ (i.e., code layer). According to the depicted architecture, the mathematical formulation for the code layer encoding process is given by Equation (8); thus,

$$\mathbf{z}=\varsigma ({\mathbf{W}}_{\mathbf{2}}\mathbf{h}+{\mathbf{b}}_{\mathbf{2}})$$

(8)

where $\mathbf{W}$ and $\mathbf{b}$ are the weight matrix and the network bias vector, respectively. Next, the decoder attempts to reconstruct the input data from code layer $\mathbf{z}$ with the smallest discrepancy between the original input $\mathbf{x}$ and the reconstructed output $\widehat{\mathbf{x}}$, calculated based on the following equation:

$${\mathbf{x}}^{\prime }=\varsigma ({\mathbf{W}}_{\mathbf{4}}{\mathbf{h}}^{\prime }+{\mathbf{b}}_{\mathbf{4}}))$$

(9)

where $\varsigma$ is the activation function.

2.9. Multi-Head Attention (MA)

AM has become a vital component in neural networks for handling long sequential data. By computing attention weights, the network learns to focus on the most significant parts of the input. An important innovation was introduced with MA, proposed by Vaswani et al. [42]. This approach improves attention by using multiple parallel attention layers or “heads” to focus on different input segments. MA greatly improves the modeling of complex dependencies in the data and increases model performance.

Unlike conventional attention models, which compute attention scores between a query vector and key-value pairs from the input, self-attention generates the query, key, and value vectors through transformations of the input itself. This allows the model to effectively extract important features and relationships within the data through self-reference. Specifically, self-attention compares different items within a single input sequence against each other. It computes interaction scores between a query matrix Q, a key matrix K, and value matrices V from the input data using Equation (10).

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(10)

where $d_k$ is the dimension of the key matrix. MA extends this by employing multiple independent attention heads, with each head learning distinct patterns. Multiple heads essentially allow for parallelization within the attention layer, providing a richer representation of the input.

As depicted in Figure 6, the input sequence is linearly projected into Q, K, and V by learned weight matrices $W_i^Q$, $W_i^K$, and $W_i^V$. Then, Q and K are multiplied and scaled to obtain the attention scores. The attention weights are multiplied by V to obtain the output values of each head $At{t}_h$. These output values, which constitute the attention of each head $At{t}_h$, are concatenated and linearly projected using a learned matrix $W^O$ to obtain the final attention output of multiple heads $Output$. This process is described by Equations (11) and (12).

$$At{t}_h=\mathrm{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$$

(11)

$$Output(Q,K,V)=\mathrm{Concat}(At{t}_1,At{t}_2,\dots ,At{t}_n)W^O$$

(12)

While masking is an optional component that can be applied to the attention scores before the softmax to prevent certain positions from attending to others, it was not used, as the full sequence context was preferred to capture all global dependencies and might exclude patterns helpful in detecting small changes in the entire sequence.

2.10. Positional Encoding (PE)

PE is an essential technique in Transformers that incorporates information about the position of input sequences. Unlike RNNs, which process sequence data in an ordered fashion, Transformers, with their AM, process sequences in a parallel fashion, which means that the position of elements within the sequence is not implicit in the model [43]. To address this limitation, PE is introduced, which adds positional information to the input vectors to the attention layer. This encoding is performed using trigonometric functions (sine and cosine) to generate a unique vector for each position, which makes it easier for the model to capture the positional relationship of the elements in the sequence.

Equations (13) and (14) are used in this process.

$$P{E}_{(pos,2i)}=sin\left({\displaystyle \frac{pos}{{10000}^{2i/d_{model}}}}\right)$$

(13)

$$P{E}_{(pos,2i+1)}=cos\left({\displaystyle \frac{pos}{{10000}^{2i/d_{model}}}}\right)$$

(14)

where $pos$ is the position in the sequence, i is the dimension, and $d_{model}$ is the size of the feature vector.

2.11. Hybrid Model LSTM-MA-AE

This model efficiently captures the spatial and temporal characteristics of SCADA data, improving fault prediction and diagnosis capabilities. The LSTM-MA-AE model developed in this study processes the input signals to reconstruct the time series and predict faults before they occur.

The LSTM-MA-AE model has two main parts: the encoder and the decoder. The structure of the model is presented in Figure 7. In the encoder, a combination of LSTM layers, PE, and MA processes the input sequences. The decoder uses a similar structure to the encoder to reconstruct the time series from the encoded representations. This means that two LSTM layers with 64 hidden units each are employed in the encoder, which are responsible for capturing complex short-term temporal dependencies in the data sequence, embedding patterns relevant to the reconstruction task. These layers are followed by normalization and a dropout of 0.1 to prevent overfitting. At the output of the LSTMs, a PE is integrated for the 8-head MA mechanism to understand the positions of the time series data. These attention features are added to a residual connection from the previous output so that no data information is lost. Each head of the MA allows the model to focus on different aspects of the input sequence simultaneously, enhancing the capability of long-term feature embedding and temporality.

Furthermore, in the decoder, we initiate the extraction of the encoder embedded features using an 8-head MA mechanism supplemented with two additional 64-layer hidden LSTM layers. Initially, zero tensors are used as the query in the MA mechanism, whereas the encoder embeddings are employed as a key and value. These are then passed to the LSTM layers, which process the historical values of the dataset along with the attention output, allowing the relevant sequential structure to be extracted and the first value to be predicted through a fully connected layer. This output is subsequently used as feedback in the query the MA mechanism for subsequent predictions. In this way, the predictions generated by the decoder are iteratively employed as a query, achieving accurate reconstruction throughout the entire sequence. Finally, we apply normalization and dropout to optimize training and prevent model overfitting.

The input to the model is a time series of length T data with d characteristics:

$$\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_T]\in {\mathbb{R}}^{T\times d}$$

(15)

The temporal sequence is then encoded through a 2-layer LSTM with dropout in between and a normalization layer LayerNorm at the end of the second layer, to obtain the encoded representation, as represented in the following Equations (16) and (17):

$${\mathbf{H}}_{\mathrm{out}}={\mathrm{LSTM}}_{\mathrm{enc}}\left(\mathbf{X}\right)$$

(16)

$${\mathbf{H}}_{\mathrm{enc}}=\mathit{LayerNorm}\left({\mathrm{LSTM}}_{\mathrm{enc}}\left({\mathbf{H}}_{\mathrm{out}}\right)\right)$$

(17)

Output series ${\mathbf{H}}_{\mathrm{enc}}$ has PE added to it so that MA can understand the position of the time series, as shown in Equation (18).

$${\mathbf{H}}_{\mathrm{PE}}={\mathbf{H}}_{\mathrm{enc}}+\mathit{PE}$$

(18)

Once PE is added, the MA is calculated to capture complex relationships in the time sequence, as shown in Equations (19)–(22).

$$\mathbf{Q}={\mathbf{H}}_{\mathrm{PE}}{W}^Q$$

(19)

$$\mathbf{K}={\mathbf{H}}_{\mathrm{PE}}{W}^K$$

(20)

$$\mathbf{V}={\mathbf{H}}_{\mathrm{PE}}{W}^V$$

(21)

$${\mathbf{att}}_{\mathbf{enc}}=\mathrm{Multi}\text{-}\mathrm{HeadAttention}(\mathbf{Q},\mathbf{K},\mathbf{V})$$

(22)

where $W^Q$, $W^K$, $W^V$ are matrices of learnable weights of query Q, key K, and value V.

To avoid losing data information, a residual connection is made from the output of ${\mathbf{H}}_{\mathrm{PE}}$ to the output of the MA, using Equation (23).

$${\mathbf{out}}_{\mathbf{enc}}={\mathbf{H}}_{\mathrm{PE}}+{\mathbf{att}}_{\mathbf{enc}}$$

(23)

The encoded representation ${\mathbf{out}}_{\mathbf{enc}}$ is first decoded by means of an MA, which uses

$${\mathbf{att}}_{\mathbf{decod}}=\mathit{Multi}\text{-}\mathit{HeadAttention}({\mathbf{H}}_{\mathbf{t}-\mathbf{1}},{\mathbf{out}}_{\mathbf{enc}},{\mathbf{out}}_{\mathbf{enc}})$$

(24)

where ${\mathbf{H}}_{\mathbf{t}-\mathbf{1}}$ is the previous hidden layer of the LSTM unit at the end of the decoder, which at the beginning is a tensor of zeros. Next, the MA output of the decoder is concatenated with the historical data ${\mathbf{R}}_{\mathbf{t}}$ to be reconstructed:

$${\mathbf{combined}}_{\mathbf{att}}=\mathit{Concat}({\mathbf{att}}_{\mathbf{decod}},{\mathbf{R}}_{\mathbf{t}})$$

(25)

A fully connected layer FL is then used to learn even more of the data representation, and the output of this is used as input to two LSTM layers, followed by LayerNorm:

$${\mathbf{H}}_{\mathbf{t}},{\mathbf{out}}_{\mathbf{decod}}=\mathit{LayerNorm}\left(\mathit{LSTM}\left(\mathit{LSTM}\left(\mathit{FL}\left({\mathbf{combined}}_{\mathbf{att}}\right)\right)\right)\right)$$

(26)

where ${\mathbf{H}}_{\mathbf{t}}$ feeds as a Q back to the MA to obtain the reconstructed predicted data. Finally, the decoder output is projected to the original dimension with a FL to reconstruct the $\widehat{\mathbf{Z}}$ time series, using Equation (27).  

$$\widehat{\mathbf{Z}}=\mathit{FL}\left({\mathbf{out}}_{\mathbf{decod}}\right)$$

(27)

The pseudocode for LSTM-MA-AE is presented in Algorithm 1.

Algorithm 1 Training procedure for LSTM-MA-AE with Early Stopping

1: Input: SCADA data, where $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_T]\in {\mathbb{R}}^{T\times v}$, T is the time series length and v is the number of variables. 2: Initialize: 3:    Load configuration parameters: hidden layers $hl$, batch size $bs$, epochs e, patience p 4:    Initialize model, loss criterion $\mathcal{L}$, optimizer Adam, and learning rate $Lr$ 5:    Initialize variables: best_val_loss = ∞, patience_counter = 0, ${\mathcal{L}}_{val}=0$ 6:    Load $X_{training}$, $X_{test}$, $X_{val}\in {\mathbb{R}}^{(e,q,v)}$, where e is the number of sequences and q is the sequence length. 7: Create DataLoader iterators for training and testing data with $bs$ 8: for each epoch e do 9:    Training Phase: 10:       Set model to training mode 11:       Initialize training loss to zero 12:    for each batch in $X_{training}$ do 13:         Zero the parameter gradients 14:         $output$ = LSTM-MA-AE($batch$) 15:         Compute loss $\mathcal{L}=criterion(output,batch)$ 16:         Concatenate $params\_without\_bias=Concat\_params\left(LSTM\text{-}MA\text{-}AE\right)$ 17:         Compute regularization term ${\mathcal{L}}_{reg}=\mathcal{L}+Reg·L{1}_{norm}\left(params\right)$ 18:         Perform backpropagation 19:         Update model parameters ${\mathcal{L}}_{reg}.backward\left(\right)$ 20:         Accumulate training loss ${\mathcal{L}}_{train}+={\mathcal{L}}_{reg}$ 21:    end for 22:       Compute average training loss ${\mathcal{L}}_{train\_avg}={\mathcal{L}}_{train}/\mathrm{number}\mathrm{of}\mathrm{batches}$ 23:    Validation Phase: 24:       Set model to evaluation mode 25:    for each batch in $X_{val}$ do 26:         Disable gradient computation 27:         $outpu{t}_{val}$ = LSTM-MA-AE($batch$) 28:         Compute loss ${\mathcal{L}}_{val}=criterion(outpu{t}_{val},batch)$ 29:         Accumulate validation loss ${\mathcal{L}}_{val}+={\mathcal{L}}_{val}$ 30:    end for 31:       Compute average validation loss ${\mathcal{L}}_{val\_avg}={\mathcal{L}}_{val}/\mathrm{number}\mathrm{of}\mathrm{batches}$ 32:    Early Stopping: 33:    if ${\mathcal{L}}_{val\_avg}<$ best_val_loss then 34:         Update best_val_loss $={\mathcal{L}}_{val\_avg}$ 35:         Save model parameters and reset patience_counter $=0$ 36:    else 37:         Increment patience_counter $+=1$ 38:    end if 39:    if patience_counter $>p$ then 40:         Break training loop early 41:    end if 42: end for 43: Test the model using $X_{test}$ 44:    Reconstruction data $outpu{t}_{test}$ = LSTM-MA-AE($X_{test}$) 45:    System of prediction faults 46:    Loss of each variable ${\mathcal{L}}_{test}=criterion(outpu{t}_{test},X_{test})$ 47:    Smoothing ${\mathcal{L}}_{test}$ 48:    Apply threshold with EWMA and sliding window anomaly detection 49:    Fault warning;

2.12. Exponential Weighted Moving Average (EWMA)

Within the failure prediction mechanism, EWMA is used. This technique applies greater weight to more recent data with higher errors. This technique is defined by Equation (28) [44].

$${\mathrm{EWMA}}_t=\alpha a_t+(1-\alpha ){\mathrm{EWMA}}_{t-1}$$

(28)

where $\alpha$ is the smoothing factor, $a_t$ is the value of the reconstruction loss in time t and ${\mathrm{EWMA}}_{t-1}$ is the value of EWMA in time $t-1$.

The threshold value at the upper bound of EWMA is calculated as shown in Equation (29).

$${\mathrm{threshold}}_{EWMA}=\mathrm{EWMA}+\lambda {\theta }_{\mathrm{EWMA}}$$

(29)

where ${\theta }_{\mathrm{EWMA}}$ is the standard deviation of the EWMA values and $\lambda$ is a constant that defines the position of the threshold. When training with normal data, $\lambda$ can be adjusted to set an appropriate threshold.

2.13. Fault Prediction System

Initially, the model reconstructs all input variables to capture deviations in each one. As depicted in Figure 8, the fault prediction system calculates the reconstruction score for each data point for each variable. A smoothing vector is then used to distinguish between anomalies possibly caused by degradation or situations where the WT is operating at its power limits. Next, this new score is passed through a threshold with EWMA. If the score exceeds this threshold, a label vector is created, where 1 indicates that the threshold has been exceeded and 0 indicates that it has not.

This label vector is processed through a sliding window anomaly detector to improve anomaly labeling. This detector takes a binary vector as input and identifies if there are more than $N_{\mathrm{anomalies}1}$ anomalies within a time window $N_{\mathrm{window}1}$, then it labels the entire window between the first and last anomaly within the window as anomalous (1). This method assumes that nearby anomalies within a maximum window size of $N_{\mathrm{window}1}$ can represent the same fault.

Once each variable’s final label vector is obtained, all vectors are concatenated and added. This time, the number of variables with anomalies that would indicate a failure warning is determined using a fixed threshold.

For each variable $v_i$ in the data set, we perform the following:

• Calculation of the reconstruction score for each variable $v_i$:

  $$L_v\left(t\right)=|y_i-{\widehat{y}}_i|$$

  (30)

  where $L_v\left(t\right)$ is the reconstruction score over time t, $y_i$ represents the actual values, and ${\widehat{y}}_i$ the predicted values.
• Calculation of the smoothed reconstruction vector $L_s\left(t\right)$:
  Consider $P\left(t\right)$ the nominal power at instant t and $P_{90}$ the 90th percentile value of the maximum power. We define the smoothing vector $V_s\left(t\right)$ as follows:

  $$V_s\left(t\right)=\left\{\begin{array}{cc}0.85\hfill & \mathrm{if}P\left(t\right)>P_{90}\hfill \\ 1\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (31)

  $$L_s\left(t\right)=V_s\left(t\right)·L_v\left(t\right)$$

  (32)
• Calculation of EWMA:

  $$S_i\left(t\right)=\alpha ·L_s\left(t\right)+(1-\alpha )·L_s(t-1)$$

  (33)

  where $S_i\left(t\right)$ is the EWMA-smoothed score.
• Comparison with the threshold:

  $$R_i\left(t\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{L}_s\left(t\right)>{\mathrm{threshold}}_{EWMA}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (34)

  where $R_i\left(t\right)$ is the vector of labels indicating an anomaly.
• Improvement of detection with the sliding window anomaly:
  For a window size maximum of $N_{\mathrm{window}1}$:

  $$R_i\left(t\right)=\left\{\begin{array}{cc}1\hfill & \mathrm{for}t\in [t_1,t_n]\mathrm{if}{N}_{\mathrm{anomalies}1}\le \sum _{t_1}^{t_n}{R}_i\left(t\right)\mathrm{within}{N}_{\mathrm{window}1}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (35)

  where $t_1$ and $t_n$ are the temporal positions of the start and end of the anomalies within the size window $N_{\mathrm{window}1}$. If two anomalies are found within this window, all data points between $t_1$ and $t_n$ are labeled as anomalous.
• Adding and fix the threshold:

  $$T_{\mathrm{add}}\left(t\right)=\sum _{i=1}^v{R}_i\left(t\right)$$

  (36)

  where v is the number of variables.

  $$\mathrm{Fault}\mathrm{Warning}=\left\{\begin{array}{cc}Warning\hfill & \mathrm{if}{T}_{\mathrm{add}}\left(t\right)>{\mathrm{threshold}}_{\mathrm{fault}}\hfill \\ normal\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (37)

  where ${\mathrm{threshold}}_{fault}$ represents the final fixed threshold of failure.

2.14. Model Hyperparameters

In this study, the early stopping technique is adopted during the training process to avoid overfitting the model. Overfitting can prevent the model from adequately understanding the temporal and spatial relationships between variables, as it fits too closely to the true values and loses generalization ability. Early stopping ensures that the model fits with small errors in normal cases. In contrast, in anomalous situations, the model cannot fit the input features, resulting in a significant discrepancy with the normal data. This improves the model’s ability to detect anomalies effectively.

The hyperparameters with which the model achieved the best performance are shown in

Table 3. Hyperparameters used in LSTM-MA-AE.

Parameters	Value
Batch size	256
Sequence length	20
Dropout	0.1
Hidden size LSTM	64
Learning rate	$1\times {10}^{-6}$
Epochs	75
Regularization factor	$1\times {10}^{-4}$
LSTM layers encoder	2
LSTM layers decoder	2
Heads encoder	8
Heads decoder	8

.

2.15. Model Evaluation Metrics

To evaluate the proposed model, we used different error metrics, common in prediction analysis, such as the root mean square error (RMSE), the mean absolute error (MAE), and the coefficient of determination ($R^2$). Equations (38)–(40) define these metrics.

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}$$

(38)

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n|y_i-{\widehat{y}}_i|$$

(39)

$$R^2=1-{\displaystyle \frac{{\sum }_{i=1}^n{(y_i-{\widehat{y}}_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}}$$

(40)

where $y_i$ represents the actual values, ${\widehat{y}}_i$ the predicted values, n is the number of samples, and $\overline{y}$ is the mean value of $y_i$.

In addition, we use the Precision ($Pr$), Recall ($Re$), and F1-Score ($F1$) metrics, which are commonly employed in anomaly detection [45]. These metrics are defined by Equations (41)–(43).

$$Pr={\displaystyle \frac{TP}{TP+FP}}$$

(41)

$$Re={\displaystyle \frac{TP}{TP+FN}}$$

(42)

$$F1={\displaystyle \frac{2·Pr·Re}{Pr+Re}}$$

(43)

where True Positive ($TP$) represents alarms that detected a fault correctly within a specific time window, False Positive ($FP$) corresponds to alarms that do not result in an actual fault, and False Negative ($FN$) indicates faults that were not correctly detected.

To conclude this section, the experiments were conducted using the Python programming language in the PyTorch library within a Google Colab environment. The environment had an Intel(R) Xeon(R) CPU @ 2.20 GHz, 51.00 GB of RAM, a Tesla K80 accelerator, and 12 GB of GDDR5 VRAM.

3. Results

This section presents the results in two subsections. Section 3.1 analyzes the model’s ability to generalize the temporal data for each feature. Section 3.2 describes the performance of the model in predicting faults and the advance time until the occurrence of faults.

3.1. Model Evaluation

The results shown in Table 4 were achieved using the hyperparameters outlined in Table 3. We evaluated three model configurations:

• The LSTM-MA-AE approach.
• The same approach without the AM (LSTM-AE).
• A version without PE and residual connection.

This enables us to confirm that the improve implemented effectively enhance the performance of the proposed model.

The data presented in Table 4 indicates that incorporating MA into the LSTM-MA-AE approach is highly effective for capturing the complexities within temporal data. It shows that the AM enhances model performance. The MA assists the model in comprehending a wide range of sequences, thereby retaining crucial information necessary for effectively reconstructing the time series. In addition, the LSTM-MA-AE approach without positional coding shows inferior performance compared to the approach that includes PE. This is because the addition of PE makes it easier for the model to understand the positional characteristics of each data in the time series, improving its ability to interpret and predict accurately.

The $R^2$ coefficient shows the difference between the series; a value of $R^2$ close to 1 reflects a high similarity between the data sets. The MAE measures the loss between two data points, while the RMSE penalizes outliers. As can be seen, the best performance was obtained by the LSTM-MA-AE model.

Table 4. Comparison of variable reconstruction results.

Feature	LSTM-MA-AE			LSTM-AE			LSTM-MA-AE without PE
	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$
3	0.44	0.33	0.91	1.50	1.22	−0.09	13.12	13.09	−80.65
2	0.52	0.37	0.88	1.71	1.35	−0.23	20.35	20.32	−143.71
6	0.80	0.67	0.84	2.64	2.24	−0.69	36.92	36.18	−19.17
8	1.55	1.17	0.82	3.34	2.68	0.18	58.62	56.84	−13.99
9	1.75	1.38	0.94	6.99	5.86	0.11	70.15	68.15	−15.24
7	3.63	2.60	0.93	16.28	13.15	−0.35	358.37	358.36	−9026.75
4	4.21	3.17	0.93	22.23	16.44	−0.88	1169.89	1052.11	−4.21
5	125.20	96.16	0.92	326.12	260.14	0.51	1094.50	986.58	−4.32
1	137.11	109.85	0.91	453.77	361.64	−0.12	1169.89	1052.11	−4.21

Note: In bold are the best results.

Table 4. Comparison of variable reconstruction results.

Feature	LSTM-MA-AE			LSTM-AE			LSTM-MA-AE without PE
	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$	RMSE	MAE	${\mathit{R}}^{\mathbf{2}}$
3	0.44	0.33	0.91	1.50	1.22	−0.09	13.12	13.09	−80.65
2	0.52	0.37	0.88	1.71	1.35	−0.23	20.35	20.32	−143.71
6	0.80	0.67	0.84	2.64	2.24	−0.69	36.92	36.18	−19.17
8	1.55	1.17	0.82	3.34	2.68	0.18	58.62	56.84	−13.99
9	1.75	1.38	0.94	6.99	5.86	0.11	70.15	68.15	−15.24
7	3.63	2.60	0.93	16.28	13.15	−0.35	358.37	358.36	−9026.75
4	4.21	3.17	0.93	22.23	16.44	−0.88	1169.89	1052.11	−4.21
5	125.20	96.16	0.92	326.12	260.14	0.51	1094.50	986.58	−4.32
1	137.11	109.85	0.91	453.77	361.64	−0.12	1169.89	1052.11	−4.21

Note: In bold are the best results.

During training, we set the maximum number of epochs to 75. However, using early stopping, the model reached its best performance at epoch 25, when the validation loss stopped improving significantly. As shown in Figure 9a, the validation loss decreases rapidly during the first 25 epochs and then stabilizes. Early stopping acted from this point to avoid overfitting, which could have occurred if training had continued up to the initially configured epochs.

This analysis is crucial to determine the optimal point where the model generalizes best without overfitting, thus ensuring robust performance on unseen data. Figure 9a depicts the behavior of losses during the training epochs, highlighting the importance of early stopping in model optimization.

Furthermore, as seen in Figure 9a, a batch size of 256 was selected due to its comparable performance in terms of loss with smaller batch sizes, such as 128, but with a lower computational time, as shown in Figure 10. In Figure 9b, the sequence length is analyzed, and although sequence length 10 presents slightly better performance than sequence length 20, the latter was chosen, as it provides more history to the model, allowing more general learning of the time series. In Figure 9c, it is shown that regularization factors of 1 $\times {10}^{-4}$ and 1 $\times {10}^{-6}$ provide similar results, so a value of 1 $\times {10}^{-4}$ was chosen. Figure 9d shows that increasing the number of heads improves the capture of long-term patterns in the time series. Figure 9e shows that a single layer presents poor performance, while a larger number of layers, such as two or three, reduces the loss but increases the computational cost. Figure 9f shows that a higher dropout affects the model’s generalization capability, with a value of 0.1 being the most appropriate in this case. Figure 9g reveals that a hidden layer size of 128 improves the fit to the real values, although it increases the computational cost and time, as shown in Figure 10. Finally, Figure 9h shows that the initial value of the learning rate, together with the scheduler, significantly influences the model fit. We aim to generalize the time series with a good fit, without overfitting, to identify anomalous patterns over time. Although a higher learning rate would be adequate for forecasting tasks, in this case, we need one that allows us to generalize the time series and obtain adequate results, as presented in Table 4.

Table 5. Comparison of training speed by epoch.

Approach	Time by Epochs
LSTM-AE	7.48 s
LSTM-MA-AE without PE	13.40 s
LSTM-MA-AE	13.28 s

indicates that adding an attention mechanism increases the time required for model training. This is due to the increased computations and the need to learn more complex features. The LSTM-MA-AE approach has a time per epoch of 13.28 s, while the LSTM-AE approach, without the AM, requires only 7.48 s per epoch. Furthermore, the use of PE in the LSTM-MA-AE approach slightly reduces the training time compared to the version without PE, with times of 13.28 s and 13.40 s per epoch, respectively. This suggests that PE and residual connection help make training slightly faster.

These results underscore the importance of balancing model accuracy with training time. Although the attention mechanism may increase training time, it can significantly improve model performance by capturing complex features and retaining crucial information in sequential data.

Additionally, the impact of the amount of training data on the overall model performance was evaluated, as shown in

Table 6. Model performance for different dataset percentages.

Percent Data	Variable 3 (MAE)
90%	0.35
50%	1.45
25%	3.62

. The model for variable $ambient\_temperature\_avg$ accurately reconstructs the data even with a reduced dataset, yielding promising results.

Figure 11a displays the reconstruction of the IGBT maximum temperature variable. This figure shows an increase in temperature from early May to late June 2018. This trend coincides with a converter failure documented in the maintenance record. This fault may be due to an accelerated degradation of the converter since, during that period, according to the power output log, the WT operated at maximum capacity most of the time.

Figure 11 reveals that the true time series and the model reconstruction reflect this increase in temperature. However, when examining the reconstruction loss (errors) shown in Figure 11b, several anomalies exceeded the threshold established prior to the failure event. Although this reconstruction loss is not significant in absolute terms, it is noteworthy when compared to normal values.

3.2. Fault Prediction

Detecting faults in WT components may depend on anomalies in one or more variables related to the target component. In addition, the presence of indications of faults in SCADA variables may not be evident, especially in electronic components. Therefore, a fault prediction system was developed to more accurately detect and anticipate these anomalies.

After implementing the failure prediction system, we obtained Figure 12, demonstrating that the prediction system can provide alerts about potential failures up to two months in advance. Figure 12 also displays two alert events that preceded failures recorded in the maintenance file: one on 10 June, and another on 24 June 2018. These alerts occurred consecutively, indicating the system’s effectiveness in identifying impending failures.

Figure 13, which is an enlargement of the window shown in Figure 12, shows that the model with the fault prediction system with an additional anomaly sliding window at the end of the stage can alert of the presence of anomalies with an average advance of approximately four months for the failures that occurred on 10 and 24 June 2018, and 2 March 2021. However, it is important to note that the model also alerted of anomalies at the end of November 2018, although there was no failure according to the maintenance record.

The fixed threshold and an adaptive threshold with EWMA were used to reduce the noise generated by the fixed threshold. A limitation of using a fixed threshold is that it can only capture anomalies when there is a trend in the reconstruction loss. Therefore, in this study, since it is necessary to determine anomalies in the WT operation over its entire operating range, an adaptive threshold with EWMA is used.

Figure 14 shows the performance of the adaptive threshold with EWMA applied to each reconstruction error of each variable, which improved fault detection and reduced false positives.

Figure 15 shows the final result of the failure prediction system. The model correctly detected the converter failures approximately 2.5 and 4.2 months before they occurred. This demonstrates the proposed model’s capability for early fault detection.

In addition, as seen in Figure 15, an warning at the end of 2019 corresponds to an FP. According to the maintenance record, this event may be due to a failure in the yaw system, and the model could have captured these deviations.

The study was also conducted for WT3 of the VWF to determine the effectiveness of the proposed model. Figure 16 validates this failure prediction method because the model detected anomalies approximately 2.3, 3.7, and 4.3 months in advance of the failures, which occurred on 25 May 2019, 23 December 2019, and 31 March 2021, respectively.

The metrics described in Section 2.15 were used to evaluate the model’s performance.

Table 7. Fault prediction system performance metrics.

WT	TP	FP	FN	Re	Pr	F1	Adv. 1	Adv. 2	Adv. 3	Adv. Avg.
WT2	2	1	0	1.00	0.67	0.80	2.50	-	4.20	3.35
WT3	3	0	0	1.00	1.00	1.00	2.30	3.70	4.30	3.43
									Avg.	3.39

shows that the model is able to anticipate failures an average of 3.39 months in advance for both WTs. The advance to fault (Adv.) is expressed in months.

Finally, we have developed a comparison with similar recent studies in WT converter failure prediction. Although direct comparisons are limited due to variability in the datasets used in each study, this comparison provides an overview of our model’s relative performance in relation to previous work. In

Table 8. Impact matrix comparing this study with previous studies.

Criterion	[37]	[24]	Our Work
Fault prediction (F1)	-	1.00	0.90
Early fault detection	Medium-High	High	Medium-High
Computational efficiency	High	High	High
Use of Multi-Head Attention	No	Yes	Yes
Application in complex terrains	No	Yes	Yes
Advance	4.32 months	4 months	3.39 months
Limit to failure	3 months	3 months	2.30 months

Note: In bold are the best performance.

, an impact matrix summarizing the contributions of each study is presented.

4. Conclusions

Since electronic converters are critical components, the ability to predict failures months in advance is crucial to reducing O&M costs. Therefore, this study addresses the prediction of converter failures in WTs located in complex terrain.

We developed a hybrid model using SCADA data that combines LSTM, Multi-head Attention, and Autoencoder, using their strengths to learn temporal and spatial characteristics between variables. The LSTM provides better generalization, while the Multi-head Attention mechanism captures complex patterns inherent in the data. The Autoencoder mechanism allows the model to reconstruct features with spatial and temporal information, facilitating the detection of malfunctions in variables related to the target component.

The LSTM-MA-AE model can predict failures an average of approximately 3.3 months in advance and with an average F1 of 90% in the evaluated WTs, showing a low false positive rate. These medium-term failures could be indicative of converter degradation.

This method can facilitate early fault detection and provides a robust mechanism for predictive maintenance in wind farms, ensuring greater reliability and generation efficiency. Moreover, we are confident that the methodological process we have developed could be successfully replicated in other wind farms. In fact, we believe it could even be applied to other components of the WT, not just the converter, further extending the reach of our research.

As a future work, it is proposed to use the parallelization capabilities of the Transformer model and the strengths of the self-attention mechanism to determine the variables that are intrinsically related to the component failure and to determine the root cause.