WindFormer: Learning Generic Representations for Short-Term Wind Speed Prediction

Abstract

In this paper, we introduce WindFormer, an innovative transformer-based model engineered for short-term wind speed forecasting, leveraging multivariate time series data. Unlike traditional approaches, WindFormer excels in processing diverse meteorological features—temperature, humidity, and power—to intricately map their spatiotemporal interdependencies with wind speeds. Utilizing a novel unsupervised pre-training strategy, WindFormer initially learns from vast volumes of unlabeled data to capture generalized feature representations. This foundation enhances the subsequent fine-tuning phase on labeled wind speed data, in which our model demonstrates exceptional predictive accuracy. Empirical evaluations across various public datasets illustrate that WindFormer markedly surpasses both conventional statistical models and contemporary deep learning techniques. The model not only achieves superior accuracy in forecasting wind speeds but also reveals a significant enhancement in handling complex spatiotemporal data dynamics. These advancements facilitate more effective wind farm management and power grid scheduling, making a substantial impact on operational efficiencies and renewable energy utilization. Our findings confirm the robustness of WindFormer in a real-world setting, underscoring its potential as a pivotal tool in meteorological and energy sectors. The integration of unsupervised pre-training with multi-task fine-tuning establishes a new benchmark for short-term wind speed prediction.

1. Introduction

Accurate short-term wind speed prediction is critical for applications in wind energy generation [1,2], aerospace [3,4], meteorological warnings [5], and agriculture [6,7]. Despite progress, the inherent complexity and unpredictability of wind speed variations continue to challenge the efficacy of existing prediction models [8,9].

Traditional methods, primarily reliant on physical or statistical models, demonstrate significant limitations, particularly in handling large-scale, multivariate datasets. These conventional approaches often fail to capture the intricate patterns of wind variability that are crucial for precise predictions. This research directly addresses these shortcomings by introducing a novel transformer-based approach, the WindFormer model [10], which is specifically designed to manage the dynamic nature of wind speed data with revolutionary efficiency and accuracy.

Recent advancements in deep learning have substantially improved the precision of time series forecasting. Techniques employing neural networks, such as convolutional neural networks (CNNs) [11] and recurrent neural networks (RNNs) [12], have shown remarkable prowess in wind speed predictions. However, these models often cater to specific datasets and lack the flexibility needed for generalizations across diverse tasks and conditions. Moreover, with the evolution of sensor technology and the expansion of data collection methods [13], the accumulation of rich multivariate time series data—encompassing variables like temperature, humidity, and power—presents new hurdles for traditional forecasting models [14].

To navigate these complexities, we have developed the Wind Speed Transformer model (WindFormer), an architecture tailored to the precise prediction of short-term wind speeds. WindFormer employs unsupervised pre-training to construct robust representations of time series data, demonstrating superior performance in various subsequent tasks. The model has been adapted to process multivariate time series, treating different variables such as temperature, humidity, and power as distinct channels. It utilizes the transformer’s self-attention mechanism to adeptly capture the intricate spatiotemporal interdependencies that influence fluctuations in wind speed.

In our approach, we segment multivariate time series data into fixed-length time windows and normalize the segments within each window. Preliminary temporal features for each variable are extracted using convolutional layers, and these features are then embedded into a high-dimensional space. The transformer encoder subsequently processes these embedded features to elucidate the spatiotemporal relationships among them. The prediction head of the model then generates forecasts based on these encoded features.

Experimental results confirm that the methodology underpinning WindFormer significantly outperforms both traditional statistical methods and contemporary deep learning models in short-term wind speed forecasting across various public datasets [15]. This study not only validates the effectiveness of WindFormer in handling complex multivariate time series data but also paves the way for innovative approaches and methodologies in future research within the domain of wind speed forecasting. The key contributions of our work are outlined below:

1.

Innovative model architecture: WindFormer utilizes a transformer-based architecture adapted for multivariate time series prediction. This model intricately processes and integrates multiple meteorological data streams like temperature, humidity, and power to capture their complex spatiotemporal dynamics with wind speed.

2.

Robust training strategy: Our approach combines unsupervised pre-training with multitask fine-tuning. Initially, WindFormer learns general feature representations from extensive unlabeled time series data, significantly enhancing the fine-tuning using labeled wind speed data.

3.

Exceptional predictive performance: Comparative assessments using multiple public datasets show that WindFormer significantly outperforms existing statistical and deep learning models in short-term wind speed prediction, confirming its superiority.

These contributions not only prove WindFormer’s effectiveness in handling complex multivariate time series data but also lay the groundwork for innovative approaches in the future wind speed prediction field.

2. Related Work

The capability to predict wind speeds accurately over short periods is crucial for various applications such as meteorology [16], wind energy production [17], and aviation safety [18]. Traditional methods for predicting wind speeds include statistical models and increasingly complex machine learning techniques. Traditional statistical approaches like the autoregressive integrated moving average (ARIMA) model [19] and the Kalman filter [20] utilize historical wind data to forecast future conditions. Although these models perform adequately for straightforward time series, they often falter when confronted with complex, multivariate datasets due to their limited ability to capture dynamic interactions between multiple variables [21].

With the evolution of machine learning, neural network-based methods such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have become prominent due to their proficiency in handling time series data. CNNs excel in extracting localized features, while RNNs are adept at discerning long-term dependencies [22]. Despite their strengths, these models require intricate feature extraction and fusion techniques to manage multivariate series, which often complicates the models’ architecture and hampers generalization and training efficiency [23].

The transformative impact of the transformer architecture, [24] initially popularized within the field of natural language processing, has extended to time series forecasting. Its self-attention mechanism facilitates the capture of global dependencies within sequences, providing a robust alternative to traditional sequence modeling techniques such as RNNs and CNNs. Transformer-based models, including Informer and Reformer, have showcased exceptional capabilities in handling long-sequence predictions [25,26]. Despite their advancements, these models have predominantly been tailored to single-feature time series forecasting, and their potential for handling complex, multivariate datasets has not been fully realized.

This research introduces the WindFormer model, a novel approach that integrates convolutional layers with a transformer encoder to masterfully manage multivariate time series data. By harnessing both convolutional and transformer architectures, WindFormer captures intricate spatiotemporal dependencies among various environmental features and wind speeds with unprecedented accuracy [27]. This model not only bridges the gap in existing transformer applications but also innovates model architecture by incorporating unsupervised pre-training, followed by multi-task fine-tuning. These techniques collectively enhance the model’s generalization capabilities and elevate its predictive performance, surpassing existing transformer-based models in the realm of complex, multivariate time series forecasting.

3. Methods

3.1. Problem Definition

This manuscript explores the short-term prediction of wind speeds [28,29] employing multivariate time series data. We specifically utilize a dataset comprising variables like temperature, humidity, and power [30,31] to forecast upcoming wind speeds. The mathematical formulation of this problem is as follows:

Consider the input multivariate time series data defined as follows:

$$\mathbf{X}=\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_T\}$$

(1)

where each ${\mathbf{x}}_t\in {\mathbb{R}}^d$ denotes the recorded values across d features at time step t, including temperature, humidity, and power.

Our objective is to predict future values of wind speed over the next h time steps:

$$\widehat{\mathbf{y}}=\{{\widehat{y}}_{T+1},{\widehat{y}}_{T+2},\dots ,{\widehat{y}}_{T+h}\}$$

(2)

Based on historical data up to T time steps, the model seeks a function f that maps the following:

$$\widehat{\mathbf{y}}=f\left(\mathbf{X}\right)$$

(3)

A model is trained to optimize the likelihood function across a training dataset, defined as $\mathcal{D}={\left\{({\mathbf{X}}^{\left(i\right)},{\mathbf{y}}^{\left(i\right)})\right\}}_{i=1}^N$, where ${\mathbf{y}}^{\left(i\right)}$ represents the actual wind speeds for the i-th instance. The goal is to maximize the following log-likelihood function:

$$\mathcal{L}\left(\theta \right)=\sum _{i=1}^{N}logp\left({\mathbf{y}}^{\left(i\right)}\right|{\mathbf{X}}^{\left(i\right)};\theta )$$

(4)

where $\theta$ symbolizes the model parameters, and $p\left(\mathbf{y}\right|\mathbf{X};\theta )$ is the probability of observing the sequence $\mathbf{y}$, given the sequence $\mathbf{X}$, under these parameters. Assuming a Gaussian distribution for prediction errors,

$${\mathbf{y}}^{\left(i\right)}=f({\mathbf{X}}^{\left(i\right)};\theta )+ϵ$$

(5)

with $ϵ\sim \mathcal{N}(0,{\sigma }^2)$, the log-likelihood can be reformulated as follows:

$$\mathcal{L}\left(\theta \right)=-{\displaystyle \frac{N}{2}}log\left(2\pi {\sigma }^2\right)-{\displaystyle \frac{1}{2{\sigma }^2}}\sum _{i=1}^N{\parallel {\mathbf{y}}^{\left(i\right)}-f({\mathbf{X}}^{\left(i\right)};\theta )\parallel }^2$$

(6)

Thus, maximizing $\mathcal{L}\left(\theta \right)$ corresponds to minimizing the mean squared error (MSE) loss function, defined as follows:

$${\mathcal{L}}_{MSE}\left(\theta \right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\parallel {\mathbf{y}}^{\left(i\right)}-f({\mathbf{X}}^{\left(i\right)};\theta )\parallel }^2$$

(7)

Minimizing ${\mathcal{L}}_{MSE}\left(\theta \right)$ enables us to fine-tune the parameters $\theta$ for accurate predictions of future wind speeds.

3.2. Model Architecture

We introduce WindFormer, a neural transformer tailored to decoding multichannel wind speed data from various sensors. Its design can be adapted to any number of timestamps and channels, a feature crucial for accommodating different measurement setups and conditions [32]. A distinctive aspect of WindFormer is its use of signal chunking—a technique inspired by block-embedding methods prevalent in image processing—which optimizes the handling of input data. This approach not only improves processing efficiency but also enhances the model’s ability to capture essential spatiotemporal dynamics from complex datasets. The details of WindFormer are illustrated in Figure 1.

Each wind speed data point with a timestamp, t, and a stride, s, forms a matrix, X, that is segmented into $⌊{\displaystyle \frac{T-t}{s}}⌋+1$ parts, each $x\in {\mathbb{R}}^{C\times t}$. We segment each channel using a non-overlapping window of length w, resulting in the following:

$$x=\{x_{cij,k}\in {\mathbb{R}}^w|j=1,2,\cdots ,C,k=1,2,\cdots ,⌊{\displaystyle \frac{t}{w}}⌋\}.$$

(8)

3.2.1. Temporal Encoder

Given the high temporal variability of wind speeds [35], it is essential to encode temporal features effectively before integrating block-level interactions. We employ a temporal encoder, comprising multiple convolutional blocks, to convert each chunk into dense feature embeddings. Each block [36] consists of a 1D convolution, group normalization [11], and a GELU activation [12]. The encoder outputs a series of embeddings, $e=\{e_{cij,k}\in {\mathbb{R}}^d|j=1,2,\cdots ,C,k=1,2,\cdots ,⌊{\displaystyle \frac{t}{w}}⌋\}$.

3.2.2. Temporal and Spatial Embeddings

To enhance the model’s ability to interpret both temporal and spatial aspects of wind data, we initialize lists of temporal embeddings, $TE=\{t{e}_1,t{e}_2,\cdots ,t{e}_{t_{max}}\}$, and spatial embeddings, $SE=\{s{e}_1,s{e}_2,\cdots ,s{e}_C\}$, both with dimension d. These embeddings are added to the output of the temporal encoder to enrich each chunk’s representation:

$$\{e_{cij,k}+t{e}_k+s{e}_{ij}|j=1,2,\cdots ,C,k=1,2,\cdots ,⌊{\displaystyle \frac{t}{w}}⌋\}$$

(9)

3.2.3. Transformer Encoder

The transformer encoder processes the enhanced sequences directly. To optimize training stability and efficiency, we apply layer normalization to queries and keys before attention computation, mitigating large attention score values and enhancing generalization across different wind datasets.

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{\mathrm{LN}\left(Q\right)\mathrm{LN}{\left(K\right)}^T}{\sqrt{d_{head}}}}\right)V$$

(10)

For downstream applications, average pooling is applied to the outputs, and the model is adapted with task-specific heads for various time-domain wind speed prediction tasks.

3.3. Neural Tokenizer Training

Further, the schematic diagram of the Neural Tokenizer training and WindFormer pre-training process are demonstrated in Figure 2. To prepare WindFormer for wind speed forecasting, we tokenize wind speed series into discrete symbols efficiently. We use a neural spectral tokenizer originally developed for signal analysis and adapt it for wind speed time series.

• Neural Tokenizer. This tokenizer transforms wind speed data into discrete symbols. It uses a neural codebook with K discrete symbols of dimension D, represented as $\mathbf{V}={\left\{{\mathbf{v}}_i\right\}}_{i=1}^K$. For each data segment $\mathbf{x}$, the tokenizer encodes it into chunk representations $\mathbf{p}={\left\{{\mathbf{p}}_i\right\}}_{i=1}^N$, where N is the number of chunks. Each ${\mathbf{p}}_i$ is then mapped to the nearest vector in the codebook via the following:

  $${\mathbf{z}}_i=\underset{{\mathbf{v}}_j\in \mathbf{V}}{arg\; min}{\parallel {\mathbf{p}}_i-{\mathbf{v}}_j\parallel }^2.$$

  (11)
• Wind speed prediction. Rather than employing traditional Fourier transforms, this advanced method leverages high-resolution temporal data chunks ${\mathbf{x}}_{c,k}=[x\left[1\right],x\left[2\right],\dots ,x\left[w\right]]$ to predict future wind speeds:

  $${\tilde{x}}_{c,k}=\sum _{n=1}^{N}x\left[n\right]·exp\left(-2\pi i{\displaystyle \frac{mn}{N}}\right),$$

  (12)

  where m represents the number of future time steps intended for prediction. This formulation allows us to capture both the linear and non-linear dependencies that characterize wind speed variations over time.

To further refine our prediction capabilities, we compute the magnitude and phase of each ${\tilde{x}}_{c,k}$. These complex features are crucial, as they encapsulate the fundamental frequency components of wind speeds, providing a robust basis for training our neural decoder:

• Advanced neural decoder training. The decoder, implemented within a transformer architecture specifically optimized for temporal sequences, is meticulously trained to predict the quantified features from the processed data chunks. This approach enhances the model’s ability to generalize across different temporal dynamics, thereby improving prediction accuracy.
• Training objective with enhanced metrics. The comprehensive training of the neural tokenizer and prediction model focuses on minimizing a mean squared error (MSE) loss, meticulously designed to enhance model performance:

  $$L_T=\sum _{\mathbf{x}\in \mathcal{D}}\sum _{i=1}^N\left(\parallel o{A}_i-A_i{\parallel }^2+{\parallel o{\Phi }_i-{\Phi }_i\parallel }^2\right),$$

  (13)

  where $o{A}_i$ and $o{\Phi }_i$ represent the magnitudes and phases predicted via the neural decoder, respectively, and $A_i$ and ${\Phi }_i$ are the actual values derived from the discrete Fourier transformation of the wind speed data. This loss function not only drives the fidelity of the magnitude and phase predictions but also ensures that the model captures the essential dynamics critical for accurate short-term forecasting.

This sophisticated method integrates the neural tokenizer training with explicit objectives for short-term wind speed prediction. By transforming historical data into discrete, interpretable symbols, the model learns to effectively forecast future wind conditions, marking a significant advancement in predictive meteorology and computational modeling.

3.4. Pre-Training Module

In the context of time series prediction, we adopt a masked modeling approach suitable for continuous data streams for pre-training [26]. Similar to neural tokenizer training, we mask portions of the time series data and train the model to predict the masked segments [37], leveraging the temporal dynamics encoded via the model. The architecture is similar to that used in Section 3.2, but here, the focus is on capturing and predicting future values in the following sequence:

$$p\left(v^{\prime }\right|e_M)=\mathrm{softmax}\left(\mathrm{Linear}\left(h\right)\right),$$

(14)

where $e_M$ represents the embeddings of the masked input, and h is the output of the transformer encoder. The training objective remains to predict the original values of the masked segments, thus fostering a model capable of effective generalization across various prediction tasks:

$$L_M=-\sum _{x\in D}\sum _{m_i=1}logp\left(v_i\right|e_M),$$

(15)

where $m_i$ designates a segment as masked. This method not only enables the model to process incomplete data effectively but also enhances its ability to forecast future outcomes from available data segments, improving its utility in real-world forecasting scenarios.

4. Experiments

4.1. Baseline Models for Time Series Forecasting

4.1.1. Fourier-Enhanced Dformer

The FEDformer [34] is an innovative deep learning model that integrates the concept of Fourier transformation with the transformer architecture to process time series data. This model was specifically designed to concentrate on the periodic characteristics of time series data. It boosts the capability of detecting long-term dependencies and cyclical patterns using the Fourier transformation technique. The FEDformer is particularly effective for time series forecasting tasks characterized by notable periodicity and extensive long-term dependencies.

4.1.2. Autoformer

Autoformer [32] is an autoregressive transformer model that utilizes an autoregressive mechanism to capture the long-term dependencies in time series. Compared to traditional transformer models, the Autoformer is more efficient in design because it reduces computational complexity through its autoregressive attention mechanism. This efficiency makes Autoformer particularly effective in handling long time series, especially demonstrating strong performance in areas such as energy load forecasting.

4.1.3. Informer

Informer [23] is a model specifically designed for long-sequence time series forecasting. Its core innovation is the introduction of a new attention mechanism called ”ProbSparse Self-attention”, which effectively reduces computational complexity while retaining the capacity to capture long-term dependencies. The Informer has shown remarkable results in tasks that require precise long-term forecasting, such as meteorological predictions and stock market analysis.

4.1.4. Pyraformer

Pyraformer [22] is a time series forecasting model that employs a pyramidal structure. It captures dependencies across different time granularities through a multi-scale attention mechanism and extracts multi-level features of time series via a layer-by-layer aggregation approach. This structural design allows Pyraformer to process long-term time series data effectively while maintaining low computational costs.

Hyperparameter settings are shown in

Table 1. Hyperparameter settings.

Hyperparameter	Description	Value
Learning rate	Step size for weight updates	0.001
Batch size	Number of samples per training batch	64
Epochs	Number of complete passes through data	50
Dropout rate	Probability of dropping neurons	0.1
Transformer layers	Number of transformer layers	4
Heads	Number of attention heads	8
Embedding dimension	Dimension of feature embeddings	256
Weight initialization	Method for initializing weights	Xavier
Optimizer	Algorithm for adjusting model weights	Adam

.

4.2. Performance Metrics’ Introduction

In this research, we adopted two key metrics to assess the accuracy of the electric power prediction model: the mean squared error (MSE) [24] and the mean absolute error (MAE) [37]. These indicators are commonly used in the field of power system forecasting, and they offer a robust evaluation of the discrepancies between predicted outputs and actual observations.

The mean squared error (MSE), a standard metric in forecasting, calculates the average squared deviations between forecasted and true values. This error metric is formulated as follows:

$$MSE={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(Y_i-{\widehat{Y}}_i)}^2,$$

(16)

where $Y_i$ is the true value for the $i\mathrm{th}$ observation, ${\widehat{Y}}_i$ is the predicted value, and n represents the total number of observations. A lower MSE value indicates higher predictive accuracy of the model.

The mean absolute error (MAE) calculates the mean of the absolute differences between the forecasts and the actual figures. Unlike MSE, the MAE is less affected by outliers. The calculation formula is as follows:

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n|Y_i-{\widehat{Y}}_i|.$$

(17)

In this context, $Y_i$ and ${\widehat{Y}}_i$ are the true and predicted values, respectively, and n denotes the number of observations. A smaller MAE indicates superior model performance. Evaluating both MSE and MAE allows for a thorough assessment of the model’s capabilities in electric power forecasting, with MSE focusing on squared error magnitudes and MAE reflecting the magnitude of errors directly, together providing a robust basis for model comparison and assessment.

4.3. Datasets

4.3.1. ERA5 Reanalysis Data

ERA5, provided by the European Centre for Medium-Range Weather Forecasts (ECMWF) [38], is a high-resolution reanalysis dataset and the successor to ERA-Interim [39]. ERA5 [40] offers global atmospheric, land, and ocean weather data from 1979 to the present, meeting various meteorological and climate research needs. The data include variables such as surface temperature, precipitation, surface pressure, wind speed, direction, and humidity, with a spatial resolution of 0.25 degrees by 0.25 degrees and an hourly temporal resolution. It features long-term records from 1979 to the present, covering diverse meteorological variables for the atmosphere, land, and oceans. Data can be downloaded from ECMWF’s Climate Data Store (CDS) [41] and customized as needed: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview (accessed on 12 June 2023).

4.3.2. NOAA’s Integrated Surface Data (ISD)

ISD, provided by the National Oceanic and Atmospheric Administration (NOAA) [42], is a global surface meteorological observation dataset. It aggregates detailed historical meteorological data from thousands of weather stations worldwide. The data include variables such as temperature, pressure, wind speed, wind direction, precipitation, and relative humidity, covering 1901 to the present, with global spatial coverage and hourly and daily temporal resolutions. Its features include extensive global station coverage, long-term records suitable for climate change and trend analysis, and detailed hourly observations for short-term weather forecasting and research. Data can be accessed and downloaded through NOAA’s National Centers for Environmental Information (NCEI): https://www.ncdc.noaa.gov/isd (accessed on 11 October 2023).

4.3.3. Wind Integration National Dataset Toolkit (WIND Toolkit)

The WIND Toolkit, developed by the National Renewable Common Energy Laboratory (NREL) [43], offers a comprehensive wind speed dataset with a high resolution, specifically tailored to supporting renewable energy studies and the development of wind power projects across North America. This dataset encompasses a variety of measurements, including wind speed, wind direction, temperature, and atmospheric pressure. It spans the years 2007 to 2013, provides data with a spatial resolution of 2 km × 2 km, and records every 5 min for temporal resolution. It is ideal for wind energy resource assessment and short-term wind speed prediction, offering high-quality, reliable data covering various meteorological variables needed for wind energy research. Data can be accessed and downloaded through NREL’s WIND Toolkit website, with options for customized datasets: https://www.nrel.gov/grid/wind-toolkit.html (accessed on 15 October 2023).

4.4. Main Results

In this study, as shown in

Table 2. Performance comparison results.

Methods		WindFormer		FEDformer		Autoformer		Informer		Pyraformer
Metric		MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MAE	MAE
ERA5	96	0.126	0.186	0.157	0.209	0.254	0.306	0.366	0.431	0.442	0.502
	192	0.144	0.247	0.181	0.231	0.278	0.335	0.387	0.438	0.488	0.532
	336	0.167	0.285	0.218	0.298	0.317	0.364	0.413	0.458	0.520	0.561
	720	0.183	0.337	0.244	0.381	0.351	0.401	0.444	0.495	0.548	0.600
ISD	96	0.202	0.318	0.267	0.333	0.386	0.459	0.524	0.592	0.671	0.735
	192	0.239	0.350	0.296	0.367	0.412	0.484	0.557	0.624	0.702	0.769
	336	0.261	0.378	0.324	0.391	0.445	0.514	0.588	0.652	0.725	0.793
	720	0.278	0.413	0.361	0.409	0.475	0.543	0.620	0.681	0.749	0.821
WIND	96	0.278	0.324	0.318	0.389	0.457	0.522	0.591	0.665	0.734	0.802
	192	0.345	0.363	0.326	0.419	0.489	0.555	0.625	0.696	0.759	0.836
	336	0.330	0.409	0.375	0.450	0.510	0.583	0.655	0.728	0.795	0.868
	720	0.357	0.434	0.408	0.475	0.547	0.616	0.685	0.752	0.821	0.896

, we compared the performance of five different prediction methods (WindFormer, FEDformer, Autoformer, Informer, and Pyraformer) using the ERA5, ISD, and WIND datasets. The table presents the MSE and MAE of these methods for four different prediction time steps (96, 192, 336, and 720).

The data show that WindFormer consistently achieved the best performance across all datasets and time steps, with the lowest MSE and MAE values. For example, with the ERA5 dataset and a 96-step prediction, WindFormer had an MSE of 0.126, significantly lower than FEDformer’s 0.157 and Autoformer’s 0.254. Additionally, WindFormer demonstrated a clear advantage with the ISD and WIND datasets, especially for longer prediction steps (720 steps), with MSE values of 0.278 and 0.357, respectively, outperforming other methods.

The findings underscore WindFormer’s enhanced predictive accuracy and robust generalization capabilities across complex time series prediction tasks. This superior performance is largely attributable to its innovative architecture and adept handling of spatiotemporal dynamics, which distinguish it from conventional models. Although alternative methods exhibit certain strengths under specific scenarios, they consistently fall short of WindFormer’s overall effectiveness.

In summary, WindFormer emerges as the superior choice in our comparative analysis, setting a new standard for time series prediction tasks. The demonstrated efficacy of WindFormer provides critical insights, highlighting its potential as a foundational model for advancing the field of time series analysis. These results not only underscore the model’s immediate practical benefits but also pave the way for ongoing enhancements and broader applications in time series forecasting.

In this paper, we compare the performance of five methods (WindFormer, FEDformer, Autoformer, Informer, and Pyraformer) using the ERA5, ISD, and WIND datasets. The table shows their mean squared error (MSE) values for different prediction time steps (96, 192, 336, and 720).

Figure 3 demonstrates the models’ performance across different wind farms via simulation, which shows the suitability and robustness of WindFormer. The results shown in Figure 4 shows that WindFormer achieves the best performance across all datasets and time steps, with the lowest MSE values, indicating its superior prediction accuracy. For example, with the ERA5 dataset, WindFormer had an MSE of 0.126 for the 96-step prediction, significantly lower than FEDformer’s 0.157 and Autoformer’s 0.254. Additionally, WindFormer maintained the lowest MSE values with the ISD and WIND datasets, demonstrating its stability and superiority under various conditions.

These findings indicate that WindFormer has a strong generalization ability and robustness in handling complex time series prediction tasks. Its superior performance may be attributed to its unique architecture and effective capture of spatiotemporal features. In contrast, while the other methods show certain advantages, they do not outperform WindFormer in most cases.

In summary, WindFormer demonstrated the best predictive performance in this comparison, making it a recommended method for similar time series prediction tasks. This result also provides valuable insights for the further improvement and optimization of time series prediction models.

Figure 5 shows the short-term wind speed prediction results for the first week of January 2023 using the WindFormer model with three datasets (ERA5, ISD, and WIND Toolkit). ERA 5 Dataset:The actual wind speed (blue line) and predicted wind speed (red dashed line) closely match. The model captured the fluctuations and trends of wind speed well, indicating good predictive performance with the ERA5 dataset. The actual wind speed (green line) and predicted wind speed (orange dashed line) generally followed the same trend, though there were some deviations during periods of rapid change. The model slightly underperforms in capturing sharp wind speed changes but still maintains overall accuracy. WIND Toolkit Dataset: The actual wind speed (purple line) and predicted wind speed (brown dashed line) show high variability. While the model tracked the general trend of actual wind speed, there were some errors in predicting the peaks and troughs.

The analysis of short-term wind speed predictions across these three datasets demonstrates that the WindFormer model consistently reflects the actual wind speed trends. However, the characteristics of different datasets led to certain prediction biases, especially during periods of rapid wind speed changes. This suggests that, in wind speed prediction, both dataset selection and model adaptability require further optimization and research.

4.5. Efficiency Analysis

Table 3. Training time and computational complexity comparison.

Method	Training Time (h)	Parameters (Millions)	FLOPs (Billions)
WindFormer	4.5	12.3	34.2
NHits	5.2	13.1	36.5
GNN	6.0	14.7	40.8

presents a comparative analysis of three methods in terms of training time, parameter count, and computational complexity (FLOPs). The WindFormer method showed a significant advantage in training time, requiring only 4.5 h, while the NHits and GNN methods required 5.2 h and 6.0 h, respectively. This indicates that WindFormer is more time-efficient than the other two methods. In terms of parameter count, WindFormer has 12.3 million parameters, which is fewer than NHits’ 13.1 million and GNN’s 14.7 million. This suggests that WindFormer not only excels in training time but also has a relatively smaller model size, potentially reducing memory usage and enhancing computational efficiency. Regarding computational complexity (FLOPs), WindFormer has 3.42 billion floating-point operations, compared to NHits’ 3.65 billion and GNN’s 4.08 billion. WindFormer again demonstrates a lower computational cost. This result further supports WindFormer’s advantage in computational efficiency. In conclusion, WindFormer exhibits superior efficiency in its training time, parameter count, and computational complexity, suggesting its potential advantages in practical applications. Future research can further explore WindFormer’ s performance in different tasks and with different datasets to comprehensively evaluate its potential and application value.

5. Conclusions

In this study, we introduced WindFormer, an innovative transformer-based model for predicting multivariate time series, tailored for short-term wind speed forecasting. WindFormer not only fills critical technological voids by providing robust handling of multivariate, dynamic time series data but also sets a new benchmark in the predictive accuracy of short-term wind forecasts, thus significantly contributing to the fields of meteorology and renewable energy management. Our contributions are threefold. Firstly, we designed the WindFormer model to adeptly handle complex spatiotemporal relationships among wind speed and ancillary data like temperature, humidity, and power, thus enhancing the accuracy of wind speed forecasts. Secondly, we adopted an unsupervised pre-training strategy to extract general feature representations from extensive unlabeled time series data, subsequently refining the model with labeled data specific to wind speed forecasting. This method exploits the vast quantities of available time series data to boost the model’s performance in targeted tasks. Lastly, our thorough validation across various public datasets has shown that WindFormer surpasses both traditional statistical techniques and current deep-learning approaches in forecasting wind speed over short intervals, confirming the efficacy and superior capability of our proposed model.