MSAFormer: A Transformer-Based Model for PM_2.5 Prediction Leveraging Sparse Autoencoding of Multi-Site Meteorological Features in Urban Areas

Abstract

The accurate prediction of PM_2.5 concentration, a matter of paramount importance in environmental science and public health, has remained a substantial challenge. Conventional methodologies for predicting PM_2.5 concentration often grapple with capturing complex dynamics and nonlinear relationships inherent in multi-station meteorological data. To address this issue, we have devised a novel deep learning model, named the Meteorological Sparse Autoencoding Transformer (MSAFormer). The MSAFormer leverages the strengths of the Transformer architecture, effectively incorporating a Meteorological Sparse Autoencoding module, a Meteorological Positional Embedding Module, and a PM_2.5 Prediction Transformer Module. The Sparse Autoencoding Module serves to extract salient features from high-dimensional, multi-station meteorological data. Subsequently, the Positional Embedding Module applies a one-dimensional Convolutional Neural Network to flatten the sparse-encoded features, facilitating data processing in the subsequent Transformer module. Finally, the PM_2.5 Prediction Transformer Module utilizes a self-attention mechanism to handle temporal dependencies in the input data, predicting future PM_2.5 concentrations. Experimental results underscore that the MSAFormer model achieves a significant improvement in predicting PM_2.5 concentrations in the Haidian district compared to traditional methods. This research offers a novel predictive tool for the field of environmental science and illustrates the potential of deep learning in the analysis of environmental meteorological data.

1. Introduction

The pervasiveness of air pollution in numerous global cities poses significant threats to public health and induces sustained detrimental effects on broader ecological systems [1,2,3]. Among the various pollutants, fine particulate matter (PM_2.5), with a diameter less than 2.5 μm, stands out [4,5]. Its adverse effects range from respiratory to cardiovascular diseases, underscoring the importance of developing a model that can accurately predict PM_2.5 concentrations, allowing for timely prevention measures and strategic responses [6,7,8,9].

Generally, PM_2.5 prediction methods are categorized into two classes: those based on physical models and those driven by data [10,11,12,13].

Physical model-based prediction methods, such as the Community Multi-scale Air Quality (CMAQ) [14,15], WRF/Chem [16,17], and Nested Air Quality Prediction Modeling System (NAQPMS) models [18,19], rely on scientific theories and equations to elucidate patterns of air pollution diffusion and transformation [20]. The strength of these models lies in their accuracy, which depends on how closely they approximate actual atmospheric conditions, and their explainability, as the predictions are based on scientific principles [21,22,23]. However, these models require a vast number of input parameters, including pollutant emission data, meteorological data, and terrain data [24,25]. The process of obtaining and processing these parameters is complex, and the models demand substantial computational resources [26].

On the other hand, data-driven prediction methods have yielded significant results with advancements in big data and computational capabilities [27,28,29]. Machine learning algorithms like Support Vector Machines (SVM) [30,31], Random Forest (RF) [32,33], and AdaBoost [34] are extensively employed in predicting PM_2.5 concentrations. However, these models primarily focus on utilizing historical data from air quality monitoring stations for predictions, largely ignoring the consideration of other influencing factors, particularly meteorological variables [35]. Meteorological conditions like temperature, humidity, wind speed, and direction significantly impact PM_2.5 concentrations by influencing and shaping the diffusion, mixing, and deposition processes of particulates in the air [36,37,38,39].

Recent widespread application of deep learning technologies is beginning to alter this situation [40,41,42]. Researchers have started integrating meteorological variables into the deep learning models for PM_2.5 prediction, but the prevalent approach still relies on manual feature design and employs relatively traditional models such as Long Short-Term Memory (LSTM) [43,44,45] and Gated Recurrent Units (GRU) [46,47]. While these models exhibit strengths in handling the time-series characteristics of meteorological data, their capacity to excavate deep and complex features in multi-source meteorological data needs further enhancement [48,49].

The current challenge lies in the requirement of extensive domain knowledge and presupposed data structures or relationships to handle high-dimensional meteorological data, posing serious challenges to the generality and adaptability of the models [50,51,52]. Indeed, the relationship between meteorological conditions and PM_2.5 concentrations is highly nonlinear and subject to complex interactions among various environmental factors [53]. This relationship could change with alterations in time and location [54]. Therefore, there is an urgent need to develop a novel prediction model that is capable of automatically extracting salient features from high-dimensional meteorological data and can adapt to changing environmental conditions.

To address these challenges, we have designed a novel deep learning model named the Meteorological Sparse Autoencoding Transformer (MSAFormer), based on the Transformer architecture and sparse autoencoding technology. The Transformer architecture, initially designed for natural language processing tasks [55], is adept at handling long-term dependencies in the input data—thanks to its powerful self-attention mechanism—and has exhibited excellent performance in many other fields [56], including environmental science [57]. On the other hand, sparse autoencoding is an effective feature learning technique that can automatically extract and learn significant features from high-dimensional data [58].

Our model comprises three core modules. Firstly, the Meteorological Sparse Autoencoding module extracts critical features from high-dimensional, multi-site meteorological data, providing key information for understanding and predicting PM_2.5 concentrations. Secondly, the Meteorological Position Embedding module utilizes a one-dimensional Convolutional Neural Network (CNN) to flatten these sparse encoded features for processing in the subsequent Transformer module. Lastly, the PM_2.5 Prediction Transformer module leverages the self-attention mechanism to handle time dependencies in the input data for an accurate prediction of future PM_2.5 concentrations. These designs confer superior performance on our model when handling multi-source, high-dimensional, and highly temporal meteorological data, offering a new and effective tool for precise PM_2.5 prediction.

2. Materials

This study relies on data from two primary sources: air pollutant concentrations, with an emphasis on PM_2.5, and meteorological variables. The PM_2.5 concentration data, expressed in micrograms per cubic meter (μg/m³), were sourced from the Beijing Municipal Ecological Environmental Monitoring Center (BJMEMC). This data can be accessed via their official website, http://www.bjmemc.com.cn/ (accessed on 21 March 2023). Concurrently, a set of meteorological factors were gathered from nine meteorological monitoring stations, facilitated through the National Climatic Data Center (NCDC) of the United States, an institution within the National Oceanic and Atmospheric Administration (NOAA). Global meteorological data, encompassing factors such as temperature, pressure, dew point, wind direction and speed, cloud cover, and precipitation, can be retrieved from their official website, https://www.ncei.noaa.gov/ (accessed on 17 April 2023). The meteorological factors included in the study and their corresponding units are detailed in

Table 1. Description and units of meteorological factors.

Meteorological Factor	Description	Unit
Temperature	Average temperature at the site	Degrees Celsius
Pressure	Atmospheric pressure at the site	Millibars
Dew Point	Temperature at which air becomes saturated	Degrees Celsius
Wind Direction	Direction from which the wind is blowing	Degrees
Wind Speed	Speed of wind at the site	Meters per second
Cloud Cover	Percentage of the sky covered by clouds	Percent (%)
Precipitation	Amount of rainfall or snowfall at the site	Millimeters

.

Data for the study were gathered at an hourly resolution from 1 January 2021 to 31 December 2022. The data collected from 1 January 2021 to 31 December 2021 were utilized as training data, while the data from 1 January 2022 to 31 December 2022 served as testing data. Figure 1 illustrates the geographical distribution of the meteorological and air pollutant monitoring stations from which the data was sourced.

Corresponding information on these stations, such as longitude, latitude, station code, and name, is outlined in

Table 2. Detailed information on the meteorological and air pollutant monitoring stations used for data collection in this study.

Category	Station Code	Station Name	Longitude	Latitude
Air Pollution Monitoring Stations	1007A	Haidian Wanliu	116.29	39.96
Meteorological Stations	54398	Shunyi	116.37	40.08
	54399	Haidian	116.17	39.59
	54424	Pinggu	117.07	40.10
	54431	Tongzhou	116.38	39.55
	54433	Chaoyang	116.30	39.57
	54499	Changping	116.13	40.13
	54514	Fengtai	116.15	39.52
	54594	Daxing	116.21	39.43
	54596	Fangshan	116.12	39.46

.

The integration of PM_2.5 concentration data from BJMEMC and meteorological data from NCDC, bolstered by comprehensive station information, provides a robust foundation for the development and validation of the MSAFormer model.

3. Methodology

3.1. Overview of the MSAFormer Model

Predicting PM_2.5 concentration accurately is crucial for understanding and managing air quality in urban areas. Traditional methods, primarily based on statistical regression models, often struggle to capture complex non-linear relationships and dynamics in meteorological and pollution data. To address these challenges, we propose a novel model, MSAFormer, which combines the power of the Transformer architecture and meteorological data analysis.

MSAFormer aims to integrate meteorological and temporal information effectively to achieve an accurate prediction of PM_2.5 concentration. Meteorological data, captured from various monitoring stations, provide essential contextual information related to air quality. Temporal data, reflected in the time-series data of PM_2.5 concentration, reveal the dynamics of air quality over time.

To harness the rich information embedded in both meteorological and temporal data, the MSAFormer model is designed with three primary modules: the Meteorological Sparse Autoencoding module, the Meteorological Embedding module, and the Transformer Prediction module.

The Meteorological Sparse Autoencoding module is designed to encode high-dimensional meteorological data into a sparse representation. The sparsity constraint facilitates the extraction of meaningful and critical features, improving the interpretability of the model.

Next, the Meteorological Embedding module receives the sparse-encoded meteorological data, flattens them, and applies a 1D Convolutional Neural Network to encode positional information. This module transforms the encoded meteorological data into a format that is suitable for processing in the subsequent Transformer module.

Finally, the Transformer Prediction module integrates the meteorological data processed by the Meteorological Embedding module with the historical PM_2.5 concentration data. The module employs the self-attention mechanism, capturing the temporal dependencies in the input data and predicting future PM_2.5 concentrations.

An overview of the MSAFormer model structure is provided in Figure 2, highlighting the journey of the data from the input stage, through each module, to the final prediction of PM_2.5 concentration. This modular design allows for flexibility and adaptability, making the MSAFormer model a promising tool for air quality prediction tasks.

3.2. Meteorological Sparse Autoencoding Module

The first module of the MSAFormer, the Meteorological Sparse Autoencoding module, deals with the processing of multivariate meteorological data. This module employs the concept of sparse autoencoding to transform the original high-dimensional meteorological data into a reduced, more manageable feature space that emphasizes critical information and suppresses noise and redundant features.

Consider the raw meteorological data collected from nine meteorological stations across Beijing. For each station, we have a time series of several meteorological features, namely temperature, pressure, dew point, wind speed, wind direction, cloud cover, and precipitation. Hence, for each station, we formulate a matrix $X\in {\mathbb{R}}^{d\times T}$, where $d$ is the number of meteorological features, and $T$ is the number of time steps.

A typical sparse autoencoder comprises two components: an encoder $f_{\theta }(\cdot )$, parameterized by $\theta$, and a decoder $g_{\varphi }(\cdot )$, parameterized by $\varphi$. The encoder aims to transform the high-dimensional input data $X$ into a lower-dimensional representation $Z=f_{\theta }(X)\in {\mathbb{R}}^{d^{\prime }\times T}$, where $d^{\prime }$ is the number of latent features.

The decoder, on the other hand, tries to reconstruct the original input from the encoded representation, i.e., $\widehat{X}=g_{\varphi }(Z)$. The goal of the training process is to minimize the reconstruction error while maintaining the sparsity of the encoded representation $Z$.

To achieve this, we employ a loss function that combines the reconstruction error, given using the Frobenius norm of the difference between the original and reconstructed matrices, and a sparsity-inducing penalty term based on the Kullback–Leibler (KL) divergence. The loss function ${\mathcal{L}}_{AE}$ is thus given using:

$${\mathcal{L}}_{AE}=\Vert X-\widehat{X}{\Vert }_F^2+\lambda D_{KL}(Z||P_0)$$

(1)

where $\Vert \cdot {\Vert }_F$ denotes the Frobenius norm, $D_{KL}(Z||P_0)$ represents the KL divergence between the empirical distribution of $Z$ and the specified sparse prior distribution $P_0$, and $\lambda$ is a tunable parameter that controls the balance between the reconstruction error and the sparsity.

The Meteorological Sparse Autoencoding module plays a vital role in the MSAFormer model, as it effectively condenses high-dimensional meteorological data into a lower-dimensional, sparse, and informative representation. This representation serves as the foundation for the following modules to build upon and produce accurate PM_2.5 predictions.

3.3. Meteorological Positional Embedding Module

Once the sparse-encoded representations of the meteorological data are obtained through the Meteorological Sparse Autoencoding module, we next transform these features into a form suitable for the Transformer Prediction module. The Meteorological Embedding module accomplishes this by employing a 1D Convolutional Neural Network (CNN) to capture local dependencies among the meteorological features and convert the multi-dimensional input data into a flat representation. Additionally, a fully connected layer (FC) is utilized to encode positional information into the flattened data.

Let us denote the sparse-encoded representation of the meteorological data from the previous module as $Z\in {\mathbb{R}}^{d^{\prime }\times T}$, where $d^{\prime }$ is the dimension of the sparse-encoded representation, and $T$ is the number of time steps. The sparse-encoded data are organized into a matrix that needs to be reformatted for the time-series prediction task.

To leverage the inherent local dependencies in the multi-dimensional meteorological data, we apply a 1D convolution over the $d^{\prime }$ dimension of the sparse-encoded data. The 1D convolution operation uses a set of learnable filters to perform a sliding dot product over the input data. This mechanism allows the model to learn local patterns in the meteorological features.

Mathematically, the 1D convolution operation can be defined as follows. Given an input $Z\in {\mathbb{R}}^{d^{\prime }\times T}$ and a filter $h\in {\mathbb{R}}^m$, the convolution operation $Z\ast h$ outputs a new representation $Z^{\prime }\in {\mathbb{R}}^{(d^{\prime }-m+1)\times T}$, where each element $Z_{ij}^{\prime }$ is computed using

$$Z_{ij}^{\prime }=\sum _{k=0}^{m-1} h_k{Z}_{i+k,j}$$

(2)

After the 1D convolution operation, the flattened meteorological features are passed through a fully connected layer to encode positional information into the data. The FC layer essentially applies a linear transformation to the input data and can be represented as:

$$Z^{″}=W{Z}^{\prime }+b$$

(3)

where $W\in {\mathbb{R}}^{d^{″}\times (d^{\prime }-m+1)}$ and $b\in {\mathbb{R}}^{d^{″}}$ are the weight matrix and bias vector of the FC layer, respectively, and $d^{″}$ is the output dimension.

The output of this module, the embedded meteorological data, thus preserves critical information from the sparse-encoded representation but also incorporates local meteorological dependencies and positional information. This form of data proves to be more suitable for processing in the next Transformer Prediction module, ultimately contributing to more accurate PM_2.5 predictions.

3.4. PM_2.5 Prediction Transformer Module

Following the Meteorological Embedding module, the Transformer Prediction module is applied to perform PM_2.5 prediction. The embedded meteorological data from the previous module, along with the historical PM_2.5 data, are used as inputs for this module. The Transformer architecture is known for its ability to capture complex dependencies in sequential data and provide robust and interpretable predictions. Therefore, it is well-suited for handling our time-series prediction task.

The Transformer module contains an encoder and a decoder, both of which are composed of several identical layers. Each layer comprises two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. An essential feature of the Transformer is that it replaces the recurrence mechanism with the self-attention mechanism, which computes a weighted sum of all the input features rather than processing the input data step by step. This mechanism gives the model the capacity to focus more on important features and less on unimportant ones.

In the context of our model, let us denote the output from the Meteorological Embedding module as $E\in {\mathbb{R}}^{d^{″}\times T}$ and the historical PM_2.5 data as $Y\in {\mathbb{R}}^{d_y\times T}$, where $d^{″}$ is the output dimension of the Meteorological Embedding module, $d_y$ is the dimension of the historical PM_2.5 data, and $T$ is the number of time steps. The Transformer Prediction module processes these data in the following way:

• Encoder: The encoder takes the Meteorological Embedding module’s output $E$ as an input and passes it through the multi-head self-attention mechanism and the feed-forward network. The self-attention mechanism allows the model to focus on different parts of the input sequence and considers their importance for the current prediction. The feed-forward network further processes the attended features;
• Decoder: The decoder receives the encoded meteorological data and the historical PM_2.5 data $Y$. Similar to the encoder, it also contains a multi-head self-attention mechanism and a feed-forward network; however, it has an additional multi-head attention mechanism that attends to the encoder’s output.

The decoder ultimately generates a sequence of predicted PM_2.5 values. The Transformer Prediction module harnesses the complex temporal dependencies in both meteorological and historical PM_2.5 data, contributing to the generation of accurate and robust PM_2.5 predicts.

3.5. Training Strategy

The architecture and optimization parameters in the MSAFormer model play critical roles in achieving a high accuracy in PM_2.5 concentration predictions. The architecture parameters are specifically configured to optimally capture the spatial and temporal dynamics within multi-station meteorological data.

Table 3. Architecture parameters for MSAFormer model.

Items	Value	Description
Sequence Window Size	5	Length of memory units
Conv1d Kernel Size	7	Kernel size of the 1D convolution
Conv1d Embedding Size	128	Meteorological data embedding size
Position Embedding Size	128	Position embedding size
Attention Heads	8	Self-attention heads in Transformer
Transformer Layers	4	Layers in Transformer
Sparse Autoencoder Coefficient (λ)	0.4 (optimal)	Controls the sparsity of the Meteorological Sparse Autoencoding module

Models in this section were configured and trained with hyperparameters above.

presents a detailed configuration of the architecture parameters.

The Sequence Window Size of 5 was chosen to balance the trade-off between capturing enough temporal dependencies and computational efficiency. For the Conv1D Kernel Size, a setting of 7 was found to be effective for learning the inherent patterns in the meteorological data. Both Conv1D Embedding Size and Position Embedding Size are set to 128, which ensures an integrated and comprehensive representation of the meteorological and spatial information. The Transformer module, with eight attention heads and four layers, can capture the complex patterns and dependencies within the data at various abstraction levels. Model optimization is carried out using an adaptive learning strategy combined with a suitable loss function. The primary optimization parameters are presented in

Table 4. Optimization parameters for MSAFormer model.

Parameters	Value	Description
Optimizer	Adam	Adaptive learning rate optimizer
Loss Function	MSE Loss	Measures the difference between predicted and actual PM2.5 concentrations
Learning Rate	0.001	Initial learning rate for the optimizer
Batch Size	8	Number of training examples utilized in one iteration
Epochs	50	Number of complete passes through the training dataset

.

Adam optimizer is used for its adaptive learning rate adjustment capability, which aids efficient and robust optimization. We employ the Mean Squared Error (MSE) loss function, which is formulated as

$$L_{MSE}=\frac{1}{n}\sum _{i=1}^n (y_i-\widehat{y_i}{)}^2$$

(4)

where $y_i$ represents the observed PM_2.5 concentration, $\widehat{y_i}$ the predicted concentration, and n the total number of samples. This loss function aligns with our goal of minimizing prediction error in PM_2.5 concentration.

Furthermore, we incorporated an early stopping strategy to monitor the validation loss during training. When the validation loss stops improving after a certain number of epochs (patience), the training process is halted. This strategy helps in preventing overfitting by not allowing the model to learn the noise within the training data.

In summary, the training and optimization strategies employed in MSAFormer ensure its robustness and reliability in predicting PM_2.5 concentrations using multi-station meteorological data. These strategies contribute to the model’s ability to generalize well to unseen data, making it a practical tool for air quality prediction tasks.

4. Results and Discussion

4.1. Data Preparation and Evaluation Metrics

This study assembled an extensive dataset comprising hourly observations of PM_2.5 concentrations from 1 January 2021 to 31 December 2022. Initially, the PM_2.5 concentration data were transformed into serialized samples using the sliding window method, with a Sequence Window Size of 5 for single-step prediction. Any serialized samples with missing PM_2.5 concentrations were then carefully eliminated to ensure data integrity. Next, meticulous data cleaning procedures were implemented for the meteorological factors. The SimpleImputer class from the scikit-learn package was employed to address any missing values, replacing them with the mean, median, or most frequent value, as appropriate. After ensuring the continuity and coherence of the dataset through these cleaning processes, temporal alignment was performed to synchronize the serialized PM_2.5 concentration samples with corresponding meteorological features, resulting in a set of 16,000 valid samples. These samples were then temporally divided into a training dataset (spanning the period from 1 January 2021 to 31 December 2021) and a testing dataset (from 1 January 2022 to 31 December 2022).

In the context of our study, the prediction problem is formulated as follows: Given the past 5 h of PM_2.5 and meteorological factor data, the goal is to predict the PM_2.5 concentration for the next hour. Mathematically, this can be defined as

$${\widehat{Y}}_{t+1}=f(Y_{t-4:t},Z_{t-4:t})$$

(5)

where $Y_{t-4:t}$ and $Z_{t-4:t}$ represent the past 5 h of PM_2.5 and meteorological factor data, respectively, and $f$ denotes the prediction model.

The performance of the models is evaluated using Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and the Coefficient of Determination ($R^2$). These metrics are defined as

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}{\sum }_{i=1}^n (Y_i-{\widehat{Y}}_i{)}^2}$$

(6)

$$\mathrm{MAE}=\frac{1}{n}{\sum }_{i=1}^n |Y_i-{\hat{Y}}_i|$$

(7)

$$R^2=1-\frac{{\sum }_{i=1}^n (Y_i-{\widehat{Y}}_i{)}^2}{{\sum }_{i=1}^n (Y_i-\bar{Y}{)}^2}$$

(8)

where $n$ denotes the number of samples, $Y_i$ and ${\widehat{Y}}_i$ represent the actual and predicted PM_2.5 concentrations, respectively, and $\bar{Y}$ stands for the mean of the actual PM_2.5 concentrations. The objective of our study is to minimize RMSE and MAE while maximizing the $R^2$ score.

4.2. Models Comparation and Performance Analysis

In order to establish the efficacy of the proposed MSAFormer model, it was juxtaposed against five widely recognized models: Support Vector Machine (SVM) [30], Random Forest (RF) [32], Adaptive Boosting (AdaBoost) [34], Long Short-Term Memory (LSTM) [43], and Gated Recurrent Unit (GRU) [46]. These models are detailed below:

• Support Vector Machine (SVM): This was implemented employing a radial basis function (RBF) kernel. The optimal parameters, C and gamma, were ascertained via a grid search over the parameters of ‘C’: [0.1, 1, 10, 100, 1000] and ‘gamma’: [1, 0.1, 0.01, 0.001, 0.0001];
• Random Forest (RF): the RF model was constructed with a forest of 100 trees, with ‘max_features’ set to ‘sqrt’, a choice guided by the nature of regression tasks;
• Adaptive Boosting (AdaBoost): AdaBoost was set up with 50 weak learners, with a learning rate of 1, ensuring an efficient trade-off between bias and variance;
• Long Short-Term Memory (LSTM): the LSTM, a popular variant of recurrent neural networks, was structured with 50 units, and the activation function was set as ‘tanh’;
• Gated Recurrent Unit (GRU): GRU, a modern variant of recurrent neural networks, shared the same structure as LSTM, with 50 units and a ‘tanh’ activation function.

The temporal dynamics and predictive performance of the six models were compared using three evaluation methods: time-series visualization, accuracy metrics, and error histogram analysis.

Figure 3 presents the time-series plots of PM_2.5 predictions for each model throughout 2022 (Figure 3a–f). These graphs reveal that the MSAFormer model closely captures the true PM_2.5 concentration trends, thereby evidencing its superior temporal modeling capability.

Table 5. Comparative performance of SVM, RF, AdaBoost, LSTM, GRU, and MSAFormer models in terms of RMSE, MAE, and R² scores.

Model	RMSE	MAE	R2
MSAFormer	11.112	8.691	0.898
SVM	19.674	14.930	0.706
RF	23.000	17.452	0.632
AdaBoost	21.623	16.100	0.662
LSTM	20.785	15.716	0.683
GRU	18.047	13.629	0.752

displays the RMSE, MAE, and R² scores for all six models, from which it is apparent that the MSAFormer model exhibits the best predictive accuracy, marked by the lowest RMSE and MAE, as well as the highest R² score.

Lastly, the histogram of prediction errors (Figure 4) shows that the MSAFormer has a narrower error distribution, centralized around zero. This underlines its enhanced prediction performance as compared to the benchmark models, with smaller and fewer errors.

From these results, it can be concluded that the MSAFormer model presents significant improvements over traditional models in terms of predictive accuracy and the ability to effectively capture temporal dependencies in PM_2.5 concentrations.

4.3. Sensitivity Analysis of the MSAFormer Model

This section conducts a sensitivity analysis to determine the influence of the Meteorological Sparse Autoencoding (MSA) module and the parameter, Sparse Autoencoder Coefficient (λ), on the overall performance of the proposed MSAFormer model. The first experiment was conducted without incorporating the MSA module into the MSAFormer model, meaning that meteorological factor data were not utilized for the PM_2.5 concentration prediction. The second part of the analysis involved adjusting the Sparse Autoencoder Coefficient (λ), from 0.0 to 1.0 in increments of 0.1, to observe its impact on the model’s performance.

The experimental results are visually presented in Figure 5 and quantitatively summarized in

Table 6. Comparative performance of the MSAFormer model for different λ values and without the Meteorological Sparse Autoencoding module, based on RMSE, MAE, and R² scores.

Model	RMSE	MAE	R2
No MSA Module	15.869	11.635	0.792
λ = 0.0	11.506	8.901	0.893
λ = 0.1	12.069	9.348	0.882
λ = 0.2	12.039	9.361	0.884
λ = 0.3	11.936	9.369	0.886
λ = 0.4	11.112	8.691	0.898
λ = 0.5	13.954	10.774	0.848
λ = 0.6	13.619	10.608	0.856
λ = 0.7	19.013	14.644	0.746
λ = 0.8	18.788	14.511	0.746
λ = 0.9	19.417	15.027	0.727
λ = 1.0	19.579	15.163	0.719

. Figure 5 shows the error distribution of the MSAFormer model with varying λ values and the exclusion of the MSA module. An observable pattern from the graph is the decline in prediction error as λ increases, hitting a minimum at λ = 0.4, beyond which the error begins to rise, indicating an overfitting scenario.

The sensitivity analysis results shed light on the integral role of both the MSA module and the λ parameter in the MSAFormer model. Removing the MSA module leads to a substantial increase in error distribution, signifying the detrimental impact on the model’s performance. This indicates the importance of meteorological factors in achieving accurate PM_2.5 concentration predictions, thereby validating the relevance and efficacy of the MSA module. As detailed in Table 6, different λ settings and the absence of the MSA module result in varying RMSE, MAE, and R² scores for the MSAFormer model. An optimal λ value of 0.4 leads to the best model performance, with the lowest RMSE and MAE values and the highest R² score. However, the absence of the MSA module does not lead to the worst performance, even though it leads to a considerable decrease in the accuracy of the predictions. The least satisfactory results are seen when λ is set to 1.0. These findings, in line with Figure 5, underscore the crucial role of the MSA module and the appropriate tuning of the λ parameter in maintaining the predictive accuracy of the MSAFormer model.

The observed outcomes reinforce the architectural rationale underpinning the MSAFormer model. The implementation of the MSA module, through a specialized sparse autoencoder, proficiently extracts meaningful features from meteorological variables sourced from nine distinct monitoring stations situated throughout Beijing. This advanced feature extraction significantly augments the model’s predictive proficiency for PM_2.5 concentrations in the Haidian district. Moreover, these outcomes highlight the pivotal role played by the Sparse Autoencoder Coefficient (λ) in fine-tuning the balance between the sparsity of the feature representation and the generalization capability of the model. Notably, the model achieved optimal performance when the λ value was adjusted to 0.4. This finding suggests that a certain degree of sparsity within the autoencoder is beneficial in encapsulating critical meteorological features whilst concurrently circumventing overfitting, thus offering a compelling trade-off.

These findings unequivocally reaffirm the effectiveness and robustness of the MSAFormer model, underscoring its considerable potential for application in air quality prediction tasks, particularly in the realm of urban environmental management and public health.

5. Conclusions

In this study, we introduced MSAFormer, a transformative approach for PM_2.5 concentration prediction, that offers significant improvement over conventional methodologies. The MSAFormer model creatively combines the advantages of the Transformer architecture with a Meteorological Sparse Autoencoding (MSA) module to tackle the inherent complexity of multi-station meteorological data. The MSA module effectively encapsulates the non-linear relationships in high-dimensional data by extracting salient features, overcoming the limitations of traditional methods. The Positional Embedding module further flattens the sparse-encoded features, enabling streamlined data processing in the subsequent Transformer module. In the final module, a self-attention mechanism is employed to capture temporal dependencies in the input data, thereby predicting future PM_2.5 concentrations with increased precision.

Our experimental evaluation reveals that MSAFormer performs remarkably well in predicting PM_2.5 concentrations in the Haidian district. In our study, we compared the MSAFormer model with traditional methods such as SVM, RF, AdaBoost, LSTM, and GRU. Specifically, the MSAFormer model demonstrates improvements in all the considered metrics: it lowers the RMSE by 6.935 to 11.888, reduces the MAE by 4.938 to 8.761, and enhances the R² value by 0.146 to 0.266, compared to these traditional methods. Among these, the greatest improvement in R² is observed over RF, with an increase of 29.621%. These quantitative advancements corroborate the efficacy of our model and the relevance of deep learning in environmental meteorological data analysis.

However, like all research, ours is not without limitations. The MSAFormer model relies heavily on the quality of input data. As such, data inconsistencies or inadequacies might affect the predictive capabilities of our model. Furthermore, while our model demonstrates superior performance in the Haidian district, the generalizability to other geographical locations and environmental contexts remains to be explored.

For future work, we recommend several avenues. First, the model’s robustness could be strengthened by incorporating additional sources of data and conducting multi-site evaluation tests. Second, the MSAFormer model could be further refined and generalized to predict other meteorological phenomena and pollutants, potentially contributing to a broader scope of environmental science. Finally, the implementation of the MSAFormer model in a real-world setting, such as urban air quality management systems, would provide valuable insights into its practical performance and utility.