Deep Random Subspace Learning: A Spatial-Temporal Modeling Approach for Air Quality Prediction

Abstract

Decrease in air quality is one of the most crucial threats to human health. There is an imperative and necessary need for more accurate air quality prediction. To meet this need, we propose a novel long short-term memory-based deep random subspace learning (LSTM-DRSL) framework for air quality forecasting. Specifically, we incorporate real-time pollutant emission data into the model input. We also design a spatial-temporal analysis approach to make good use of these data. The prediction model is developed by combining random subspace learning with a deep learning algorithm in order to improve the prediction accuracy. Empirical analyses based on multiple datasets over China from January 2015 to September 2017 are performed to demonstrate the efficacy of the proposed framework for hourly pollutant concentration prediction at an urban-agglomeration scale. The empirical results indicate that our framework is a viable method for air quality prediction. With consideration of the regional scale, the LSTM-DRSL framework performs better at a relatively large regional scale (around 200–300 km). In addition, the quality of predictions is higher in industrial areas. From a temporal point of view, the LSTM-DRSL framework is more suitable for hourly predictions.

1. Introduction

The pervasiveness of poor air quality in both developing and developed countries has brought about a global threat, having huge negative impacts on the environment and health. According to the Health Effects Institute (HEI), more than 90% of the world’s population, around 7 billion people, live in unhealthy air environments [1]. Many studies across the world seek ways to quantify the magnitude of health harm caused by air pollution through systematic scientific efforts (e.g., [2]). Based on World Health Organization (WHO) data, about 4.2 million deaths happen every year due to ambient air pollution. Premature diseases such as lung disease, heart disease, and stroke, etc., have been noted as being mainly caused by air pollutants. The direct reason is that minors and babies are exposed to air pollutants for prolonged periods [3,4]. Long-term exposure can also cause ill health, such as new-onset type 2 diabetes among adults [5]. To protect people from these adverse health impacts, people are encouraged by the American Lung Association to pay more attention to air quality forecasts and take timely precautions [6]. Air quality forecasts making predictions of air pollutant concentrations have become imperative and urgent necessities for air quality control. These forecasts play a role as an essential component of air pollutant control strategies implemented on a regional scale. For instance, an O₃ action plan employed in French Bouches du Rhône uses air quality forecasting as a tool to trigger an emission reduction strategies selection process [7].

In the past few decades, researchers have devoted their efforts to exploring trends in air pollutant concentrations. Existing prediction studies can be generally separated into three categories based on modeling methods, namely, numerical methods, statistical methods, and machine learning methods. Numerical models for air quality prediction, also called chemical transport models, model the transport velocity, diffusion path, and possible chemical reactions during pollutant movements so that the concentration of pollutants can be predicted using mathematical algorithms [8]. The input matrices for these models commonly include source terms (historical concentrations of air pollutants) [9], meteorological conditions (wind direction, humidity, and temperature, etc.) [10], emission source parameters (location and height, etc.) [11], terrain elevations [12], and properties of blocks in the path of pollutant movements [13]. Various numerical models have been developed to improve forecasting processes, such as the Community Multiscale Air Quality (CAMQ) model, the Weather Research and Forecasting (WRF) model, and the WRF model coupled with Chemistry (WRF-Chem) [14,15,16]. Furthermore, the COPERNICUS project uses a multi-model ensemble approach to provide regional air quality forecasts since the results derived from such an approach are more robust and are of better quality [17]. Although nowadays numerical models are still popularly implemented in forecast systems by governments, in order to achieve high-quality prediction performance, a relevant high portion of input data requires more accuracy than the accuracy which is currently available [18]. In addition, most practical air environment situations are complex and hard to express mathematically. This is another significant limitation of the numerical models available at present [19].

Plenty of statistical approaches have been used in the air quality prediction field. Traditional statistical methods have been widely employed to deal with air quality prediction problems. Good predictive effects have been gained through modeling and understanding of data probability generation mechanisms [20]. These models are substantiated by validating correctness of the probability distribution hypothesis for data. However, the assumptions for data probability distribution also imply limitations to further development of the traditional statistic model for practical applications [21].

With the development of artificial intelligence and big data analytics, prediction methods based on machine learning technologies are becoming increasingly common. These kind of models directly explore complicated hidden patterns in data, requiring neither hypothetical distributions of variables or data nor an in-depth understanding of physical or chemical properties of air pollutants. The models’ quality is basically judged by their predictive accuracy and effectiveness [22]. Commonly used machine learning algorithms include multiple linear regression (MLR), random forest (RF) [23], support vector regression (SVR) [24], artificial neural networks (ANN) [25], and so forth. Previous studies have found that machine learning methods achieve excellent performance due to the nonlinear relationships within data, meaning that these methods are better suited to parameter statistic models and need less training time than dispersion models [26,27,28]. Deep learning algorithms, as a relative newcomer, have obtained outstanding prediction or detection performances in various application domains such as speech recognition, natural language processing, and computer vision [29]. Some pioneering research has used deep learning techniques to address air quality prediction problems.

In this paper, we developed a long short-term memory-based deep random subspace learning (LSTM-DRSL) framework to achieve high forecast accuracy. The framework employs a deep learning long and short-term memory model and combines it with a random subspace learning approach. As discussed above, deep learning methods make a better fit than traditional numerical and statistic methods since forecasting accuracy becomes more important than interpretability [27]. The LSTM model is a popular variant of the recurrent neural network (RNN) method, a famous deep learning algorithm. It performs better than traditional RNN when facing gradient vanishing or gradient explosion problems [30]. Besides, when using LSTM to make predictions or detections, it considers the effect of previous values on the current one in the model calculation [31,32,33]. This feature makes LSTM one of the best suited models for air quality prediction problems since temporal dependence is a typical phenomenon observed in air pollutant concentration series. We further combined LSTM with RSL to obtain the proposed framework, in which individual LSTM models employing different subsets of features serve as base models. The RSL approach is incorporated here to achieve more generality. In previous research, preliminary frameworks of a simpler combination of deep learning methods and RSL have been applied in several domains, such as crop disease prediction [34], stock price manipulation detection [35], and financial market prediction [36]. The results of performance evaluation in these studies show that these frameworks outperform all baseline models and are capable of dealing with prediction and detection problems. The main differences between the framework proposed in this paper and previous frameworks exist in the base models selected for RSL and the spatial-temporal feature engineering processes.

Spatial-temporal feature engineering for air quality prediction is designed to capture the internal dependence between input variables. While previous research has mainly focused on predicting air quality using time-series data, the LSTM-DRSL framework considers spatial dependences caused by air pollutant interactions and transmissions as well. The direction of pollutant transmission is affected by the relationship between relative positions of pollutant emission sites to local monitoring sites and meteorological conditions like wind direction. The spatial dependence caused by fluidity of air pollutants is featured in our proposed method through two steps: zoning and regional synthesizing.

Real-time pollutant emission data are incorporated into the prediction framework to improve the prediction accuracy. Normally, the accuracy of the forecast increases as the data become more comprehensive and the quality of the data becomes better [37,38]. Although the emission data in our study are limited to stationary sources such as heating plants, it is reasonable to suppose that the incorporation of these data can improve the prediction performance and that replacing the estimation of emissions with the real-time observations can enhance the forecast ability. Moreover, the addition of real-time pollutant emission information into the prediction model can help with potential simulation of air planning effects [38]. Air planning refers to taking a series of structural measures to improve air quality and is another important activity in this domain in addition to forecasting. It is necessary to analyze what would happen when these structural measures are applied. Previous studies have shown that emission reduction policy and emission source relocation are effective in air quality control [39]. Taking the real-time emission data as part of the model input is helpful for decision-makers to estimate the performance of related strategies based on model output and make air planning decisions. However, as far as we know, there is still no research that includes real-time hourly emission data, other than emission inventories, in air quality forecasts. To mitigate and control the inaccuracy caused by the errors and hysteresis contained in raw data, real-time emission data is first employed in our prediction framework.

The main contributions of our study are the following. First, we propose a novel LSTM-DRSL framework using multiple data sources and spatial-temporal features to obtain air quality predictions. Second, we incorporate pollutant emission data into the model and design a spatial-temporal analysis approach for them. Third, a LSTM model combined with random subspace learning is developed and adopted in our framework to predict air pollution concentrations. Fourth, we demonstrate the effectiveness of our proposed methods by systematical comparative experiments on practical data.

2. Method

2.1. Overview

The method we propose in this study focuses on tackling the problem of lacking prediction accuracy. To address this issue, we introduce an important input data dimension, pollutant emission, into the prediction model. The extraction of emission features is based on spatial-temporal dependencies. The temporal autocorrelations existing in air pollutant concentrations are effectively captured by applying LSTM as the base model. According to the characteristics of prediction tasks, a data mining model which we called deep random subspace learning is developed to make good use of the features in a big data context. The DRSL model, which integrates the deep learning algorithms of LSTM and ensemble learning methods is represented as LSTM-DRSL below. The framework of the proposed method is shown in Figure 1.

2.2. Data Collection and Preprocessing

During data collection, we collected datasets for the required variables of our prediction model from different data sources. The formation and propagation of air pollutants are complicated processes and have significant correlations with environmental circumstances. To achieve accurate predictions, it is crucial to contain all the information of the propagation process in the input to the prediction model as much as possible. We approached responsible government departments to get approval for obtaining data through the application programming interface (API) in their systems, while other data were collected from public government websites, such as Earthdata, via which data are regularly released by NASA (https://earthdata.nasa.gov/). The collected data have been classified into five types of variables, as illustrated in

Table 1. Independent variables. Legend: PM_2.5, particulate matter with an aerodynamic diameter of 2.5 $\mathsf{\mu }\mathrm{m}$ or less; PM₁₀, particulate matter with an aerodynamic diameter of 10 $\mathsf{\mu }\mathrm{m}$ or less.

Type	Variables	Observations	Data Source
Air Quality	Ground pollutant measurement (GPM)	Hourly concentrations of PM2.5, PM10, CO, NO, NO2, NOX, O3, SO2	National air quality monitoring network
	Atmospheric air quality (AAQ)	Aerosol optical depth, total ozone burden	MODIS
Meteorology	Surface meteorological measurement (SMM)	Hourly atmospheric pressure (hpa), humidity (%), temperature (°C), wind speed (m/s), wind direction (deg)	Automatic weather monitoring system
	Atmospheric meteorology (AM)	Atmospheric stability, moisture, atmospheric temperature, atmospheric water vapor	MODIS
Emission	Pollutant emission (PE)	Hourly emissions of SO2, NOX, particles (kg/h); hourly benchmark gas flow (m3/h)	National key monitored enterprise

.

Ground pollutant measurement variables (GPM). China established a national-scale ground monitoring network for typical pollutants in late 2012, since the air quality problem in various regions has become severe [40]. The network consists of more than 1400 monitor stations covering 300 cities. Each station site monitors multiple air pollutants including PM_2.5 (particulate matter with an aerodynamic diameter of 2.5 $\mathsf{\mu }\mathrm{m}$ or less), NO₂, and SO₂, etc. The concentration data of these pollutants are generated at an hourly frequency. Surface meteorological measurement variables (SMM). As stated in previous literature [41], pollutant concentrations are strongly associated with meteorological conditions as the factors affecting physical processes of pollutant generation and dispersion are associated with meteorological variables like temperature. Therefore, it is important to integrate various meteorological variables within the prediction model. Atmospheric air quality variables (AAQ) and atmospheric meteorology variables (AM). Both AAQ and AM are derived from satellite data collected by sensors like the moderate-resolution imaging spectroradiometer (MODIS). These two types of variables have been widely used to estimate air quality (e.g., PM_2.5 exposures) in previously unmonitored areas [42]. Among these remote sensing variables, the aerosol optical depth (AOD) has been proven to be one of the most valuable predictors [43]. Pollutant emission variables (PE). It is universally acknowledged that pollutant emissions are one of the major air pollution sources. China launched an initiative in the year 2013 to nationally monitor crucial industrial enterprises’ emissions, which required the polluting enterprises to self-monitor and publish their data to a government platform. According to the relevant regulations from the Ministry of Ecology and Environment, the hourly average of an enterprise’s emissions should be obtained through automatic monitoring equipment and should be published in real time. It should be noted that the emission data used in our study can only express the emission situation of “point” sources due to the data acquisition limitation.

In summation, there are five kinds of independent variables in the prediction framework, i.e., GPM, SMM, AAQ, AM, and PE. From the view of connotation, these variables can be grouped into three categories, namely, air quality, meteorology, and emission (see Table 1). For each individual dataset from a different source, the data can be transformed into time series by clustering data based on spatial information first regardless of the type of variable and the initial form of the data. The extraction of spatial information and the unification of data forms lay the foundation for data preprocessing.

During data preprocessing, we first integrated the multisource data from all the original datasets by making full use of time and space information, as shown in Table 1. The integration process helps obtain a unified representation of input variables that can improve the quality of data mining. The GPM data and SMM data were first integrated based on the latitude and longitude value of the monitoring sites as well as the data’s acquisition time. For the sites serving as monitoring stations for both air pollution and weather conditions, we were able to simply put these two types of data together in one sample. We constructed samples for other sites based on the proximity principle, which used the SMM data collected by the nearest automatic weather monitoring station. Remote sensing data were the average values assigned to each 1 km $\times$ 1 km space grid in the area; we combined them with the GPM and SMM data in a way that adopted the AAQ and AM data of the grid occupied by the station site. For the monitoring sites that fell on the grid boundary, we calculated the average of the AAQ and AM variable values of all the grids sharing their edge or vertex and took the average as the corresponding values for the site.

We then imputed the missing data for the sample set constructed by the integration process so that substantial bias created by missing information could be avoided and the analysis of data be more efficient. The data form for the samples were unified as time series. The imputation values were determined only by the values before and after missing data in the temporal dimension. Our fill values were calculated by adhering to the following formula in the data imputation process:

$$Input={\alpha }_{1}av{g}_h+{\alpha }_2\left({\beta }_1{V}_{forw}+{\beta }_2{V}_{backw}\right)$$

where $Input$ is the imputation value of the missing data, $av{g}_h$ is the average of all the valid data collected at $\mathrm{h}$ o’clock, which is also the time of situation that is supposed to be reflected by the missing data, $V_{forw}$ and $V_{backw}$ are the nearest valid values after and before the missing value, ${\beta }_1$ and ${\beta }_2$ are the weights determined by the ratio of distance between the missing value and the nearest one (before or after) and the distance between the two nearest valid values, respectively, before and after the missing one, and $\alpha$ is the parameter representing the weight of the corresponding component in the addition operation which are experimentally determined to optimize the filling effect. Adhering to previous research, we set ${\alpha }_1={\alpha }_2=0.5$ in our experiments [44].

In addition, we performed reprocessing steps like data cleansing and data standardization. We conducted data cleansing by identifying the irregular and abnormal data caused by fortuitous equipment malfunctions and then corrected them according to relevant monitoring standards. During data standardization, the effect of dimensions was removed through defining a specific base dimension for each variable and transforming them into a unified base dimension.

2.3. Emission Features Based on Spatial-Temporal Analysis

Spatial dependence in a variety of monitoring data denotes the circumstance in which observations associated with one site depend on those at surrounding sites. For instance, particulate matter (PM_2.5) concentrations at a particular position will subsequently rise as an outcome of the wind blowing particles from a nearby region to the local area. Considering the latent regional interactions observed in multipoint monitoring, there is a critical need to quantify and model the indeterminate spatial dependence. In this study, we proposed an elementary design to incorporate spatial information and the internal dependence on them into the forecasting framework.

Determining and partitioning the neighboring region of a site is a principal consideration. The distance between the nearby pollutant emission sites and the target monitoring site plays a crucial role in the process since it is believed to have a substantial impact on the spatial dependence between sites. Generally, the distances between encompassing sites and the central site vary enormously among the different sites. As shown in Figure 2a, three circles employ different diameters but the same center to determine and separate the surrounding area of the site located at the center. The size of diameters should range within the distances of any two monitoring sites. The three diameters applied in this study were 1 km, 10 km, and 100 km, respectively. The diameter values were set based on two principles. The first one referred to previous research. Based on this principle, we set the minimum diameter as 1 km, since the maximum ground concentrations of air pollutants calculated in most studies are less than 1 km. The second principle was to ensure that there were emission sources in each subzone. For each monitoring site, around 60% of the distances between the emission sources and the site were less than 10 km and the maximum distance was less than 100 km. To take into account the orientational relationship of sites and the influence of the wind direction, we further separated each circular area into eight equal pieces by four lines passing through the common center. All in all, the marked neighboring regions (rounded by the outermost circle) were ultimately divided into several parts where closer regions had a finer granularity and farther regions had a coarser granularity. It should be noted that the partitioning process has practical significance since it leads to an upper bound to both the number of variables and the number of corresponding model parameters in prediction models. This process helps control the total input data amount and reduce the training time.

The variable data of pollutant emission sites falling within the same part of the surrounding regions were be aggregated in different ways considering the nature of the data. The emission sites were projected onto the corresponding piece of neighboring regions based on their geographic coordinates. For PE data, we summed the emission volume for each pollutant and took the sum as the representation of the discharge load of pollutants in certain regions. Eventually, each part of the neighboring region obtained values for a set of PE variables. Every center site added 24 extra sets of PE variable values representing the 24 parts of its neighboring regions, respectively, in the sample. As a result, the spatial dependence and relevant spatial information of pollutant emissions were integrated into the prediction model.

Temporal dependence refers to the impact of past situations on the status quo. This dependency has been broadly considered in prediction models employed to different domains, including agriculture, environment, finance, and so forth [36,45]. We prepared the data in chronological order so that the prediction model could mine the dependence through time-series data. We took the variable $T$, representing the length of the time series, as the key parameter to enhance the predictive capability. That is, when forecasting the air quality at time $t$, the sample included the variable data at times from $t-1$ to $t-T$ (as illustrated in Figure 2b). In agreement with previous research, we set $T=3$ in this study [44]. For each time slice, the spatial information from nearby stations within the circle of the outermost diameter was added on the basis of temporal dependence. In addition, this kind of dependency is believed to exist among the panel data of air pollutant concentration and meteorological conditions as well, since these factors vary by time and affect their future values by influencing the propagation process of pollutants through air. Hence, temporal dependence was also considered for the remaining variables, including GPM, SMM, AAQ, and AM.

2.4. Modeling and Prediction

The architecture for the LSTM network applied in our study is shown in Figure 3. As can be observed from the figure, each dotted box represents an individual module which transforms input into output. Repeating these modules in the form of a chain enables LSTM to learn long-term dependencies. The memory cell layer, the major difference between LSTM and traditional RNN, acts as a conveyor belt to connect the information, meaning the LSTM can remember information for long periods of time. The introduction of memory cells ameliorates the gradient training through using the cell to determine the extent of the absorption of previously acquired knowledge and the extent of updating the hidden state. Gate mechanisms are designed to regulate the ability to remove or add information, and an LSTM here has three of these gates (denoted $G_i$, $G_f$, and $G_o$).

In a traditional RNN network, as introduced by Elman (1990), the hidden state is simply updated from the previous state $H_t$ through the adoption of new input information $I_{t+1}$ as shown in the formula given below:

$$H_{t+1}=f\left(H_t,I_{t+1}\right)$$

(1)

Yet, with the addition of gate mechanisms built on the linearly-connected memory cells, the update process becomes more complicated for the hidden layer. To explain the process clearly, we basically split the updating course into three phases.

When a new sample, $I_t$, is put into an LSTM model, the most recent information contained in the input sample will first be mixed with the information accumulated in the previous state of the hidden layer $H_{t-1}$ to estimate a value for the memory cell unit, $\widetilde{M_t}$, containing all the latest information, which can be calculated as

$$\widetilde{M_t}=\tanh \left(U_M{I}_t+V_M{H}_{t-1}\right)$$

(2)

where $U$ and $V$ represent the coefficient matrices of the input state and last hidden state optimized by training, separately. The gate mechanism facilitating decision-making function depends on three vectors: $G_{i,t}$, $G_{f,t}$, and $G_{o,t}$. These gating vectors are iterated in every step. $G_{f,t}$ and $G_{i,t}$ are used to control the extent to which the memory cells should be erased or updated. These two vectors, as input gate and forget gate, respectively, have entries ranging from 0 to 1. On the basis of these two gates at time $t$, we can determine the ultimate value of $M_t$ as follows.

$$M_t=G_{f,t}\odot M_{t-1}+G_{i,t}\odot \widetilde{M_t},$$

(3)

where the operation $\odot$ denotes elementwise multiplication between the memory cell states and the gate vectors. Subsequently, the hidden state $H_t$ is able to be obtained by $M_t$ and the output gate $G_{o,t}$, as represented in the following:

$$H_t=G_{o,t}\odot \tanh M_t$$

(4)

The transition equations of those three gates mentioned above are the following:

$$G_{i,t}=\sigma \left(U_i{X}_t+V_i{H}_{t-1}+P_i{M}_{t-1}\right),$$

(5)

$$G_{f,t}=\sigma \left(U_f{X}_t+V_f{H}_{t-1}+P_f{M}_{t-1}\right),$$

(6)

$$G_{o,t}=\sigma \left(U_o{X}_t+V_o{H}_{t-1}+P_o{M}_t\right),$$

(7)

where $P$ represents the coefficient matrices of the memory cell state and $\sigma$ denotes the logistic sigmoid function $f_{\sigma }$, i.e.,

$$f_{\sigma }\left(x\right)=\frac{1}{1+e^{-x}}.$$

So far, the LSTM units at time $t$ containing a hidden state $H_t$, a memory cell $C_t$, and three gates including an output gate $G_{o,t}$, a forget gate $G_{f,t}$, and an input gate $G_{i,t}$ have all been presented.

As for the training, we adjusted parameters $M$, $N$, and $P$ through the backpropagation through time (BPTT) method (see Appendix A) to minimize the loss function constructed on mean squared error (MSE):

$$L_{MSE}=\frac{1}{N}{\displaystyle \sum }_{t=t_0}^{t+N}\Vert F_t-True{V}_t{\Vert }_2^2$$

(8)

where $N$ is the batch size employed in the mini-batch BPTT training, $True{V}_t$ represents the ground-truth value of the prediction target at time $t$, and $F_t$ denotes the predictive value obtained from the equation, as in

$$F_t=f_F\left(H_t\right)=W_F{H}_t+b_F$$

(9)

where $f_F$ represents the activation function in the forecast layer, $W_F$ represents the weight matrix, and $b_F$ represents the bias term. The squared Euclidean norm, expressed as $\Vert \cdot \Vert$ in Formula (9), represents the prediction error of each timestamp.

We then developed a data mining model integrating the LSTM model and random subspace ensemble (RSE) method, referred to as the LSTM-DRSL. The RSE approach was introduced into the air quality prediction task since the total number of constructed features ($N_F$) reached several hundred and RSE achieves great performance in modeling high-dimensional data. In our proposed framework, we employed the bootstrap method to construct random spaces by randomly sampling ${\mathrm{n}}_f$ emission features ($n_f\le N_F$) during the feature selection process. It is worth noting that the number of emission features in each random subspace has been confirmed to be sufficiently large to demonstrate the usefulness of the pollutant emission. We repeated the sampling process $N$ times so that we could obtain $N$ random subspaces. For each random subspace, we combined emission features with air quality and meteorology features to train an LSTM predictor. Then, the predictions resulting from the base LSTM classifiers served as inputs to the stack classifier. The training process of LSTM-DRSL is interpreted detailly in the following pseudocode (Algorithm (1)).

Algorithm 1: Learning LSTM-DRSL through BPTT

Input: Training samples $S$;

Output: Weight matrices $U_i$, $V_i$, $P_i$, $U_f$, $V_f$, $P_f$, $U_o$ $V_o$, $P_o,$ and ${\mathrm{W}}_F$ for $N$ base models respectively ($N$ is the number of random subspaces); stack model $StackM$;

1: Initialize $U_i$, $V_i$, $P_i$, $U_f$, $V_f$, $P_f$, $U_o$ $V_o$, $P_o$, and $W_F$ randomly; 2: Set prediction time window $T_w$;

3: Sort input samples in chronological order from $t_o$ to $t_T$;

4: Set time stamp $t=t_o+T_w$; 5: Initialize the serial number of random subspaces $n=0$; 6: // Build random subspaces 7:  while $n\le N$ do 8: Randomly sample $n_f$ of emission features and integrate them with other 9: features into $S_n$; 10: $n=n+1$; 11: end while 12: $n=0$; 13: // Training the base LSTM models and the stack model

14: while not converge do

15: while $t\le t_T$ do

16:  while $n\le N$ do

17:   Compute $F_t$ based on $S_n$; (Formula (1)~(7), (9))

18:   Compute $L_{MSE}$ at $t$; (Formula (8))

19:   Update weight matrices for model $Pre{M}_n$; (BPTT) 20:   Obtain prediction value $Pr{e}_n$; 21:   $n=n+1$; 22:  end while 23:  Train the stack model $StackM(Pr{e}_n) \left(n\in \left\{n|0\le n\le N\right\}\right)$;

24:  $t=t+1$; 25: end while

26: end while

3. Experiment

3.1. Data Description

We collected observations of the major air pollutants including PM_2.5, PM₁₀, CO, NO, NO₂, NO_X, O₃, and SO₂. These data were generated from monitoring networks set up by the Chinese government. A dataset containing 451,509 records was obtained from an air quality monitoring system through an authorized API. The data range from January 2013 to September 2017. Other related data from data sources including MODIS and automatic weather monitoring systems were matched to the air quality observations by time and location in the preprocessing progress. After that, we excluded samples from January to December in 2013 since the meteorological data collection mechanism had not been established by then, and 275,362 records remained. Data representing pollutant emission conditions were also incorporated as supplemental spatial information. Pollutant emission data were regularly reported by 179 national crucial monitoring enterprises. However, the enterprise monitoring system was not established until 2015. We dropped the data collected before the monitoring system was operated and the total amount of records further decreased to 212,424. The fluctuation of air pollutant concentration data is shown above (Figure 4).

3.2. Evaluation Metrics

A ten-fold cross-validation technique was employed in our study. The original training dataset was randomly partitioned into ten subsets, each of which contained approximately one-tenth of the training data. This was referred to as the cross-validation process and in it we used nine subsets as training data and the remaining subset as testing data. The process was then repeated 10 times, during which every subset was taken as testing data once. Each round of trial yielded performance results. The average values of each metric were used as the estimates of the predictive capacity of algorithm.

When evaluating models, the data was first sorted into chronological order and then the last 20% were used as the hold-out test set. We trained each model on the remaining 80% of the data and applied the 10-fold cross-validation to verify the effectiveness of the model. The cross-validation process also helped tune models’ parameters and complete model selection.

We calculated statistical indicators including root-mean-square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE) between observations in the hold-out test set and predictions for them to assess the prediction accuracy of models for the entire study period.

3.3. Experiment Design

We performed five sets of experiments to methodically appraise the efficacy of our proposed framework. Each set of comparative experiments were designed to assess the effectiveness of one innovative aspect of our method. The first set experiments were set to compare the performance of the LSTM algorithm which we applied in this work to the benchmark machine learning models discussed above (in Section 1). It is worth noting that the results of the experiments were also an estimation of the usefulness of considering time dependency, since LSTM is the only algorithm for which its the computational processes include time-relevant mechanisms. In the second comparative experiment set, we made air quality forecasts using and without using the emission features and assessed the improvement when adding feature dimensions. As shown in Section 2.3, the pollutant emission variables were reorganized based on the spatial environment before being incorporated into the models. Accordingly, the experiments assessed the validity of spatial information as well. The third set of experiments compared the predictive performance before and after combining random subspace ensemble learning with LSTM models. In the fourth set of experiments, we tested the performance of the proposed framework across variable regions. The input data was separated by sub-regions so we could examine whether the framework achieved the same performance given consideration of the regional variations. The fifth set of experiments were designed to see the capability of the framework from a temporal point of view, in which we observed if the prediction results aggregated per season were in the same quality.

4. Results and Analysis

4.1. Comparison with Baseline Models

In this subsection, a few machine learning technologies which are commonly used in previous research were employed as benchmark methods. As introduced in Section 1, MLR is one of the most widespread models in the air quality prediction field. ANN, SVR, and RF are also popular machine learning techniques. They usually achieve better forecast performance through exploiting the latent relationship between independent variables and dependent variables.

The features data were organized in a time sequential way so that the input for time-independent models could be equivalent to those for time-dependent ones. The input for time-independent models turned into a collection of sample data from t–T to T, where t represents the time that the forecast value corresponds to and T indicates the length of the time window as introduced in Section 2.3 (according to pre-experiment results, T = 3 here [44]).

The forecast of the PM_2.5 concentration, which has become a top public concern in recent years, was taken as an example to evaluate the validity of all models mentioned above. Although not shown in tabular forms in this paper, the predictive effectiveness for other pollutants exhibited the same trend. As illustrated in

Table 2. Performance of prediction models using only non-spatial features. Legend: RMSE, root-mean-square error; MAE, mean absolute error; MAPE, mean absolute percentage error; MLR, multiple linear regression; ANN, artificial neural networks; SVR, support vector regression; RF, random forest.

Predictors	RMSE	MAE	MAPE (%)
MLR	14.761	12.709	32.523
ANN	13.686	11.894	29.132
SVR	13.454	11.817	28.685
RF	12.896	11.273	27.508
LSTM	12.241	10.534	24.867

, the LSTM algorithm outperformed all the baseline models. The results demonstrate that LSTM attained 5.07%, 6.56%, and 9.60% improvement over the best baseline model RF in RMSE, MAE, and MAPE, respectively. Additionally, the average improvement rate between the LSTM model and the time-independent models achieved 10.43%, 11.49%, and 15.27%. By applying algorithms containing sequence learning mechanisms, air pollutant concentration can be predicted with an around 75% precision on the hold-out test dataset. This suggests that including temporal analysis within the prediction process leads to improvements in prediction accuracy.

We also performed a Wilcoxon signed-rank test to assess whether there was a statistically significant change in the pair of values predicted by the matched pair of models. Each pair of models contained the LSTM model and one of the baseline models. As shown in

Table 3. Wilcoxon signed-rank test results on prediction performances.

Predictors	${\mathit{H}}_0:\mathbf{There}\mathbf{are}\mathbf{No}\mathbf{Significant}\mathbf{Differences}\mathbf{in}\mathbf{the}\mathbf{Sample}\mathbf{Means}$
	LSTM
MLR	0.00 ***
RF	0.00 ***
SVR	0.00 ***
ANN	0.00 ***

*** $p-\mathrm{value}\le 0.01$.

, differences existed between the above two types of models and they were all significant. Since improvement can be realized by replacing the predicting model with LSTM, the test results further demonstrate the validity of the proposed methods.

4.2. Incremental Effect of Combined Spatial Features

To test the incremental effect of combining spatial emission features, we added spatialized variables to prediction frameworks based on different models and then predicted concentrations for different air pollutants. First, we compared the performance of all five models using and not using emission feature sets to evaluate the incremental effect of the combined spatial features. In this experiment, PM_2.5 was also taken as an example and the best-performance model was selected for further experiments. The results of this pre-experiment are recorded in Figure 5. Second, we applied the previous best-performance model, i.e., the LSTM model (as shown in Figure 5), to all the major pollutants so we could estimate the generality of our prediction framework to different pollutants. The performance of LSTM on eight pollutants is shown in Figure 6.

Comparing the performance of all models shown in Table 2 and Figure 5, it is evident that the performance was enhanced when spatial features were incorporated, achieving average improvements by 6.079%, 6.179%, and 9.274% on each evaluation, respectively. The LSTM model attained the best predictive capacity with an around 80% accurate rate. However, for the MLR model, the performance of adding spatial emission features was somewhat worse, which demonstrated that more features do not necessarily lead to effectiveness. Instead, the performance on hold-out test datasets may be decreased when overfitting occurs.

We trained predictors for each of the eight major air pollutants correspondingly. The results applying the best performance model (LSTM) are presented in Figure 6. The average MAPE for each pollutant is 20.908%, with the minimum and maximum values being 17.327% (CO) and 23.902% (PM₁₀). The average improvements of RMSE, MAE, and MAPE are 8.069%, 8.235% and 11.965%. The positive influence that spatial information can contribute to predictive precision was further confirmed by these evaluation results.

From the performance of the two different feature sets above in the comparison experiments, it can be seen that putting in spatial features is essential for air quality prediction process. The best prediction performance was observed for the LSTM-based predictor whose input incorporated spatial features. Moreover, we note that the spatial analysis is universally effective for all models and contexts. This further implies the potential for the spatial analysis approach to be applied to other advanced techniques or research problems.

4.3. Overall Improvement with Random Subspace Ensemble

Random subspace ensemble learning was utilized in our framework to enhance the air pollutant concentration prediction by building subspaces and training models for each subspace. According to the principle of random subspace ensemble learning introduced in Section 2.4, various but a sufficient number of features should be selected to develop subspaces to seek a higher generalization ability. The feature selection process can be illustrated as a bias-variance tradeoff which influences the effectiveness of the random subspace ensemble.

The random subspace ensemble learning mechanism was employed in the LSTM-based framework (the best predictors demonstrated in Section 4.2) with spatial information added. The evaluations detailed in

Table 4. Performances using LSTM combined with random subspace ensemble.

TASKS	RMSE		MAE		MAPE (%)
	LSTM	LSTM-DRSL	LSTM	LSTM-DRSL	LSTM	LSTM-DRSL
PM2.5	11.138	10.537	9.585	9.094	21.295	20.057
PM10	18.197	17.252	15.812	14.899	23.902	22.283
NO	0.664	0.625	0.596	0.564	20.006	18.957
NO2	7.40	7.198	6.328	6.128	20.106	19.141
NOX	5.91	5.655	5.129	4.884	22.790	21.799
SO2	4.454	4.265	3.846	3.665	21.787	20.527
CO	0.138	0.133	0.120	0.115	17.327	16.683
O3	8.955	8.562	7.723	7.344	19.931	19.029
Average Improvements		4.501%		4.763%		5.124%

reveal the overall performance of the prediction framework for each pollutant. The predictive results were effectively improved after adding random subspace ensemble learning. The RMSE, MAE, and MAPE of major pollutant concentrations decreased by 4.501%, 4.763%, and 5.124% on average. The greatest increase occurred when predicting PM₁₀ concentrations, for which the MAPE was decreased by 6.774%, followed by 5.815% for PM_2.5 and 11.815% for SO₂. The highest accuracy appeared in CO concentration prediction, with an accuracy of around 82%. The mean MAPE of CO prediction results dropped from a previous 19.777% to 17.327%. Therefore, we can conclude from the third set of experiments that random subspace ensemble learning can help improve the performance of air quality prediction.

4.4. Performance Comparison with Consideration of Spatial Variations

We verified the quality of the predictions at a smaller regional scale to see how our framework performs with consideration of spatial variations. Liaoning province is a main industry base in China. The urban agglomeration of central Liaoning, which suffers from severe air pollution, is comprised of eight cities, i.e., Shenyang, Anshan, Fushun, Benxi, Yingkou, Liaoyang, Tieling, and Fuxin. We applied our framework to predict PM_2.5 concentrations for each city in this urban agglomeration and compared the overall performance to previous prediction performance (at a larger regional scale). The performance of prediction models for PM_2.5 concentrations in these cities are contrasted with each other in Figure 7.

As shown in the following figure, the overall RMSE, MAE, and MAPE values for these eight individual cities are higher than those values for a larger regional scale (RMSE = 10.537, MAE = 9.094, MAPE = 20.057, as shown in Table 4). We suggest that the combination of RSL is a possible reason for our framework performing better at the larger regional scale.

In addition, for each evaluation metric, the comparison among these sub-regions show a similar result. It can be seen in the figure that the southern cities of the urban agglomeration, such as Yingkou, Benxi, and Anshan, have smaller forecasting errors, while the northern cities including Tieling and Shenyang experience lower accuracy. Through further analysis, we found that most steel plants and petrochemical factories in this area are in the southern region, which has a plain terrain, and the northern region has a higher terrain and is mostly mountainous areas. We suppose that the difference in factory pollution emission between south and north may be a contributing factor to this comparative result.

4.5. Performance Comparison with Consideration of Temporal Variations

We examined the capability of the LSTM-DRSL framework from a temporal point of view by segregating the PM_2.5 forecasting results per season. The seasons were characterized following the classification of the China Meteorological Administration as winter (November, December, January, February, and March), spring (April and May), summer (June, July, and August), and autumn (September. and October). A characteristically seasonal variation of PM_2.5 concentrations can be observed from Figure 8. In this figure, we also used dark blue bars to present the MAE of the prediction model and drew a polyline based on the right axis to better compare the MAE of daily predictions to hourly predictions.

As shown in Figure 8, PM_2.5 levels peaked in winter for the seasonal cycle due to increased fuel burning for heat. However, the model usually performed better in winter, followed by autumn, spring, and summer. We think that the incorporation of pollutant emission information on heating plants is a possible reason for our framework to perform best in winter and worst in summer.

When comparing between the MAE in Figure 8a,b, the quality of daily predictions is poorer than hourly predictions. We believe that the application of LSTM accounts for the better performance of the hourly concentration prediction. The reason to consider the LSTM model as a top priority is that the main difference between the daily and hourly prediction models is the $\mathrm{T}$ parameters of the LSTM models ($\mathrm{T}=1$ in daily predictions and $\mathrm{T}=3$ in hourly predictions). The parameter $\mathrm{T}$ represents a time interval limitation. When the time interval between the time point of the historical data and prediction is smaller than the T parameter, it can be incorporated into an input for a prediction framework. Since the advantage of the LSTM model is learning long-term dependence, it cannot exhibit its advantage when the parameter value is small.

5. Conclusions

In recent years, air pollution issues worldwide have become increasingly serious, resulting in many adverse health outcomes [46]. This study focused on air pollution predictions and proposed a LSTM-DRSL framework to estimate future concentrations of major air pollutants. The input variables were gathered from multiple sources including remote sensing satellites as well as mainstream monitoring networks. Feature engineering based on spatial-temporal analysis is designed to capture the dependence of air conditions on nearby pollutant emissions and the influence from a previous state to the present state. The LSTM-DRSL model was developed to make concentration predictions with higher stability and suitability. We employed our LSTM-DRSL framework to forecast air quality and conducted systematical comparative experiments to demonstrate the effectiveness with regard to each aspect.

In the empirical analysis, we carried out five sets of comparative experiments on real-world datasets to assess the effectiveness of our framework. The results of the first-set experiments showed that LSTM significantly outperformed all the benchmark machine learning models. We could draw the conclusion from these results that the introduction of LSTM models to air pollution forecasts is valuable and applicable. The second set of experimental results revealed improvements with the addition of spatial features in both baseline models and the LSTM model from which we could see the validity of spatial feature engineering. In the third set of experiments, the forecast results evinced great enhancements after combing random subspace learning technique with the LSTM model, which achieved the best performance compared with the above two experiments. Overall, the completed results of the first three comparative experiments displayed the competence of the LSTM-DRSL framework in air pollutant concentration prediction based on spatial-temporal feature engineering and a combination of new techniques. However, there are still some limitations that should be noted in the application of our framework. The first two limitations were revealed by the fourth set of experiments. The results demonstrated that the effectiveness of the LSTM-DRSL framework cannot act completely at a smaller regional scale. Additionally, the framework performed a little poorer in non-factory-intensive areas. This limitation was also disclosed in the fifth set of experiments. Finally, the ability of our framework to deal with temporal variations, which was tested in the last set of experiments, was worse when predicting daily pollutant concentrations. In other words, the LSTM-DRSL performs better in solving problems which have a small unit of time or long-term time dependency.

In future work, in addition to further research regarding computational costs, the sources of input variables for air pollutant emissions can be further enriched, especially for mobile pollution sources such as motor vehicle exhaust. Another possible direction of future study is applying newly developed techniques for time sequence problems to existing prediction frameworks. Such work is worth the research effort since techniques are continuously being developed and bring new opportunities.