Validation and Comparison of Long-Term Accuracy and Stability of Global Reanalysis and Satellite Retrieval AOD

Abstract

Reanalysis and satellite retrieval are two primary approaches for obtaining large-scale and long-term Aerosol Optical Depth (AOD) datasets. This study evaluates and compares the accuracy, long-term stability, and error characteristics of the MERRA-2, MODIS combined Dark Target and Deep Blue (DT&DB), and VIIRS DB AOD products globally and regionally. The results indicate that the MERRA-2 AOD exhibits the highest accuracy with an expected error (EE, ±0.05 ± 20%) of 83.24% and mean absolute error (MAE) of 0.056, maintaining a stability of 0.010 per decade. However, since the MERRA-2 AOD ceased assimilating observations other than the MODIS AOD in 2014, its accuracy decreased by approximately 5.6% in the EE metric after 2014. The VIIRS Deep Blue (DB) AOD product, with an EE of 79.43% and stability of 0.016 per decade, is slightly less accurate and stable compared to the MERRA-2 AOD. The MODIS DT&DB AOD demonstrates an EE of 76.75% and stability of 0.011 per decade. Regionally, the MERRA-2 AOD performs acceptably in most areas, especially in low-aerosol-loading regions, with an EE > 86% and stability ~0.02 per decade. The VIIRS DB AOD excels in high-aerosol-loading regions, such as the Indian subcontinent, with an EE of 69.14% and a stability of 0.049 per decade. The performance of the MODIS DT&DB AOD falls between that of VIIRS DB and MERRA-2 across most regions. Overall, each product meets the accuracy and stability metrics globally, but users need to select the appropriate product for analysis based on the validation results of the accuracy and stability in different regions.

1. Introduction

Atmospheric aerosols are an important part of the Earth’s climate system. They reduce the solar radiation reaching the Earth’s surface through direct and indirect radiation effects, thus affecting climate change [1,2,3]. Excessive aerosols can cause extreme particle pollution events by changing meteorological phenomena, which has negative effects on the environment and health [4,5]. Atmospheric aerosols have a short life cycle (a few hours to a few weeks, depending on chemical composition and environmental conditions, etc.) [6,7] and are concentrated near emission sources, so they have high temporal and spatial heterogeneity [8]. Satellite retrievals and reanalysis can capture the spatial and temporal heterogeneity of aerosols on global and regional scales [9].

Aerosols can absorb and scatter solar radiation, thus affecting the radiance observed by satellite sensors [10,11]. By quantifying the variations in satellite observation signals through the radiative transfer equation, properties such as Aerosol Optical Depth (AOD) can be retrieved. Based on this principle, the aerosol community has developed a variety of satellite aerosol products. The Moderate Resolution Imaging Spectroradiometer (MODIS) and Visible Infrared Imager Radiometer Suite (VIIRS) aerosol retrieval products are easy to obtain, have a long-term series (2000 to present), and are accurate globally, so they are widely used. Yu et al. [12] used MODIS AOD data to study the aerosol radiation effects in the Qinghai–Tibet Plateau. Hammer et al. [13] combined AOD data retrieved from satellites and atmospheric chemistry models to obtain the global spatiotemporal distribution of PM2.5. Reanalysis aerosol products, such as Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2), are also used to obtain the spatiotemporal distribution of aerosol properties. MERRA-2 uses a radiatively coupled version of the Goddard Chemistry, Aerosol, Radiation, and Transport model (GOCART) to simulate and treat five individual aerosol species: dust, black carbon, organic carbon, sulfate, and sea salt [14,15,16]. MERRA-2 is the first long-term global reanalysis to assimilate space-based observations of aerosols (such as bias-corrected MODIS, Multi-angle Imaging Spectro-Radiometer (MISR), and Advanced Very High-Resolution Radiometer (AVHRR) AOD) and represent their interactions with other physical processes in the climate system. Based on these technologies, MERRA-2 can retrieve aerosol composition and optical properties [17,18].

AODs from satellite retrieval and reanalysis have different advantages. Satellite AOD has higher spatial resolution, while reanalysis AOD has higher temporary resolution and better space–time continuity. the good accuracy of the aerosol product is the basis for the application, and numerous studies have been carried out to validate the operational MODIS, VIIRS, and MERRA-2 AOD. For example, Gueymard et al. [19] reported that the root mean square error (RMSE) of the MERRA-2 AOD is 0.126 compared with global Aerosol Robotic Network (AERONET) observations, and the performance varied widely across Köppen–Geiger climate zones. The global validation results show that the accuracy of the MERRA-2 AOD and MODIS AOD are comparable [18,20]. Comparisons of their validation accuracy in different regions show differences. For example, in the eastern Mediterranean region, the validation metrics of the MERRA-2 AOD, such as correlation coefficient (R), mean absolute error (MAE), and RMSE, are better than those of the MODIS AOD [21]. Comparisons of daily AOD grid data from MODIS, VIIRS, and MERRA-2 in China have shown that MODIS has the highest validation accuracy, while MERRA-2 has the lowest [22]. In Asia, the VIIRS DB AOD demonstrates the best accuracy, outperforming MODIS Deep Blue (DB) and Dark Target (DT) in terms of R and RMSE metrics [23]. Although these studies provide valuable references for accuracy validation and comparison, the results and conclusions vary by region. For example, the product with the best accuracy differs across regions. With the continuous upgrading of satellite retrieval algorithms and changes in the AOD observations assimilated by MERRA-2, there is a pressing need for comprehensive global and regional evaluations and comparisons of satellite-retrieved and reanalysis AOD products.

Some studies have used MODIS, VIIRS, and MERRA-2 to investigate long-term variations in AOD. For example, Hu et al. [24] used MODIS and MERRA-2 datasets to analyze the 20-year AOD change trend in central and eastern China and the Indian subcontinent. The results show that MODIS and MERRA-2 datasets can capture the significant rising trend of AOD in the Indian subcontinent and the decreasing trend of AOD in China. Kuttippurath et al. [25] obtained similar results in the Indian region using MODIS and long-term MERRA-2 datasets. MODIS and MERRA-2 data also show an increasing trend of AOD in the eastern Mediterranean region [26]. Although these AOD products have been used for long-term trend analysis, the long-term stability of their accuracy has rarely been evaluated. Accuracy stability is an important metric of AOD products. Unstable products may cause errors or increase or decrease the AOD trend. The stability and consistency of AOD products also have an impact on subsequent applications (such as PM_2.5 estimation) [11,13,26]. Sayer et al. [27] evaluated the long-term stability of the MODIS and VIIRS DB products and found that they meet the Global Climate Observing System (GCOS) stability metric of 0.02 per decade on a global scale. However, Su et al. [28] found that the accuracy of MODIS and VIIRS in the Asian region does not meet the GCOS stability metric, with a bias stability of 0.027–0.045 per decade. Therefore, more evaluation is needed to illustrate the stability of the commonly used MODIS, VIIRS, and MERRA-2 AOD products, especially the MERRA-2 AOD, whose stability does not seem to have been evaluated. Previous studies have demonstrated regional variations in stability, highlighting the need for regional assessments.

This study focuses on the accuracy, stability, and error analysis of three AOD products of reanalysis (MERRA-2) and satellite retrieval (MODIS DTB and VIIRS DB) globally and regionally. This will provide a critical reference for product selection for long-term scientific research. Section 2 describes the data and methods. Section 3 provides the results and discussion. The conclusion is presented in Section 4.

2. Data and Method

The reanalysis AOD data used in this study were MERRA-2 products, and the satellite AOD data were MODIS DTB and VIIRS DB products, which were evaluated using global long-term AERONET observation data. The indicators evaluated included two categories: accuracy validation and stability assessment, which are carried out on both global and regional scales. Finally, error dependence was analyzed, that is, testing whether the bias of AOD changes with a certain variable to characterize the systematic bias of AOD retrieval. Auxiliary variables used in the error dependence analysis included aerosol properties (loading and type) and surface properties (vegetation index and land cover). The flow chart of this study is shown in Figure 1. The data and methods are elaborated in detail below.

2.1. MERRA-2 Reanalysis Data

MERRA-2 is the common modern satellite era (1980–onward) atmospheric reanalysis product from the NASA Global Modeling and Assimilation Office (GMAO) [29]. It uses a radiatively coupled version of the Goddard Chemistry, Aerosol, Radiation, and Transport model (GOCART) to simulate and treat five individual aerosol species: dust, black carbon (BC), organic carbon (OC), sulfate (SO4), and sea salt [14,15,16]. Compared with the first generation MERRA, the MERRA-2 project broadens the scope of assimilated data to include bias-corrected MODIS AOD data, bias-corrected AOD data from the AVHRR, AOD data from the MISR over bright surfaces, and AODs measured from ground-based AERONET data [16,29,30,31]. At the same time, MERRA-2 also performs meteorological assimilation to generate a 3D grid of aerosol field time series since 1980 [31]. In this study, the MERRA-2 AOD at 550 nm, with a 0.5° × 0.625° spatial resolution and 3-h temporal resolution, was used and downloaded from https://disc.gsfc.nasa.gov, last accessed 1 June 2024.

2.2. Satellite Retrieval Data

The MODIS sensor has 36 bands covering the range from visible to thermal infrared. The AOD products derived based on MODIS observation data include DT, DB, DTB, and MAIAC products [9,32,33,34,35,36,37,38]. Since this study primarily analyzed MODIS level 3 (approximately 1° resolution) products, the MAIAC product was not used. Due to the differences in the cloud mask method and the limitations of retrieval algorithms, there are differences in the spatial coverage of the DT and DB algorithms. In general, the DT algorithm can only retrieve aerosols above the dark surface, while the DB algorithm can retrieve aerosols above both the bright and dark surfaces. The DTB product merges the retrieval results of the DT and DB algorithms, has higher spatial coverage, and its accuracy is comparable to that of DT and DB [39], so the DTB product was used. Compared with Terra-MODIS, the Aqua-MODIS AOD has better radiometric calibration and higher validation accuracy [40,41], so the DTB product of Aqua-MODIS was used. According to previous global validation results, the AOD retrieval accuracy of the VIIRS DB product is better than that of the VIIRS DT and VIIRS NOAA products [42], so the VIIRS DB product was used. In summary, this study used the level 3 MODIS atmosphere daily global DTB aerosol product (MYD08_D3) and VIIRS NASA standard level 3 daily DB aerosol product (AERDB_D3_VIIRS_SNPP) at a 1° × 1° horizontal resolution grid. Note that only the high-quality (or best-estimated) satellite retrievals were used. Both DTB AOD data and DB AOD data are available from https://ladsweb.nascom.nasa.gov, last accessed 1 June 2024.

2.3. Other Auxiliary Data

This study used the following supporting products for error analysis: (1) MODIS MOD13C1 product, providing the Normalized Difference Vegetation Index (NDVI) variable; (2) MODIS MCD12C1 product, which is the yearly land surface cover product. Additionally, the IGBP global vegetation classification scheme was used in this study.

2.4. Ground-Based Observation Data

The AERONET is a ground-based aerosol property observation network with over 1000 sites, which can routinely provide AOD measurements at 440, 675, 870, and 1020 nm spectral channels with low uncertainty (approximately 0.01–0.02), and some instruments provide observations at other wavelengths [43,44]. The AERONET sites provide three levels of products: level 1.0, level 1.5, and level 2.0. The latest level 2.0 data (Version 3.0) were used [45]. To minimize the impact of changes in available sites on long-term performance evaluations, 102 long-term sites were selected for this study (Figure 2). Long-term sites are defined as sites with valid data for more than 15 years and more than 75% of the effective observation duration. Selected sites are divided into 9 sections based on geographical location to allow for regional discussions [27].

2.5. Accuracy Validation Method

Satellite retrievals, reanalysis, and AERONET ground-based observations of AOD differ in their spectral, temporal, and spatial sampling characteristics. AERONET spectral AOD measurements are converted to 550 nm AODs using Equation (1) [44]. The time-matching strategy between satellite retrievals and AERONET observations is to select average AERONET observations within a ±30 min time window of the satellite overpass time [27,46]. Since MERRA-2 provides AOD data at 3 h intervals, and considering that the overpass time for Aqua and VIIRS is 13:30, the average of the MERRA-2 AOD data at 12:00 and 15:00 is matched with the average of the AERONET observations between 13:00 and 14:00 [18,22]. The validation metrics include the bias (Equation (2)), mean absolute error (MAE, Equation (3)), root mean square error (RMSE, Equation (4)), relative mean bias (RMB, Equation (5)), fractional gross error (FGE, Equation (6)), index of agreement (IOA, Equation (7)), expected error (EE, Equation (8)), and Pearson correlation coefficient (R, Equation (9)) [21,47,48]. Bias, RMB, and FGE metrics can be used to evaluate the overestimation or underestimation of AOD retrieval. When bias < 0, RMB < 1, or FGE is −200%–0, the product underestimates AOD; otherwise, the product overestimates AOD. MAE and RMSE represent the overall error of retrieval, and the smaller the value, the better. The EE represents the fraction of the expected error, and the closer the value is to 1, the better. IOA and R characterize the consistency between retrieval and observation, and the closer the value is to 1, the better.

$$\log \left({\mathsf{\tau }}_{\mathsf{\lambda }}\right)={\mathrm{a}}_0+{\mathrm{a}}_1\log \left(\mathsf{\lambda }\right)+{\mathrm{a}}_2\log {\left(\mathsf{\lambda }\right)}^2$$

(1)

where ${\mathsf{\tau }}_{\mathsf{\lambda }}$ is the AOD at wavelength $\mathsf{\lambda }$. ${\mathrm{a}}_0$, ${\mathrm{a}}_1$, and ${\mathrm{a}}_2$ are the fitting coefficients obtained by fitting the AERONET spectral AOD.

$$\mathrm{Bias}={\displaystyle \frac{1}{\mathrm{n}}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\left({\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)$$

(2)

$$\mathrm{MAE}={\displaystyle \frac{1}{\mathrm{n}}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}\left|{\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right|$$

(3)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{\mathrm{n}}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)}^2}$$

(4)

$$\mathrm{RMB}=\overline{{\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i} }}/\overline{{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i} }}$$

(5)

$$\mathrm{FGE}={\displaystyle \frac{2}{\mathrm{n}}}{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\displaystyle \frac{\left({\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)}{\left({\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}+{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)}}\times 100$$

(6)

$$\mathrm{IOA}=1-{\displaystyle \frac{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}-{\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}\right)}^2}{{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{n}}{\left(\left|{\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-\overline{{\mathrm{AOD}}_{\mathrm{Product}}}\right|+\left|{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}-\overline{{\mathrm{AOD}}_{\mathrm{AERONET}}}\right|\right)}^2}}$$

(7)

$$\mathrm{EE}=\pm \left(0.05+0.2\times {\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)$$

(8)

$$\mathrm{R}=\sqrt{1-{\displaystyle \frac{{{\displaystyle \sum }}_1^{\mathrm{n}}{\left({\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-{\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)}^2}{{{\displaystyle \sum }}_1^{\mathrm{n}}\left({\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}-{\overline{\mathrm{AOD}}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}\right)}}}$$

(9)

where the ${\mathrm{AOD}}_{\left(\mathrm{Product}\right)\mathrm{i}}$ is $\mathrm{product}$ AOD, ${\mathrm{AOD}}_{\left(\mathrm{AERONET}\right)\mathrm{i}}$ is AERONET AOD, and n is the number of AODs matched.

2.6. Stability Assessment Method

The definition of stability is the maximum acceptable change in systematic error (uncertainty) over a period, which is required to be less than 0.02 per decade [27]. This study used bias and uncertainty metrics as stability evaluation metrics, where the uncertainty metric refers to the standard deviation of the bias. Since GCOS evaluates stability based on a 10-year scale, we performed two methods on the above metrics: (1) expanding the fitting coefficient by 10 times ($10\times \mathrm{fitting} \mathrm{coefficient}$), and (2) doubling the standard deviation on a 10-year scale $\left({\displaystyle \frac{10}{\mathrm{Number} \left(\mathrm{years}\right)}}\times 2\times \mathrm{standard} \mathrm{deviation}\right)$ [27,28]. These two methods are similar to the “maximum acceptable error” in the definition of stability.

2.7. Error Dependence Analysis Method

Error dependence analysis tests whether the AOD bias of satellite and reanalysis against AERONET measurement systematically depends on a certain variable. In this step, the mean and standard deviation of AOD bias in each aerosol and surface variable bin were analyzed. In this study, the dependence of AOD bias on aerosol properties (loading and type) and surface properties (NDVI and land cover type) was analyzed. Aerosol loading and type were characterized using AERONET AOD and single scattering albedo (SSA). Aerosols are classified into three types based on the thresholds of AOD and SSA: background (AOD < 0.2), scattering-dominated (AOD > 0.2, SSA > 0.92), and absorption-dominated (AOD > 0.2, SSA < 0.92). NDVI data come from the MOD13C1 product. Land cover type data come from the MCD12C1 product, which is reclassified into five types: forest, grassland, cropland, urban land, and arid land.

3. Result and Discussion

3.1. Overall Accuracy Evaluation and Comparison

MERRA-2 (2000–2020), MODIS Aqua DTB (2002–2020), and VIIRS DB (2012–2020) AODs are compared with AERONET measurements at 550 nm. Note that MERRA-2, MODIS, and VIIRS cover different periods, and available sites of AERONET change a lot in these periods. AERONET measurements from long-term sites (see Figure 2) were used to evaluate and compare the performance of all products. This minimizes the impact of AERONET site changes while capturing performance variations. Figure 3 shows the comparison results and

Table 1. Global metrics statistics for MERRA-2, MODIS DTB, and VIIRS DB AOD against AERONET measurements.

Products	N	R	Bias	RMSE	MAE	RMB	FGE (%)	IOA	>EE (%)	=EE (%)	<EE (%)
MERRA-2	128,815	0.826	−0.0009	0.122	0.056	0.994	12.239	0.900	6.23	83.24	10.53
MODIS	81,379	0.856	−0.0002	0.119	0.066	0.999	−4.475	0.924	9.83	76.75	13.42
VIIRS	45,569	0.851	0.0068	0.126	0.062	1.042	1.543	0.919	7.92	79.43	12.64

shows the evaluation metrics. The matchups of the MERRA-2 AOD are 58.29% more than those of the MODIS AOD. VIIRS has 10 fewer years of data records than MODIS, so its matchups are 44.01% less than MODIS. Overall, the MERRA-2 AOD has an RMSE of 0.122, an MAE of 0.056, an EE of 83.24%, an R of 0.826, and an IOA of 0.9, indicating high simulation accuracy. These validation metrics are similar to those of Che et al. (2019), with R of 0.85, MAE of 0.06, RMSE of 0.12, and IOA of 0.94. This shows that the overall validation results of this study are reasonable and credible. In this study, the bias (−0.0009) and RMB (0.994) metrics show that MERRA-2 slightly underestimates the AOD. However, the FGE of 12.239% shows that MERRA-2 slightly overestimates the AOD. The divergence between the RMB and FGE metrics was also found in the studies of Che et al. (2019) and Shaheen et al. (2020). The validation results of Che et al. (2019) have an RMB of 0.97 and an FGE of 9.86%. The validation results of Shaheen et al. (2020) have an RMB of 0.995 and an FGE of 6.61%. In any case, the MERRA-2 AOD bias is very slight. The bias metric of MODIS AOD is close to 0 (−0.0002), the RMB is 0.999, and the FGE is −4.475%. These metrics indicate that MODIS retrievals underestimate the AOD slightly. The R, IOA, and RMSE metrics of MODIS AOD are better than those of the MERRA-2 AOD, but the MAE and EE metrics are weaker than those of the MERRA-2 AOD. Previous studies have also shown that the validation accuracy of the MERRA-2 AOD is comparable to or even better than that of the MODIS AOD regionally and globally [18,21,49]. The bias (0.0068), RMB (1.042), and FGE (1.543%) metrics of the VIIRS AOD show that it slightly overestimates the AOD. The error and accuracy metrics (including RMSE, MAE, and EE) of the VIIRS AOD are worse than those of the MERRA-2 AOD. In general, MERRA-2 shows better validation accuracy. The AODs of the three products all meet the expected accuracy globally and can be used for quantitative scientific research.

The interannual variations of the accuracy of each product at long-term stations are analyzed (see Figure 4). Figure 4a shows that all products have a similar annual mean of the AERONET AOD, indicating the reference of matchups is almost consistent. In contrast, the annual means of the AODs from MERRA-2, MODIS, and VIIRS are more different (see Figure 4b). Figure 4c shows that MERRA-2 has little bias (better than ±0.01) most of the time. MODIS DTB underestimates the AOD before 2010 and slightly overestimates the AOD after 2011. VIIRS DB slightly overestimates the AOD in all available years. Interestingly, bias is on the rise across all products. In terms of the R, RMSE, EE, and MAE metrics, MERRA-2 shows a decreasing trend in accuracy between 2011 and 2014 and is stable in the remaining years. Compared to the average value of metrics before 2014, the MAE of MERRA-2 increases by 13.76–25.91%, RMSE increases by 11.86–47.10% (excluding 2018, which was 0.38%), and R and EE decrease by 1.99–17.15% and 6.69–9.46%. This may be due to the fact that the MISR and AERONET AODs for the Neural Net Retrievals (NNR) assimilation stopped in 2014, and the associated bias is not calibrated well [16,30]. The validation metrics of MODIS and VIIRS fluctuate from year to year but are generally stable.

3.2. Stability Evaluation and Comparison

Apart from accuracy, stability is also a crucial aspect of evaluating product performance. The stability metrics of the three products were calculated according to the stability assessment method in Section 2.5. Figure 5 shows the least squares fit lines for the bias and uncertainty metrics involved in the stability assessment, and

Table 2. Stability metrics summary of bias and uncertainty. Underlined indicates that the metric meets the GCOS requirement (i.e., less than 0.02).

Product (Years)	10 × Fitting Coefficient		2 × Normalized Standard Deviation		Total *
	(Bias and Uncertainty)		(Bias and Uncertainty)		Mean
MERRA-2 (21)	−0.006	0.011	0.006	0.020	0.010
MODIS (19)	−0.012	−0.007	0.009	0.017	0.011
VIIRS (9)	−0.006	0.016	0.012	0.029	0.016

* The absolute values of all values are used in the calculation of the total mean.

summarizes all stability metrics for each product. All metrics for MERRA-2 and MODIS DTB AOD products meet the GCOS stability requirements. The two metrics of stability for the bias of the MODIS DTB AOD, “10 × fitting coefficient” and “2 × normalized standard deviation”, are −0.012 and 0.009, respectively, which are higher than those of MERRA-2 (−0.006 and 0.006). Note that only the magnitude of the value is considered when comparing, ignoring the positive and negative. However, the uncertainty stability metrics for MODIS DTB (−0.007 and 0.017) are smaller than those for MERRA-2 (0.011 and 0.020). The stability of the VIIRS DB AOD is slightly worse, with most metrics exceeding those of the MERRA-2 and MODIS DTB products. Overall, the stability performance of these three products is good, with the total mean stability metrics being less than 0.02 per decade (0.010 per decade for MERRA-2, 0.011 per decade for MODIS, and 0.016 per decade for VIIRS). Sayer et al. [27] evaluated the bias stability of the MODIS and VIIRS DB AODs on a global scale, and their results using the fitting coefficient (0.005–0.01 per decade) were similar to our bias fitting coefficient (0.006–0.012 per decade). However, regional characteristics (AOD loading, surface type, etc.) can affect the stability of the products differently. The stability of DB products (including MODIS and VIIRS) in East Asia does not meet the GCOS requirement due to the high aerosol loadings and complex aerosol types [28]. Therefore, we discuss the stability of each product regionally later.

3.3. Impact of Assimilated Data Changes on Accuracy and Stability

Since MERRA-2 stopped assimilation of the AERONET AOD on 29 October 2014, and the MISR AOD on 30 June 2014, here we divide all data into two parts for validation: before 29 October 2014, and after 29 October 2014, to analyze the changes in product accuracy and stability.

Table 3. The impacts of the assimilated data change on the accuracy of MERRA-2. MODIS and VIIRS are used for comparison.

Product	Timespan	N	R	Bias	RMSE	MAE	RMB	FGE (%)	IOA	<EE (%)	=EE (%)	>EE (%)
MERRA-2	Before	84,021	0.843	−0.003	0.114	0.053	0.983	9.246	0.911	5.95	85.19	8.86
	After	44,794	0.796	0.003	0.135	0.062	1.018	17.852	0.879	6.76	79.57	13.67
MODIS	Before	52,548	0.851	−0.003	0.118	0.067	0.983	−0.068	0.921	10.44	76.63	12.93
	After	28,831	0.865	0.005	0.120	0.064	1.031	−0.002	0.928	8.72	76.96	14.32
VIIRS	Before	14,410	0.826	0.006	0.123	0.065	1.033	0.011	0.906	8.58	77.97	13.45
	After	31,159	0.860	0.007	0.127	0.061	1.047	0.018	0.923	7.62	80.11	12.27

and

Table 4. The impacts of the assimilated data change on the stability of MERRA-2. MODIS and VIIRS are used for comparison. Underlined indicates that the metric meets the GCOS requirement (i.e., less than 0.02).

Product	Timespan	10 × Fitting Coefficient		2 × Normalized Standard Deviation		Total
		(Bias and Uncertainty)		(Bias and Uncertainty)		Mean *
MERRA-2	Before	0.004	0.002	0.007	0.026	0.010
	After	0.016	−0.025	0.016	0.046	0.026
MODIS	Before	0.016	−0.023	0.013	0.030	0.020
	After	0.012	0.045	0.012	0.038	0.027
VIIRS	Before	/	/	/	/	/
	After	0.000	0.049	0.015	0.055	0.030

* The absolute values of all values are used in the calculation of the total mean.

show the validation results from before and after the assimilation data switching. For accuracy, MERRA-2 AOD showed a decline in accuracy over time, with its bias shifting from −0.003 to 0.003, RMSE increasing from 0.114 to 0.135, and MAE rising from 0.053 to 0.062. The relative mean bias (RMB) shifted from 0.983 to 1.018. The R and IOA metrics decreased (R from 0.843 to 0.796 and IOA from 0.911 to 0.879). The FGE metric grew (from 9.246% to 17.852%). These results indicate the reduced accuracy of the MERRA-2 AOD after the assimilation data switching. The MODIS AOD and VIIRS AOD did not change significantly in metrics after the assimilation data switching, maintaining relatively stable performance. The stability of the MERRA-2 AOD was significantly better than that of MODIS and VIIRS before the assimilated data change, with stability assessment metrics below 0.02 except for the “2 × normalized standard deviation” metric of uncertainty (0.026). However, after the assimilated data change, all metrics for MERRA-2 increased by at least two-fold in value, indicating a decrease in stability. The stability of the MODIS DTB AOD showed a slight decline after October 2014. Since there are too few VIIRS data before 2014 (2012–2014), they are not analyzed. Overall, if the study period is before the assimilated data change, then MERRA-2 is suitable for use; if the study period includes or is after 2014, then the user needs to carefully consider whether to select the MERRA-2 AOD.

3.4. Regional Performance

Due to the spatial heterogeneity of aerosol product accuracy, this section analyzes the accuracy and stability of each region. Figure 2 shows the scope and abbreviated meaning of the regions in the following discussion.

Table 5. Regional metrics statistics for MERRA-2, MODIS, and VIIRS product AODs against AERONET ground measurements. Bold font indicates the best performance among all products for each metric and region.

Region	Products	AERONET AOD	Product AOD	N	R	Bias	MAE	RMSE	RMB	FGE (%)	IOA	EE (%)
WNA	MERRA-2	0.077	0.087	24,837	0.751	0.010	0.037	0.101	1.136	30.230	0.844	88.98
	MODIS	0.083	0.081	16,734	0.836	−0.002	0.045	0.107	0.973	−9.800	0.908	84.70
	VIIRS	0.085	0.090	9474	0.844	0.006	0.046	0.148	1.068	1.388	0.886	89.09
ENA	MERRA-2	0.135	0.130	27,299	0.796	−0.005	0.042	0.089	0.960	4.968	0.877	89.38
	MODIS	0.130	0.132	16,149	0.717	0.002	0.054	0.102	1.014	4.723	0.843	81.44
	VIIRS	0.126	0.133	8659	0.788	0.006	0.046	0.094	1.051	9.472	0.881	85.56
SAM	MERRA-2	0.179	0.167	11,753	0.944	−0.012	0.050	0.092	0.932	−1.628	0.966	85.42
	MODIS	0.181	0.142	7554	0.922	−0.039	0.069	0.118	0.783	−31.098	0.954	72.89
	VIIRS	0.138	0.110	5405	0.886	−0.028	0.061	0.088	0.794	−36.305	0.932	70.90
EUR	MERRA-2	0.153	0.148	25,772	0.800	−0.005	0.043	0.073	0.967	0.923	0.888	87.88
	MODIS	0.148	0.147	18,329	0.801	−0.002	0.051	0.075	0.988	−5.678	0.892	82.21
	VIIRS	0.136	0.143	9635	0.804	0.006	0.044	0.064	1.045	3.963	0.893	85.88
IND	MERRA-2	0.493	0.397	4258	0.667	−0.095	0.155	0.272	0.806	−18.429	0.757	67.10
	MODIS	0.544	0.485	3033	0.800	−0.059	0.150	0.228	0.892	−13.482	0.883	64.59
	VIIRS	0.570	0.530	2126	0.822	−0.040	0.140	0.222	0.930	−0.375	0.869	69.14
AFC	MERRA-2	0.223	0.253	15,779	0.917	0.030	0.070	0.115	1.134	33.395	0.954	71.73
	MODIS	0.277	0.341	8644	0.885	0.065	0.114	0.153	1.234	35.325	0.926	52.15
	VIIRS	0.273	0.349	3958	0.909	0.075	0.115	0.162	1.275	40.266	0.939	50.73
SEA	MERRA-2	0.218	0.180	3582	0.795	−0.038	0.071	0.193	0.826	−6.953	0.785	83.92
	MODIS	0.185	0.195	1253	0.784	0.010	0.093	0.165	1.055	−16.263	0.850	70.15
	VIIRS	0.216	0.267	577	0.652	0.051	0.155	0.245	1.235	−14.484	0.742	48.53
ESA	MERRA-2	0.242	0.245	10,644	0.599	0.003	0.121	0.222	1.013	11.706	0.753	62.20
	MODIS	0.201	0.211	6260	0.774	0.009	0.091	0.159	1.046	−0.076	0.875	68.79
	VIIRS	0.193	0.209	4152	0.735	0.017	0.087	0.160	1.086	4.053	0.854	74.13
OCE	MERRA-2	0.086	0.104	4593	0.771	0.018	0.039	0.077	1.204	28.843	0.854	86.07
	MODIS	0.085	0.048	3423	0.773	−0.038	0.045	0.092	0.559	−52.891	0.846	84.84
	VIIRS	0.083	0.072	1583	0.739	−0.011	0.036	0.056	0.864	−21.384	0.847	88.69

and

Table 6. Stability metrics summary of bias and uncertainty for each region. Bold font indicates the best performance among all products for each metric and region. Underlined indicates that the metric meets the GCOS requirement (less than 0.02 per decade).

Region	Product	10 × Fitting Coefficient (Bias and Uncertainty)		2 × Normalized Standard Deviation (Bias and Uncertainty)		Total Mean *
WNA	MERRA-2	−0.012	0.011	0.009	0.039	0.018
	MODIS	−0.007	0.030	0.012	0.036	0.021
	VIIRS	0.008	0.111	0.020	0.152	0.073
ENA	MERRA-2	−0.015	−0.031	0.011	0.029	0.021
	MODIS	−0.031	−0.045	0.02	0.044	0.035
	VIIRS	−0.038	−0.046	0.024	0.071	0.045
SAM	MERRA-2	−0.013	−0.031	0.019	0.046	0.027
	MODIS	−0.027	−0.059	0.027	0.061	0.044
	VIIRS	−0.01	0.017	0.036	0.031	0.023
EUR	MERRA-2	−0.005	−0.017	0.009	0.023	0.013
	MODIS	−0.014	−0.016	0.013	0.019	0.015
	VIIRS	−0.018	−0.003	0.017	0.006	0.011
IND	MERRA-2	0.065	0.149	0.053	0.114	0.095
	MODIS	−0.039	0.078	0.032	0.064	0.053
	VIIRS	−0.036	−0.035	0.061	0.062	0.049
AFC	MERRA-2	−0.014	0.018	0.012	0.021	0.016
	MODIS	0.015	0.008	0.026	0.021	0.018
	VIIRS	0.040	−0.017	0.048	0.056	0.04
SEA	MERRA-2	0.033	0.063	0.04	0.098	0.058
	MODIS	−0.050	0.055	0.121	0.093	0.08
	VIIRS	0.477	−0.302	0.33	0.228	0.334
ESA	MERRA-2	−0.016	−0.004	0.030	0.044	0.024
	MODIS	−0.042	−0.034	0.034	0.052	0.04
	VIIRS	−0.001	−0.030	0.016	0.049	0.024
OCE	MERRA-2	−0.002	−0.005	0.012	0.037	0.014
	MODIS	0.006	−0.039	0.025	0.047	0.029
	VIIRS	−0.037	0.027	0.036	0.027	0.032

* The absolute values of all values are used in the calculation of the total mean.

show the statistical results of accuracy and stability metrics. The validation of this study in Europe (EUR) shows that the MERRA-2 AOD R is 0.800, MAE is 0.043, RMSE is 0.073, and RMB is 0.967. This is similar to the previous validation results of MERRA-2 in Europe, with an R of 0.79, MAE of 0.04, RMSE of 0.07, and RMB of 0.95 [47]. The MERRA-2 AOD validation results for the eastern Mediterranean region, which is close to the European region, showed an R of 0.759, MAE of 0.067, RMSE of 0.104, and RMB of 0.862 [21]. In the Middle East region, previous MERRA-2 AOD validation results showed an R of 0.95, MAE of 0.04, RMSE of 0.07, and RMB of 1.02 [47]. The validation results of the MERRA-2 AOD in Iran, a country in the Middle East, showed that R was 0.858, MAE was 0.067, RMSE was 0.118, and RMB was 1.007 [49]. The MERRA-2 AOD validation results in the AFC region of this study are comparable to those of these studies, with an R of 0.917, MAE of 0.070, RMSE of 0.115, and RMB of 1.134. These validation comparisons show that the validation results of this study are credible.

The aerosol loading in WNA, ENA, EUR, and OCE is relatively low (the mean AERONET AOD is approximately 0.08–0.14), but the aerosol sources are complex, including biomass-burning aerosols caused by forest fires caused by occasional dry and hot weather [50,51], anthropogenic aerosols generated by large developed cities and industries, and sea salt aerosols formed by the evaporation of high concentrations of seawater along long coastlines [52], and dust aerosols in extremely arid areas. The accuracy of the MERRA-2 AOD in these regions is good, with an EE above 86% and an MAE of approximately 0.045. The performance of the VIIRS AOD is similar to that of the MERRA-2 AOD. The MODIS AOD shows relatively poor validation performance. The EE metric of the VIIRS AOD is above 85% (approximately 3–5% higher than the MODIS AOD). As mentioned above, the bias and RMB metrics have high consistency in characterizing overestimation and underestimation, but they may disagree with the FGE metric. Therefore, we use the RMB metric here to analyze the overvaluation and undervaluation in different regions. MERRA-2 overestimates the AOD in the WNA (RMB of 1.136) and OCE (RMB of 1.204) regions and underestimates the AOD in the ENA (RMB of 0.960) and EUR (RMB of 0.967) regions. MODIS underestimates the AOD in WNA (RMB of 0.973), EUR (RMB of 0.988), and OCE (RMB of 0.559), and overestimates the AOD in ENA (RMB of 1.014). VIIRS overestimates the AOD in WNA (RMB of 1.068), ENA (RMB of 1.051), and EUR (RMB of 1.045) regions, and underestimates the AOD in OCE (RMB of 0.864). The stability of the three products is slightly different. The bias stability of MERRA-2 meets the GCOS stability requirements, with the “10 × fitting coefficient” metric within −0.015–−0.002 and the “2 × normalized standard deviation” metric within 0.009–0.012. In comparison, the two metrics of MODIS bias stability are within −0.031–0.006 and 0.012–0.025, and those of VIIRS are −0.038–0.008 and 0.017–0.036. The uncertainty stability of the MERRA-2 AOD is also impressive: the “10 × fitting coefficient” metric in the 3/4 regions performs better than the other two products, and the total mean stability metric in the WNA, EUR, and OCE regions is less than 0.02. In general, the MERRA-2 AOD performs well in areas with low aerosol loading, especially stability.

The aerosol loadings of the SAM and SEA regions are at moderate levels. There are two main types of sites in the SAM region: urban sites (mainly coastal) and sites located in or affected by the Amazon Basin. Sites in the SEA region are mainly located near tropical rainforests in Southeast Asia. The climate here is humid, cloudy, and rainy with lush vegetation. The number of matchups for the MERRA-2 AOD in the SEA region far exceeds that for other products, being approximately three times that for the MODIS AOD and seven times that for the VIIRS AOD. Furthermore, the accuracy of the MERRA-2 AOD in the SEA region is better than the other two products, with an EE 83.92% and R 0.795. The VIIRS DB AOD performs relatively poorly (EE = 48.53%, R = 0.652), likely due to the high vegetation cover affecting the accuracy of the retrieval algorithm [53]. In the SAM region, all three products show an underestimation of the AOD, with RMB ranging from 0.783 to 0.932. Satellite products underestimate the AOD more severely than MERRA-2. In the SEA region, the MERRA-2 AOD shows an underestimation, while the satellite AOD shows an overestimation. The MODIS AOD has the best RMB metric (1.055). Similar to the accuracy, the stability of the three products is poor in the SEA region. The total mean stability for the MERRA-2, MODIS DTB, and VIIRS DB AODs is 0.058, 0.08, and 0.334, respectively, which are significantly worse than the GCOS requirements. Although the stability of the three products in the SAM region has not reached the GCOS target (0.023–0.044 per decade), it is better than that in the SEA region.

The ESA, AFC, and IND regions have high aerosol loading (AERONET AODs are approximately 0.2–0.5). The ESA region is densely populated with more than 20% of the world’s population. Industrial emissions and vehicle exhaust produce large amounts of anthropogenic aerosols in this region. The AFC region has several AERONET sites in the desert of North Africa and the Middle East peninsula. These sites can monitor a large number of dust aerosol transport events generated annually. IND has both a dense population and desert regions (such as the Thar Desert). Combined with the monsoon winds and the blockage of the Himalayas [25], the regional average AOD reaches more than twice that of the AFC. The effect of high aerosol loading on the accuracy degradation is significant. In the IND region, the VIIRS AOD has the best accuracy, with an R of 0.822, bias of −0.040, RMB of 0.930, FGE of −0.375%, IOA of 0.869, and EE of 69.14%, while the MERRA-2 AOD has the lowest accuracy. In comparison, MERRA-2 performs relatively well in the AFC region, with an EE of 71.73%, FGE of 33.395%, R of 0.917, IOA of 0.954, and MAE of 0.07, while the performances of the MODIS AOD and VIIRS AOD are poor, with an EE of approximately 51% and MAE of approximately 0.115. In the IND area, all three products underestimated the AOD, with RMB ranging from 0.806 to 0.930. The MERRA-2 AOD underestimates the most, with an RMB of 0.806. In the ESA area, the three products are slightly overestimated, with RMB ranging from 1.013 to 1.086. In the AFC area, the three products also overestimate the AOD, with RMB of 1.134–1.275. For stability, the MERRA-2 and MODIS AODs have good stability in the AFC region. The bias stability metrics of MERRA-2 are less than 0.02, and the total stability is 0.016, while the MODIS AOD is 0.018. The VIIRS AOD has the best bias stability metric in ESA and uncertainty stability metric in IND, and its overall stability is better than or comparable to that of MERRA-2 products. In general, VIIRS DB performs well in high-aerosol-loading areas, both in terms of accuracy and stability.

3.5. Error Dependence Analysis

As mentioned above, error dependence analysis is used to determine whether the bias of AOD products is systematically dependent on certain variables. We analyzed the bias of AOD in terms of two aspects: aerosol properties and surface properties.

3.5.1. Aerosol Properties

The absorption properties of aerosols are one of the main factors affecting the accuracy of aerosol retrieval. As SSA is a parameter characterizing aerosol absorption characteristics, aerosol types are divided by different AOD and SSA: background (AOD < 0.2), scattering-dominated (AOD > 0.2, SSA > 0.92), and absorption-dominated (AOD > 0.2, SSA < 0.92) [54]. Figure 6 shows the overall bias distribution for different aerosol types. Although the AOD bias is small at different AOD loadings, all products have an overestimation of the AOD at low aerosol loadings and an underestimation of the AOD at high aerosol loadings. When the AOD is equal to 2, the MERRA-2 AOD has a large negative bias of approximately −0.1, the MODIS DTB AOD has a small negative bias of approximately −0.05, and the bias of the VIIRS DB AOD is almost 0. This agrees with the regional validation results in the IND region. In terms of aerosol type, three aerosol type lines in Figure 7a all show similar trends to the overall line in Figure 6a. This indicates that the MERRA-2 AOD bias does not depend on aerosol type. However, MODIS DTB and VIIRS DB significantly underestimate the absorption-dominant AOD and have a smaller bias for the scattering-dominant AOD.

3.5.2. Surface Properties

To analyze the influence of surface type on AOD errors, we defined five common surface types: forest, grassland, cropland, urban land, and arid land [28]. Figure 8 shows the product bias of different land cover types. All products show similar AOD bias variations: in arid land, AOD bias is positive. In the rest of the four land types, the AOD bias is small. The standard deviations for forest, grassland, and urban are larger than those for cropland and arid land. In addition, we discussed the effect of vegetation coverage on the error using NDVI parameters. Figure 9 shows the distribution of AOD bias at different NDVIs. The bias trends are almost the same for all products: when NDVI < 0.1, the AOD bias is positive; when 0.1 < NDVI < 0.4, the AOD bias is negative; when 0.4 < NDVI < 0.6, the AOD bias is small. However, when NDVI > 0.6, the AOD bias of MERRA-2 and MODIS DTB is small and VIIRS DB has a slight positive bias.

4. Conclusions

The long-term data provided by reanalysis and satellite retrieval can be used to study large-scale aerosol spatiotemporal distribution patterns and changing trends. The accuracy and stability of their retrieval need to be validated and analyzed to ensure accuracy in long-term time series studies regionally and globally. In this study, the MERRA-2 AOD was selected to represent the reanalysis product, and the MODIS DTB AOD and VIIRS DB AOD were chosen to represent the satellite retrieval product. AOD observations from 131 long-term AERONET sites around the world were used for accuracy validation and stability assessment on the global and regional scales. The mean bias, MAE, RMSE, R, and EE metrics were used to characterize accuracy, while the 10-year changes in bias and uncertainty metrics were used to evaluate stability. Finally, the dependence of retrieval bias was analyzed in terms of two aspects: aerosol properties and surface properties.

The results show that the MERRA-2 AOD has the highest accuracy and stability, with an EE of 83.24%, MAE of 0.053, and stability of 0.010 per decade. However, its accuracy decreased significantly (approximately 5.6% in EE metric) in 2014 due to the variation in the assimilation of AOD observations. Therefore, this needs to be considered when using MERRA-2 AOD products for long-term analysis including 2014. The performance of VIIRS DB (EE = 79.43%, stability = 0.016 per decade) is slightly worse than that of MERRA-2. MODIS DTB has a worse but acceptable performance, with an EE of 76.75% and stability of 0.011 per decade. Regional validation results show that MERRA-2 performs best in low-aerosol-loading regions, with an EE exceeding 86% and stability close to 0.02 per decade. VIIRS performs best in the IND region (high-aerosol-loading region), with an EE of 69.14% and stability of 0.049 per decade, but performs poorly in the SEA region (high-vegetation-coverage region), with an EE of 48.53% and stability of 0.334 per decade. The performance of MODIS DTB products in various regions is not outstanding, but we found that the accuracy of MODIS DTB in the SAM and SEA regions with high vegetation coverage is more similar to MERRA-2, and closer to VIIRS DB in the IND, AFC, and ESA regions with high aerosol loading. MODIS DTB products are not as stable as the other two products in various regions. The results of the error characteristic analysis show that MERRA-2 AOD bias depends on the aerosol loading, while MODIS and VIIRS bias depends on the aerosol type. Therefore, each product meets the accuracy and stability metrics on a global scale, but users need to select the appropriate product for analysis based on the validation results of accuracy and stability on the regional scale.

Our study shows that the AOD retrieval of the MERRA-2 reanalysis aerosol product is reliable and is comparable to or even better than the satellite AOD retrieval, which is consistent with previous studies [18]. Note that MERRA-2 assimilated AERONET observations before 2014, so comparing only with AERONET measurements may overestimate its accuracy, and the results also show that its accuracy decreased in 2014. More independent measurement validation is necessary in the future (e.g., [55]).