Non-Parametric and Robust Sensitivity Analysis of the Weather Research and Forecast (WRF) Model in the Tropical Andes Region

Abstract

With the aim of understanding the impact of air pollution on human health and ecosystems in the tropical Andes region (TAR), we aim to couple the Weather Research and Forecasting Model (WRF) with the chemical transport models (CTM) Long-Term Ozone Simulation and European Operational Smog (LOTOS–EUROS), at high and regional resolutions, with and without assimilation. The factors set for WRF, are based on the optimized estimates of climate and weather in cities and urban heat islands in the TAR region. It is well known in the weather research and forecasting field, that the uncertainty of non-linear models is a major issue, thus making a sensitivity analysis essential. Consequently, this paper seeks to quantify the performance of the WRF model in the presence of disturbances to the initial conditions (IC), for an arbitrary set of state-space variables (pressure and temperature), simulating a disruption in the inputs of the model. To this aim, we considered three distributions over the error term: a normal standard distribution, a normal distribution, and an exponential distribution. We analyze the sensitivity of the outputs of the WRF model by employing non-parametric and robust statistical techniques, such as kernel distribution estimates, rank tests, and bootstrap. The results show that the WRF model is sensitive in time, space, and vertical levels to changes in the IC. Finally, we demonstrate that the error distribution of the output differs from the error distribution induced over the input data, especially for Gaussian distributions.

1. Introduction

The Weather Research and Forecasting Model (WRF), is an important operational tool in weather, meteorology, and air quality research, especially in forecasting [1]. Furthermore, it is employed to provide information to other chemistry transport models (CTMs), such as Long Term Ozone Simulation–European Operational Smog (LOTOS–EUROS) [2,3,4], WRF with Chemistry (WRF-Chem), Community Multiscale Air Quality (CMAQ) [5,6,7,8,9], and FLEXible PARTicle dispersion (FLEXPART) [10]. However, the fields provided by WRF are uncertain, partly due to the model’s sensitivity to initial and boundary conditions (IC/BC) [11,12,13,14,15,16,17,18], terrain, domain resolution [19,20,21], meteorological fields [11,14,15,18,22,23,24,25], planetary boundary layer (PBL) schemes [26,27,28,29], micro-physics [30], land-surface models (LSM) [31], radiation schemes, and radiation options [32], when put together with other models such as [8,10,33,34,35], and dynamical downscaling [36,37,38].

Regarding the sensitivity analysis, refs. [39,40] evaluated the WRF model in the Iberian Peninsula and found a poor representation of variables such as temperature, wind, and humidity fields at the surface, because of the terrain. The WRF model offers numerous opportunities for parametrization, allowing for customization and refinement of its performance. As demonstrated by [41]’s research, the sensitivity of WRF-Chem’s predictions for the PBL and LSM, highlights the importance of carefully selecting parameters based on the variable of interest, location, and meteorological processes in question. This research suggests that the behavior of the WRF model is not fixed, but can be modified depending on the specific parameters chosen. It underscores the need for a thorough understanding of the underlying physics and the ability to accurately represent the local meteorological conditions.

For instance, the PBL scheme can lead to overestimation problems in the near-surface temperature and wind speed, or can improve the boundary layer’s thermal structure [7]. In Portugal, ref. [14] realized a wind simulation using the WRF model and found evidence supporting the influence of the local terrain and simulation domain resolution. For instance, higher terrain complexity was associated with underestimating the wind speed. Moreover, low wind speeds increased the error between the simulated and observed data.

As stated, the model’s input data quality may limit or improve its performance. For example, ref. [24] compared the higher-resolution Global Forecast System (GFS), with a multi-scale model for data that covers the spring of 2012, using Binary Universal Form for the Representation of meteorological data (PREPBUFR), observations from the National Centers for Environmental Prediction (NCEP), and lateral BC from the GFS. It showed that the GFS forecast outperformed the WRF (local ensemble transform Kalman filter (LETKF): LETKF-multi and LETKF-single) estimates in the different scenarios at a coarser resolution (27 km vs. 40 km). Otherwise, the research of [14,18] in the northern Iberian Peninsula, concluded that the WRF model is sensitive to IC and physical scheme changes.

In Colombia, ref. [37] showed, in a sensitivity study on the Cauca River, that the representation by ERA-Interim of the precipitation in the Colombian Andes, is poor. In contrast, a downscaling using WRF, performed better. Conversely, ref. [9] presented a sensitivity analysis to PBL schemes in the Aburrá Valley, using a sub-kilometer representation. With the configuration used, they found that WRF represented the surface temperature and wind direction well, but overestimated the wind speed, especially in the inner domain. Ref. [42] applied a WRF model over Bogota, Colombia, and [28], who developed a sensitivity analysis study over the northern region of Colombia for PBL schemes in WRF, found that the selection of the PBL schemes is key to replicating pollution patterns in regional and local weather conditions accurately.

As highlighted by [43,44], a sensitivity analysis of a model can be developed using an ensemble (multi-realization of the model), where random perturbation of IC, using a probability density function (PDF), is used. However, most sensitivity analysis research in the WRF model, focuses on the same aspects: modifying the parametrizations of IC/BC [10,14,15,18,32,45,46], or other parameters.

From an ensemble perspective, ref. [47] realized a sensitivity analysis of WRF-Chem in China, for particulate matter (PM) below 2 μm (PM_2.5). They modified the IC/BC, finding that the ensemble better captured the state variable’s behavior.

In [44], the Bred Growing Mode (BGM) method is mentioned as a technique to generate disturbances, where the perturbation is defined as the differences between two previous forecasts. Other techniques that include uncertainty are, the conditional non-linear optimal perturbation, singular vector, ensemble transform with rescaling, perturbations in BC, including land surface and topography, or ensemble Kalman filter (EnKF) [44,48]. The latter was used by [25], in a variant ensemble adjustment Kalman filter (EAKF), to evaluate the sensitivity of WRF to IC on a dry-line convection initiation. On the other hand, a sensitivity-based study on data assimilation was realized by [49]. In that sense, ref. [48] developed an ensemble-based WRF sensitivity analysis on the relationship between the errors in the forecast and the IC, using EnKF. As a result, they found places where the model was more sensitive. Similar research was performed by [50] on the Global Forecast System (GFS), analyzing the sensitivity to new observations, changes in IC, the data assimilation technique, and stochastic parametrizations. The authors found that the results are sensitive to observations, but more sensitive to the assimilation method and parametrizations used in these techniques. Nonetheless, an adjoint method for sensitivity analysis was used by [51]. The results indicated the impact of the observations in an assimilation technique on the Navy Operational Global Atmospheric Prediction System (NOGAPS) model.

The tropical Andes region (TAR) is a very complex terrain, with global interactions, which generate many changes in the weather and meteorology; hence it is not surprising that it is a challenge to find a suitable representation. Regarding human and ecosystem health, chemistry transport models are relevant tools for decision-making and assessment, but their performance is limited by the representation of the regional and local weather conditions; therefore, improving the performance of numerical models like WRF is crucial [28], as was evidenced in research in the Aburrá Valley by [52,53,54]. The Aburrá Valley is a metropolitan area, composed of ten adjacent cities located in the department of Antioquia, Colombia, a tropical country with erratic weather. Thus, it is necessary and a challenge to improve the capabilities of the WRF model.

Knowing the WRF model is sensitive to parametrizations of the micro-physics or PBL, sources of IC/BC, topography, nested domain, and the importance of accuracy in the meteorological field for CTMs [28,42,55], this paper aims to assess, from an ensemble perspective, how the results produced by the model are impacted when there are slight changes in the IC. Therefore, it explains how variations in the model’s IC change the evolution of the results [40,56].

The methodology follows random perturbation. As a result, different probability distributions are considered for the error term: two normal distributions and one exponential distribution. We analyzed the nature of the output’s PDF and its changes following different kinds of disruption. The aforementioned, allows for laying the foundations to enhance the model performance, by using data assimilation for uncertainty reduction, and improving air pollution estimates.

The main contributions of this paper are threefold. First, we carried out a sensitivity analysis on the IC of the WRF model, using ensembles from three statistical distributions. Second, we introduced non-parametric statistics to the WRF literature, an innovative approach, as far as we knew at the time of carrying out this study. Finally, from an understanding of the uncertainty in the output of WRF via an ensemble, we opened a path to improve the model representation of variables important for CTMs such as LOTOS–EUROS, CMAQ, and WRF-Chem.

The rest of this paper is organized as follows: Section 2 introduces the model and experimental setup for the uncertainty analysis, while defining the Monte Carlo simulations generated by the stochastic processes on the model for disturbing the IC/BC. Section 3 provides the main results from a qualitative perspective and expands the results obtained via non-parametric techniques. Section 4 concludes the paper.

2. Materials and Methods

Through the WRF model, it is possible to understand the behavior of a group of meteorological variables by using simulations, where the model parameters are configured in concordance with the domain and research interest [6,7,14,23,24,41]. By definition, this is a deterministic model, that tries to describe the atmosphere using a set of partial and ordinary differential equations [1]. Nevertheless, this representation is a “simplified form’’ of the dynamic nature of a determined domain. This study intends to understand the model’s behavior when the information received is altered, particularly the IC. Ultimately, our global interest in the TAR region, is to provide scientific evidence for decision-making, by referencing the results in the Aburrá Valley. This valley traces the course of the Medellin River along 60 km of a deep mountain canyon, which ranges in width between 3 and 10 km, characterized by its narrowness, high population density, and air pollution (especially PM₁₀ and PM2_.5 [55]). Due to the terrain complexities, it is necessary to address the appropriate configuration rigorously and reduce the uncertainty in models, not only from the meteorological point of view but also from the chemistry and air quality.

2.1. Model Setup

The experiments were developed using WRF model version 3.7, from 25 February 00:00:00 to 28 February 00:00:00, 2019. These dates correspond to a period of meteorologic transition from dry to wet in Colombia, which creates atmospheric conditions favorable to accumulate contaminants in the Aburrá Valley and reduce the air quality. The results obtained here can be compared with experiments developed using CTMs where the WRF model’s output was used. This date range also encourages the WRF model’s performance with the parametrization selected for the experiment, correspondent with the research developed by [37] around the Cauca River, a zone with similar characteristics to the Aburrá Valley.

The simulations were performed in four nested domains, as the Medellin Air Quality Initiative (MAUI) suggests [57], as shown in Figure 1. This configuration includes the totality of the Colombian territory, including its offshore territories. Domain 01 (D01) includes Sint Maarten, Sint Eustatius, and Saba; it is composed of a $90\times 93$ grid and covers the latitude range $(-8.86401,19.0911)$ and longitude range $(-86.6947,-59.2753)$. Domain 02 (D02) is composed of a $193\times 193$ grid; it includes the totality of the Colombian territory and covers the latitude range $(-4.94672,14.7199)$ and longitude range $(-84.929,-65.0916)$. Domain 03 (D03) has a $130\times 157$ grid, and includes the entirety of the Antioquia and Choco departments and the capital cities Bogota D.C and Manizales; it corresponds to the latitude range $(3.7342,9.0649)$ and longitude range $(-78.1088,-73.6774)$. Domain 04 (D04) includes the regions of Belmira, Valle de San Nicolas, and Biosuroeste, including the Aburrá Valley, its surroundings, and the municipalities neighboring the valley, and the moors in Antioquia. D04 has a $130\times 169$ grid and covers the latitude range $(5.3792,7.2945)$ and longitude range $(-76.4586,-74.9814)$. Finally, the domains have a 3:1 grid ratio; that is 1.1 (D04), 3.3 (D03), 10 (D02), and 30 (D01) km, respectively. The surface topography data have 32 Eta vertical levels. The upper boundary is set at 50 hPa and was obtained from the United States Geological Survey (USGS) database, with 2 min resolution for D01 and 30 sec resolution for D02, D03, and D04. The land-use categories are based on the Moderate Resolution Imaging Spectroradiometer (MODIS) land-cover classification of the International Geosphere–Biosphere Program, and modified by the Noah land surface model [37]. The IC/BC for the simulations were obtained from 0.25 deg× 0.25 deg grids prepared and provided operationally every six hours by GFS, that use observational data from different sources. The WRF model’s physical parametrization for this study, was based on [9,37,58]. The micro-physics correspond to the WRF single moment 6-class, a process suitable for high-resolution simulations. The short-wave and long-wave radiation corresponds with the National Center for Atmospheric Research (NCAR) Community Atmosphere Model. The LSM corresponds to the thermal diffusion scheme. For the PBL, the scheme corresponds to the MYJ (Mellor–Yamada–Janjić). The KF (new Eta) scheme is used for the cumulus option in D01 and D02; this is a deep and shallow sub-grid scheme, using a mass flux approach, with downdrafts and CAPE removal time scale.

The Eta surface (Janjić Eta) layer is used. The Eta scheme is based on Monin–Obukhov similarity theory [59,60]. The PBL and surface layer schemes are chosen based on previous research, that evidenced their appropriate performance in our region. For example, ref. [61] researched the spatiotemporal variability in the atmospheric boundary layer in Aburrá Valley. They assessed the ability of the WRF model to represent the data observed with ceilometers, radar wind profilers, and microwave radiometers. Ref. [61] found that, even though Aburrá Valley has a complex topography, the model represented the diurnal and annual cycle of the atmospheric boundary layer.

Additionally, ref. [37] used the WRF model, and showed that the model adequately represented the precipitation in the region after a downscaling for different phases of El Niño–Southern Oscillation. Their research looked at the representation of the precipitation in the central Colombian region, a mountainous area. They employed MYJ to represent the PBL. Likewise, ref. [28] developed a sensitivity analysis of the PBL in northern Colombia, a terrain with moderately complex topography, a sea breeze, and a mix of flat terrain and dry season. According to the results, the YSU (Yonsei University) scheme generally showed the best performance for MYJ and MYNN (Mellor–Yamada–Nakanishi–Niino), but they noted that it has the largest sources of error, when determining the behavior of the surface layer during stable conditions. Representing the atmospheric boundary layer for numerical weather prediction models is challenging, especially in mountainous terrain such as Aburrá Valley. On the other hand, selecting parametrizations in a model like WRF, is challenging for practitioners and researchers. Therefore, we use previous results in our region to select the model parametrizations. A sensitivity analysis to select the appropriate configuration is out of the scope of the research presented.

We utilized the results from the simulation of the inner domain D04, corresponding to the variable temperature and pressure, for the 34 levels (height) and three locations inside the Aburrá Valley, over 72 h. The locations were chosen to match with the positions of three stations from the SIATA (Sistema de Alerta Temprana del Valle de Aburrá) network. The SIATA network is a grid of sensors that provides measurements for different meteorological and air quality variables over the Aburrá Valley region. Among the variables monitored by SIATA are O₃, temperature, pressure, wind velocity, SO₂, PM₁₀, and PM_2.5. The locations are: AMVA, with coordinates $(\mathrm{lat},\mathrm{lon})=(6.24215,-75.55064)$; I.E Manuel Jose Caicedo-Barbosa (Barbosa), located at $(\mathrm{lat},\mathrm{lon})=(6.43498,-75.32722)$; and La Ye, with location $(\mathrm{lat},\mathrm{lon})=(6.18254,-75.55064)$, as shown in Figure 1. These points are selected to study the model’s performance in the same domain but with different topographic conditions. La Ye station is near the valley’s west hillside; AMVA is located in the center, the closest to the Medellin River, corresponding to the region with the highest population. However, Barbosa is located in the northeast, flat part of the valley, a rural area with a lower population density. The results are mainly based on comparing unperturbed and perturbed model output, to analyze WRF’s behavior when IC are perturbed under different kinds of errors. The exact configuration outlined in Section 2.1 was used for every model simulation, which will be elaborated on later.

2.2. Changes to Initial Conditions

The IC/BC have been identified as essential factors in the accuracy of the model results, because they are a significant source of uncertainty [14,15,16,18].

The uncertainty must be quantified to estimate the variables of interest adequately. A way to do this, is by performing multiple perturbations to the IC and initializing the model [62]. Consequently, we executed the WRF model, using the parametrizations described, 301 times: one without noise or perturbation and 300 with it, 100 runs for each of the three kinds of errors described later. With the model output, we establish the distribution or posterior distribution functions for the model’s output. Although all the model variables can be analyzed, we focused on temperature and pressure, for simplicity. The former was the perturbed variable, and the latter was implemented, to assess the impact of the disturbance in a correlated variable.

We aimed for a mathematical characterization of the WRF model uncertainty, while also discovering a suitable configuration for the TAR region in the Aburrá Valley. This configuration can be further tested by applying methodological results to other cities and specific locations in the territory.

To understand the procedure followed to perturb the IC in WRF, we should note that, for its execution, WRF followed Figure 2. The procedure can be split into two steps, the WRF Pre-processing System (WPS) and Real-WRF in the Advanced Research WRF (ARW) system [60]. In the WPS process, the static (geographic) and dynamic (IC/BC) information is revised and organized for the second part in real-WRF. The information is disposed of in the last part of the WPS, the metgrid process, because it arrives at the last part.

The IC in the metgrid files 2019-25-02_00:00:00 from WPS were perturbed for the present study. The literature suggests that the disturbance should be randomly generated, based on some statistical error or using support operational analyses, such as the NCEP and the European Centre for Medium-Range Weather Forecasts (ECMWF) [44] to estimate the parameters of the distribution considered for the error. However, here, we considered two normal distributions and one exponential distribution.

The purpose of using the three types of perturbation, is to assess how the WRF model behaves when perturbed in various scenarios, for a single variable. The first two cases involve perturbations with equal genesis and form but with different parameters, to represent different magnitudes of error, enabling an assessment of the model’s performance. The third case involves a perturbation with different origins in form, allowing a comparison of results and understanding the model’s behavior in such a scenario. An exponential distribution is used in the third case, as it represents the failure rate of a system [63] and events of low occurrence [64] in statistical terms, despite violating a common assumption in data assimilation of Gaussian noise. However, non-parametric statistics were used to address this issue.

It is assumed here, that the mathematical representations given by the ARW can be written in a discrete form:

$$z_k=\mathcal{M}\left(z_{k-1},\theta ,t_k,ϵ\right).$$

(1)

where $\mathcal{M}:{\mathbb{R}}^n\to {\mathbb{R}}^n$ represents the model. $z\in {\mathbb{R}}^n$ are the variables of state, $\theta$ are the parameters, $t_k$ is the time, and $ϵ$ is the error distribution of the model, which is included due to the inaccuracy of the IC/BC or the mathematical representations of the physical processes and nature. As said before, the variable chosen for the perturbation is the temperature, T. Let $T_0$ be the initial temperature of the model in time and space. We defined $T_0^{*}$ as the perturbation of the IC $T_0$, with an additive error,

$$T_0^{*}=T_0+ϵ.$$

(2)

where $ϵ$ is a matrix or vector with adequate dimensions. Here, $ϵ\sim f\left(\beta \right)$, with f being the PDF and $\beta$ the parameter set. The disturbance $ϵ$, was independently sampled 100 times for each distribution and used in (2) to run the model. The error distribution for each case is defined as follows:

1.

${ϵ}_1\sim N(\mu =0,{\sigma }^2=1)$.

2.

${ϵ}_2\sim N(\mu =0,{\sigma }^2=0.25)$.

3.

${ϵ}_3\sim exp(\lambda =0.7291)$.

The scale parameter of the exponential distribution is determined by the normalized average surface temperature (at 2 m) of the WRF model in the inner domain (D04) of the unperturbed simulation. Calculating the scale parameter in this way ensures model stability, by restricting values to within limits set in the model configuration.

2.3. The Statistical Analysis

After the simulation, we obtained the noisy response for the variables temperature and pressure. Obtaining the noisy response for the temperature and pressure variables after the simulation is essential for several reasons:

1.

It allows for an assessment of the spatial distribution of these variables, which can provide insights into the model’s behavior under different conditions.

2.

Analyzing the noisy response over time and the topography, makes it possible to identify any model sensitivity to these factors, which can inform future model improvements.

3.

Noise analysis can also help to identify potential sources of error or bias in the model, which can be crucial for ensuring accurate results in scientific research or practical applications.

Overall, the analysis of noisy responses is a crucial step in understanding and improving the performance of models, and it can provide valuable insights into the behavior of complex systems.

In this work, we preferred non-parametric techniques rather than parametric statistics, since the former does not rely on strong assumptions. Furthermore, this provides a flexible framework, that can easily be extended to any assumption on errors.

We began by testing the normality and exponential hypothesis of errors, both visually and numerically. That was an unnecessary step, since we knew the prior distribution. Nonetheless, the small sample size could generate a false negative result on the tests. Subsequently, the evolution of the temperature and pressure variables’ errors was evaluated in time and space, allowing a contrast of the posterior distribution with the prior noise distribution. In addition, it enabled us to analyze the divergence or convergence of the model error over time. Notice that the error in the output (${ϵ}_z$) was estimated as the difference between the undisturbed (z) and noisy output ($z^{*}$):

$${ϵ}_z=z-z^{*}.$$

(3)

Although a histogram is a common method for approximating probability density functions (PDFs), kernel density estimation (KDE) can provide more accurate estimates. KDE involves the use of a kernel, which is a function that calculates local averages. According to [65], a kernel is defined as:

$$\begin{array}{c}K\left(x\right)\ge 0\hfill \\ \int K\left(x\right)dx=1\hfill \\ \int xK\left(x\right)dx=0\hfill \\ \int x^{2}K\left(x\right)dx>0.\hfill \end{array}$$

(4)

Although various kernels, including boxcar, Gaussian, and tricube, were available, the Epanechnikov kernel (refer to Equation (5)) was utilized, as it is considered the most efficient based on previous research by Moraes (2021) [66].

$$K\left(x\right)=\frac{3}{4}\left(1-x^2\right)I\left(x\right),$$

(5)

where $I\left(x\right)$ is the indicator function, defined as

$$I\left(x\right)=\left\{\begin{array}{ccc}1,\hfill & \mathrm{if}\hfill & \left|x\right|\le 1\hfill \\ 0,\hfill & \mathrm{if}\hfill & \left|x\right|>1.\hfill \end{array}\right.$$

(6)

A critical parameter of this technique is the bandwidth h. For its estimation, we employed a common zero-stage plug-in selector, defined as follows:

$$\widehat{h}={\left(\frac{4}{3}\right)}^{0.2}{n}^{-0.2}\widehat{\sigma },$$

(7)

n and $\widehat{\sigma }$ are the sample size and unbiased standard deviation, respectively.

Besides the KDE, we employed the rank test [67], a non-parametric homogeneity test based on range. This test has the advantage of having no strong assumptions about the data. The rank test evaluates whether two samples come from different populations. Let X and Y be two datasets. The rank test has a null and alternative hypothesis:

$$\begin{array}{cc}{H}_0:P(X\ge Y)\hfill & =P(X\le Y)\hfill \\ H_1:P(X\ge Y)\hfill & \ne P(X\le Y).\hfill \end{array}$$

(8)

The preceding, implies that this test only allows the rejection of the hypothesis that two samples come from the same distribution. In our case, $X=ϵ$ and Y are the deviations of all realizations from the model, compared to the true result. This was performed at all time points $\left(t\right)$ in the window evaluated $(t=1,\dots ,72)$, and at each level $(1,\dots ,34)$. Through the homogeneity test, it was possible to establish which one of the levels for every data point was homogeneous concerning the induced perturbation and, consequently, with a fixed significance level ($\alpha =0.05$ in our case).

We also applied the non-parametric bootstrap method. This technique was introduced by [68] and aims to estimate the distribution for an estimator T. It is necessary to assume $X_i\sim f$ is a sample from f, independent and identically distributed for all $i=1,\dots ,n$ and the observations $\left\{x_i\right\}$, to apply the bootstrap method. Let $X^{*}$ be k samples (in our case 1000 [69]) of size n from $\left\{x_i\right\}$ with a replacement. Then, $T\left(X^{*}\right)$ is called the point bootstrap estimate [68,70,71].

On the other hand, there exist many methods to make the interval estimates; we employed the percentile method (with a significance level of $\alpha =0.05$), for its low computational cost; however, it is possible to use a more expensive method, such as the adjusted percentile method, the bias-corrected and accelerated $B{C}_{\alpha }$, the pivotal method, or the studentized one, each with its advantages and disadvantages [65]. A significant advantage of this technique is that it can be applied to any function, particularly to mean, standard deviation, skewness, and kurtosis. This allowed us to characterize the output error at any time and level, from the point and interval estimates of its numerical descriptive statistics. Finally, we developed an indicator, defined as the percentage of times the real value is inside the confidence interval.

3. Results and Discussion

The results are organized into different sections. The first section covers the histogram, KDE, and theoretical density of the disturbances ${ϵ}_i$. Next, the average field of the unperturbed realizations is presented, followed by the variance of the ensemble for each type of perturbation. The information is divided into daytime (06:00–18:00 LT) and nighttime (18:00–06:00 LT) cycles. In the subsequent section, the temporal response of the selected location is obtained, for the variables temperature and pressure. These responses are compared with the unperturbed model’s output, and the output error is calculated using Equation (3). All of the above sections are repeated for each type of error ${ϵ}_i$.

After this, a rank test is performed to compare the distribution of the perturbation ${ϵ}_i$, induced over the model with the theoretical distribution. The rank test is also applied to the input and output error for all times and levels. Lastly, non-parametric bootstrap estimates of the mean, standard deviation, skewness, and kurtosis are presented for the output error for all times and levels, which are used as comparison values associated with the input error.

3.1. Error Distribution

For the uncertainty quantification for the perturbation of the initial condition, the study used three types of disturbances, and compared their distributions with the theoretical ones. Figure 3 displays the histogram, KDE, and theoretical distribution of the three error samples, each with a size of 100. The plot in Figure 3 reveals that the estimated density of the three error samples, of size $n=100$, differs significantly from the theoretical distribution, especially for the exponential error. These differences imply that comparing the distribution of the output error with the theoretical distribution, would introduce a significant error. As a result, the study employed non-parametric statistics to address this issue and obtain more accurate results.

3.2. Spatial and Temporal Qualitative Results

3.2.1. Ensemble Spatial Response over the Aburrá Valley

Figure 4 displays the average temperature and pressure of the unperturbed simulations for the Aburrá Valley. The data is shown for daytime (06:00–18:00 LT) and nighttime (18:00–06:00 LT) cycles, and the differences between the cycles in each variable are also included. The SIATA stations AMVA, Barbosa, and La Ye are indicated as points on the graph. It is worth noting that the areas with higher values correspond to lower areas in both variables, which is particularly relevant in a densely populated valley. Based on the results presented in Figure 4, it can be argued that the temperature and pressure vary significantly over a day in the Aburrá Valley. The panel for daytime (Figure 4a), shows higher temperatures compared to the nighttime panel (Figure 4c). The maximum temperature value recorded during the daytime cycle is 30.0225 °C, while during the nighttime cycle, the maximum temperature value is 25.7391 °C. This difference can be attributed to various factors, including solar radiation and anthropogenic activities.

Figure 4d,f help to understand the pressure variability caused by the topography and meteorological conditions, an essential characteristic of the TAR region.

Figure 5 corresponds to the average variance of the temperature ensembles for each distribution ${ϵ}_i$, and over the three days of simulation. In general, nights have a higher sensitivity than days, and each time has a similar shape in the distribution for all errors. In contrast, the south of the valley is the most sensitive area, where the stations of AMVA and La Ye are located. Finally, we can observe a directly proportional relationship between the variance of the error distribution and the sensitivity of the related area. It was for this reason that ${ϵ}_1$ (Figure 5b,e) show the highest values, followed by ${ϵ}_2$ (Figure 5a,d), and lastly ${ϵ}_3$ (Figure 5c,f).

Figure 6 represents the pressure variance for each error type ${ϵ}_i$. The first row, Figure 6a–c, is daytime, and the second row, Figure 6d–f, is nighttime. Compared to Figure 5, daytime exhibits a greater sensitivity than nighttime, and the impact over all the panels differed. Nevertheless, the AMVA station area presented a high variability during the daytime. In general, the impact was greater outside the valley. Finally, there was no relationship between the error variance and the results’ sensitivity. In this case, ${ϵ}_3$ (Figure 6c,f) have the highest values, followed by ${ϵ}_1$ (Figure 6b,e), and lastly ${ϵ}_2$ (Figure 6a,d). Then, Figure 5 and Figure 6 show that the model was sensitive to the topography.

3.2.2. Ensemble Temporal Response of the Model

Following the methodology described in Section 2, we have the original output (unperturbed) and the perturbed outputs named here, the background, and the ensembles accordingly. From this, we obtained the results shown in Figure 7, for surface temperature and pressure in the AMVA location, since this area has a high sensitivity in both variables. Henceforth, the background is the blue line, and the 100 ensembles are the yellow lines. In Figure 7, we also show observations from the SIATA network.

Figure 7 shows that, for some variables, the realizations were similar to the original at some point. However, there were slight differences between the background and ensembles for variables such as relative humidity and wind velocity, as shown in Supplementary Materials Figures S1–S7.

The results in Figure 7a–d were obtained using the errors ${ϵ}_1$ and ${ϵ}_3$. At first glance, Figure 7 does not reveal clear differences between the results from the Gaussian distributions and the exponential distribution; however, the distinction between temperature and pressure is clear. The main difference is that pressure has a large error in the first few hours and then becomes stable. In addition, temperature appears to have a larger relative error than pressure.

A possible explanation, is that the perturbations were over the temperature field. However, this hypothesis was rejected, through the results for the relative humidity and wind velocity variables (see Supplementary Material, Figures S1–S7). It is important to remember that the error peaked in the middle period for those variables and decreased towards the end of the simulation. This highlights that the change in the IC of a variable was reflected in other states of the model, even more than in the altered variable.

Figure 7b,d reveal significant variability in atmospheric pressure disturbances on 25 February, while there was a reduction in the following two days. Thermal changes are one of the contributing factors to the stabilization of the atmosphere after an abrupt pressure shift. For example, a cold air mass moving into an area can cause an initial pressure drop. However, as the cold air sinks and warms up, it becomes denser, and the pressure increases again. Similarly, if warm air moves into an area, it can cause an initial increase in pressure. Nevertheless, as the warm air rises and cools, it becomes less dense, and the pressure decreases again. Therefore, thermal changes can play a role in stabilizing atmospheric pressure after a sudden change.

Figure 7 compares the SIATA observations and the unperturbed/perturbed models for the surface temperature and pressure variables. Despite introducing one hundred perturbations to the model, noticeable discrepancies exist between the observed data and the model output. To quantify the differences between the model output and observed data, the root mean square error (RMSE) over time for these variables is illustrated in Figure 8. Figure 8a,b depict the RMSE values for surface temperature and pressure, respectively. The RMSE is computed as the average difference between the observed data and the model output in the AMVA location for each of the three perturbations (${ϵ}_1,{ϵ}_2,$ and ${ϵ}_3$).

From Figure 8, it can be observed that the average error concerning the observed data is comparable for all three perturbations, despite the differences in their sources. The most noticeable variation is observed at the beginning. However, for the other times, the RMSE is comparable. This could indicate the potential impact of errors in the initial input conditions on the model’s performance. However, it also highlights the model’s ability to maintain consistency, on average, over time.

3.3. Quantitative Results Based on Non-Parametric Techniques

In concordance with the previous section, non-parametric techniques were used, which allowed us to exploit the fact that the prior distribution did not need to be known.

Estimation of Density Function for Different Centrality Measures

Figure 9, Figure 10 and Figure 11 depict the density estimation for the temperature and pressure variables. The estimations are presented for median and mode values, and the model aims to obtain the most accurate estimator for calculating the distribution. In this study, the determination coefficient $R^2$ was used as a cost function, to assess the quality of the estimations.

Figure 9, Figure 10 and Figure 11 represent the temporal response and approximate density plot for level 34 (level with the highest altitude) with the respective histogram. In the case of temperature, the curve of the median and mode are closer than in the case of pressure.

As seen in Figure 9, Figure 10 and Figure 11 and in the Figures S8–S10 in the Supplementary Materials, the estimated densities were similar for both variables even in different locations. In the first two cases, the perturbation came from a symmetrical distribution (normal); in the last case, it came from a skewed distribution (exponential). The distribution of the initial temporal response, corresponding with the center, and the perturbations, were similar and almost symmetric. In conclusion, both estimators seemed to be suitable as robust choices. This could be useful for new techniques that want to overcome the sensitivity of the model.

3.4. Rank Test of the Error in the Output Concerning the Disturbance Induced by the IC

One of the goals of this study, was to establish if the original perturbations ${ϵ}_i$ have a relationship with the error, in the temporal response of the model for each time and level in the experiment. The rank-test methodology of Section 2.3, was applied for the three SIATA stations, to meet this objective. The results can be seen in Figure 12 and Figure 13, where every yellow square implies insufficient statistical evidence to reject the null hypothesis, i.e., it seems that the samples came from the same distribution. On the other hand, a black square means that the null hypothesis has been rejected, i.e., the samples came from different distributions.

Figure 12a–c show the rank test grid for temperature and ${ϵ}_1$, where the null hypothesis was rejected in nearly all levels in the first five hours; this occurred because, during this time, the output errors had a small variability, particularly, at $t=0$, the PDF is a Dirac delta function, i.e., the error was zero. In conclusion, the temperature was robust to a normal error in its IC in the short term. This was also true for pressure, as shown in Figure 13a–c.

In the case of temperature with standard distribution in errors, Figure 12a–c, there was a high concentration of levels where the hypothesis was rejected in the middle levels. ${ϵ}_2$ presented similar behavior (see Figures S10 and S11 in the Supplementary Materials); however, the normality of errors was present in more levels and for more time. Conversely, for the exponential distribution (see Figure 12d–f), the prior seemed to be maintained for almost all cases. This result is relevant, since it provides initial evidence that the Gaussian error is not conserved in time and space (vertical), a common assumption in data assimilation [72].

Similarly, the pressure results led to the same conclusions (Figure 13); nonetheless, some differences should be stressed. For instance, the normality of errors was not preserved for all levels periodically (see Figure 13a–c). Besides, ${ϵ}_3$ errors had a higher rejection ratio for pressure (see Figure 13d–f).

Combining the findings presented in Figure 12 and Figure 13, it appears that the output distribution error varied across different variables and types of error distribution. However, when the perturbations came from a similar distribution, the rank test yielded comparable results (refer to Figures S11 and S12 in the Supplementary Material), confirming that the model is sensitive to the nature of the error in the input data.

Finally, the information of the rank test can be complemented with histograms and KDE, where it is possible to see consequent results and the reason for rejecting the null hypothesis, such as a Dirac delta function as the PDF.

3.5. Bootstrap around the Time and over the Error for Certain Statistics

Finally, the bootstrap methodology for the mean, standard deviation, skewness, and kurtosis, was applied last. The idea was to evaluate the behavior of these statistics in time [70,73,74], to characterize the error distribution and compare it to the real values. Figure 14, Figure 15 and Figure 16 and their complement (see Figures S13–S27 in the Supplementary Material) collect the bootstrap results for the mean (blue), standard deviation (orange), skewness (yellow), and kurtosis (purple) at the three SIATA stations. The black line corresponds to the punctual estimate, while the dashed lines are the confidence intervals.

In Figure 14, Figure 15 and Figure 16, significant variability is apparent in the estimates over time for all locations and variables, particularly for the mean, standard deviation, skewness, and kurtosis. The variability in the mean and standard deviation are lower than the other two statistics. These observations suggest that the realizations obtained using perturbations were inconsistent with the disturbance generated, indicating a lack of fit between the model and the data. Additionally, the rate of occurrence inside the confidence interval was relatively low for the mean, standard deviation, and skewness. While the descriptive statistics were generally unbiased, the standard deviation was typically downward biased.

In the case of kurtosis, this metric worked better, especially at the end of the simulation window, falling inside the confidence interval more than 60% of the time. For the kurtosis and skewness, after roughly hour 20, the estimated parameter started to stabilize. This verifies the rank test results, since the forecast had minor errors in the first hours.

In the Supplementary Materials, Figures S18–S23, it is seen that the kurtosis tended to be more stable than the other variables; this stabilization was more perceptible around 20 h, as previously stated, and worked better in the case of the Gaussian perturbations. The rate of fall in the confidence interval for the exponential case was lower, but the difference was irrelevant.

The results introduced in this manuscript use both classical statistics and non-parametric statistics. In all cases, we aimed to show the behavior of the WRF model when its input data is perturbed or contaminated. The WRF model is used in the meteorological community to forecast and study phenomena previously observed in the atmosphere, so understanding its behavior in different scenarios is essential for the practitioner and expert user.

The outcomes obtained here were consistent with [18]; using an ensemble can be an alternative to forecasting, because the mean of the ensemble is close to the control run. The advantage of this work, concerning [18], is that there is no problem with interpreting results, because the same parametrization in the model was always used, so the interpretation was simple.

Figure 9, Figure 10 and Figure 11 might be evidence of the suggestions from [18], that a way to reduce the error associated with the configuration and parametrization of the model, is through the use of ensemble or multiple realizations, with the advantage that, in this case, the source of the errors is known. However, it is essential to study more variables or use more model realizations.

The previous results showed that the model is sensitive to perturbations in the input data, and this impact depends on the variables analyzed. Besides, the error induced in the input was maintained over time, at least for the 72 h analyzed here, which is similar to the time used by the weather forecast for the Colombian Air Force (FAC in Spanish) [58]. Note, that a variable as crucial as the wind velocity was very sensitive to perturbations, impacting the other outputs critically. All of the above was independent of the location. The results confirm the chaotic nature of the model [75] and evidence the difficulty of obtaining accurate predictions of its behavior following the introduction of errors in the input data.

Another significant result, is the behavior’s consistency in distributions. The origin of the perturbations in the input was inconsistent in the output, despite exceptions at some levels, time, or error distributions. Nevertheless, it was clear that assuming the Gaussian hypothesis was a mistake, a key finding for data assimilation, a technique used to reduce the uncertainty in a model [76], and where the prior and posterior distributions were assumed to be normal [53,77]. For instance, ref. [48] developed a sensitivity analysis of WRF based on ensembles using data assimilation and, therefore, the Gaussian assumption. So this finding is expected to foster good practices in sensitivity analysis, avoiding wrong assumptions.

To our knowledge, non-parametric and robust statistics have not previously been used in WRF sensitivity analysis. We found that it is a helpful tool to provide more information about the model’s behavior, either to validate the hypothesis or to reduce uncertainty, without making strong assumptions. Another differentiating factor of our work, is that WRF sensitivity analysis using ensembles usually includes data assimilation [48], while we avoided this and analyzed the output directly.

4. Conclusions

In this work, we analyzed the evolution of the WRF model’s behavior associated with changes in IC, and the relationship between the disturbance in the input data and the model’s output. As perturbations, we used three distributions and tried to trace the evolution of the error in time and space. The analysis performed is supported by the non-parametric statistics.

The results showed that the perturbation induced in the model’s input is propagated to all vertical levels and times of the model’s output, demonstrating a modification of the output PDF of the variable studied, compared with the input. Consequently, the error in the input data was not preserved through the model. It was necessary to model the output PDF differently to the input. Nevertheless, this provides a positive insight; the non-preservation of the prior output error in the short term, was due to the model’s robustness in the first five hours.

Concerning the error evolution in time, we found that the expected value of the variables studied was around zero, with some variation. However, relevant changes were seen in the standard deviation, skewness, and kurtosis.

The results are consistent with the work of [18]. An ensemble can be an alternative to a forecast, because the mean of the ensemble was close to the control run. The advantage of our work, concerning [18], is that there were no problems interpreting the results, because the same parametrization was always used in the model, in addition to our knowledge of the theoretical prior distribution.

On the other hand, our results consistently showed that the model was sensitive to the input used. Moreover, it was possible to see that the outputs’ results were inconsistent with the perturbation used in the input. Additionally, we observed that the growing up of the error in the WRF model was non-linear [75] for the analyzed variables.

The results show, that the WRF model is sensitive to the location, topography, and in general, to the domain chosen to run it [9,14,17,78]. Furthermore, even when they supported their sensitivity analysis to variations in the parametrization, and compared it with other locations, our results consistently show variations in the variables in their prognostic or distribution obtained.

It was observed in the analysis of the error for the exponential perturbations, that there is less rejection of the rank test. So, it appears more probable that the Gaussian nature disappeared in the output, compared to the exponential case. Additionally, it was noted that the model produced similar results when exposed to errors from the same source or nature.

On the other hand, changes observed in the model’s error were similar when the locations changed. Thus, the impact of the perturbations is consistent in latitude and longitude.

The approach allows for the calculation of a mathematical framework for uncertainty reduction in WRF for several applications. For example, the results suggest that the prior PDFs used in a data assimilation method must have been modeled dependent on the purpose and location [79,80,81,82], and did not assume a Gaussian.

We addressed the goal of explaining the contributions of errors in the input data, while extending technical details. Reducing uncertainty required a deep understanding of the dynamics and a mathematical characterization of the TAR. Our non-parametric and robust statistics approach improved on previous findings, constitutes the fundamentals for future model data assimilation, and fosters good practices in avoiding wrong assumptions. The next step may be using the perturbed output from WRF in CTMs, to better understand the possible disruption in their results.

Finally, the methodology based on the ensemble used here, could have been operationally feasible due to the computational cost and the availability of outputs for its processing. However, it could lead to understanding the WRF sensitivity, adequate setup, parametrization, and assimilation to couple with CTMs such as CMAQ, LOTOS–EUROS, and WRF-Chem.