**1. Introduction**

Global warming has caused significant changes in the climate. In recent years, the severity of droughts, floods and extreme events has increased in different parts of the globe. The Intergovernmental Panel on Climate Change (IPCC) was established to identify its effects and especially how human activities affect it. In order to conduct climate change studies, climate variables under the influence of greenhouse gas emissions must first be simulated [1].

One of the important consequences of climate change is the change in the meteorological parameters' trend, especially the precipitation trend [2]. Therefore, a grea<sup>t</sup> deal of research has been conducted to evaluate climate change's effect on extreme rainfall events. These studies showed that global warming is affecting and causing climate changes based on the Coupled Model Intercomparison Project Phase 5 (CMIP5) and Coupled Model Intercomparison Project Phase 6 (CMIP6) climate reports in Canada [3]. Compared to other methods such as the multi-model ensemble (MME), the CMIP5 and CMIP6 models showed better performance. In addition, various methods can be used to reduce their

**Citation:** Salimi, A.; Ghobrial, T.; Bonakdari, H. Comparison of the Performance of CMIP5 and CMIP6 in the Prediction of Rainfall Trends, Case Study Quebec City. *Environ. Sci. Proc.* **2023**, *25*, 42. https:// doi.org/10.3390/ECWS-7-14243

Academic Editor: Silvia Kohnova

Published: 16 March 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

uncertainty [4]. With the release of the sixth report (CMIP6), the desire to examine the performance of this report compared to the fifth report (CMIP5) has increased among researchers. One of the major improvements in CMIP6 is the introduction of socio–economic scenarios [5]. Examining the difference between the data from the report 5 and 6 climate models for temperature and precipitation shows that for the fixed time intervals, most of the temperature indices show higher predicted changes in CMIP6 when compared to CMIP5 in Canada. Rainfall changes in CMIP6 mainly occurred in extreme precipitation indices [3]. However, it is clear that the method of General Circulation Models (GCM) ensembles can lead to different estimates of future mean changes and different levels of uncertainty in those estimates [4]. Overall, current research has shown that the CMIP6 ensemble provides a narrower band of the uncertainty of future climate projections specifically for North America and brings more confidence to hydrological impact studies [6]. The importance of such analysis is in assessing risk and future vulnerability, and implementing efficient measures to control the changes made in the flow of rivers and their ability to warn of floods [5,7].

It is necessary to examine the system's response as a general unit for determining the possible effects of climate changes such as the increase in the concentration of greenhouse gases and the impact of socio–economic activities on the climate system [5]. For this purpose, it seems necessary to use climate models. These models include the main stages that occur in the climate system and calculate the corrections of different components when responding to the changes in the forcing factors. Therefore, evaluating the accuracy of the data of these models and choosing the most efficient and adaptable models are important and necessary steps for any forecasting [8]. This assessment is more important for precipitation, which has a more significant behavioral complexity than other meteorological phenomena. Therefore, identifying the mechanism and evaluating the effectiveness of atmospheric general circulation models in estimating precipitation and knowing their temporal and spatial frequency significantly affects the preparedness for such extreme events. Therefore, in this research, the effectiveness of the CMIP5 and CMIP6 for predicting extreme rainfall events is assessed, and the future trend of rainfall reported by the superior model is evaluated.

### **2. Methods and Materials**

### *2.1. Data and Models*

Monthly precipitation records of the Jean Lesage Intl Station (Figure 1) were collected from Canada Gov. historical meteorology data records [9]. The data-set period was from 1953 to 2020 (67 years). In addition, the CMIP5 and CMIP6 models were used to investigate the accuracy and evaluate future climate change under different scenarios. For these purposes, 2 different models (one from each CMIP) were selected according to previous research results [3,5]. These models are reported in Table 1. The historical period for evaluating the accuracy of the selected models was chosen. For CMIP5, this period was between 1953 to 2005 and for CMIP6, the period from 1953 to 2010 was selected.

**Table 1.** The selected CMIP models.


### *2.2. General Circulation Models*

Climatic variables are simulated under the influence of increasing or decreasing greenhouse gases. There are different methods for this task; however, the most reliable is the use of atmospheric general circulation models or GCMs. GCMs can be used to understand

the dynamics of the physical components of the atmosphere that are related to climate change phenomena. The purpose of using GCMs is to obtain spatial–temporal patterns of climate changes as well as long-term forecasting of climate variables [5]. Climate modeling is an important tool for understanding past, present and future climate changes [2]. In other words, currently the most reliable tool for investigating the effects of climate change on different systems is the use of a GCM. These models are able to model the trends of atmospheric and oceanic parameters for a long-term period using approved IPCC scenarios [2]. Their main weakness is the low spatial resolution and the simplification they consider for climate processes. To overcome the weakness of low resolution, it is necessary to scale the output of these models before using them in climate-change impact-assessment studies [1].

**Figure 1.** Selected Station position.

### *2.3. Mann–Kendall Trend Analysis*

The Mann–Kendall method was first presented by Mann (1945) and then expanded and developed by Kendall (1970). Among the non-parametric tests, the Mann–Kendall test is the best choice for checking the uniform trend in series [10]. This test is used to determine the randomness and trend in the series. First, to determine the non-parametric nature of the statistical series, data are arranged and ranked in ascending order and then based on that, the randomness of the data with no trend is specified. If there is a trend in the data, then it is non-random.

The null hypothesis of the Mann–Kendall test indicates randomness and the absence of a trend in the data series, and the acceptance of the one hypothesis (rejection of the null hypothesis) indicates the existence of a trend in the data series [10].

### *2.4. Evaluation of Performance*

Five types of statistical indices were employed to assess the performance of the CMIP data sets. The correlation coefficient (R) (Equation (1)) as a correlation-based index, Normalized Root Mean Square Error (NRMSE) (Equation (4)), Bias (Equation (2)), Root Mean Square Relative Error (RMSRE) (Equation (5)) and Slope (Equation (3)). The mathematical definitions of the mentioned indices are as follows:

$$R = \frac{\sum\_{i=1}^{n} \left( y\_i - \overline{y} \right) \left( \mathbf{x}\_i - \overline{\mathbf{x}} \right)}{\sqrt{\sum\_{i=1}^{n} \left( y\_i - \overline{y} \right)^2 \sum\_{i=1}^{n} \left( \mathbf{x}\_i - \overline{\mathbf{x}} \right)^2}} \tag{1}$$

$$Bias = \frac{\sum\_{i=1}^{n} (x\_i - y\_i)}{n} \tag{2}$$

$$SLOPE = \frac{\sum \left(\mathbf{x}\_i - \overline{\mathbf{x}}\right)(y\_i - \overline{y})}{(n-1)S^2\mathbf{x}}\tag{3}$$

$$NRMSE = \frac{1}{\overline{\mathcal{Y}}} \sqrt{\frac{1}{n} \sum\_{i=1}^{n} (x\_i - y\_i)^2} \tag{4}$$

$$RMSRE = \sqrt{\frac{1}{n} \sum\_{i=1}^{n} \left(\frac{x\_i - y\_i}{y\_i}\right)^2} \tag{5}$$

where *xi* and *yi* are the *ith* samples of the estimated and actual values (respectively), *x* and *y* are the average of the estimated and actual values (respectively), *n* is the number of samples, and *S*2*x* is the Variance of *x*.

### **3. Results**

#### *3.1. Evaluation the Performance of the Models*

The performance of the CMIPs' models is shown in Table 2, which reported the accuracy of the models to compare to the observation data. Five different indices were used for evaluation at two different time scales. Based on the correlation coefficient (R), CanESM5 had better performance when compared to the CanESM2 model. Nevertheless, the correlation coefficient on a monthly scale was poor since R values were less than 0.5, although if the scale changed to the seasonal, they improved ideally (between 0.65 and 0.75).

**Table 2.** Performance of CMIP models.


In addition, Normalized Root Mean Square Error (NRMSE) results showed that CanESM5 had better performance when compared to CanESM2 on the seasonal scale. These results are promising and show that the model can estimate precipitation within acceptable errors because the NRMSE values are close to one, which means that the deviations in precipitation estimates are small.

The RMSRE is a criterion similar to RMSE; their main difference is that RMSRE is divided by projected values. The best value for these criteria is 0, meaning there is no difference between projected and observed values. Based on Table 2, the CanESM5 model performed better with an RMSRE of 0.3 and 0.67 for the monthly and seasonal scales, respectively.

Mean Bias deviation shows the systematic error in the amount of precipitation. A value of zero indicates that the difference between the observed and predicted precipitation amount is not systematic, while a large Bias indicates that the amount of precipitation deviates greatly from the observed amount of precipitation. The fact that the Bias parameter is close to zero also indicates the model's accuracy in the simulation. A negative Bias indicates underestimation, while a positive Bias indicates overestimation. Based on the results in Table 2, both models underestimated precipitation at this Station.

Finally, the Slope is used to assess the direction of the projection line or the angle coefficient. If the Slope is negative, the relationship between the two variables (X and Y) will be inverse, and the Slope expresses the amount of change in Y relative to each unit of change in X. For this index, a Slope of 1, or regression 1:1 between the two variables, means

perfect correlation. The value of the Slope in Table 2 shows that the estimated data were far from the regression line (1:1) and the value of the Slope statistic for both models was equal to or less than 0.5.

Taylor diagrams provide a visual framework for comparing a suite of variables from one or more test data sets to one or more reference data sets. Commonly, the test data sets are model experiments while the reference data set is a control experiment or some reference observations (e.g., Station data sets). Generally, the plotted values are derived from climatological monthly, seasonal or annual means. Because the different variables (e.g., precipitation, temperature) may have widely varying numerical values, the results are normalized by the reference variables. The normalized variances ratio indicates the model's relative amplitude and observed variations [11]. Figures 2 and 3 provide information about monthly and seasonal Taylor diagrams of the CanESM2 and CanESM5. It is clear that CanESM5s performance on the monthly scale is better than CanESM2's, although both models have poor performance on the seasonal scale. All in all, CanESM5 can provide better results compared to CanESM2.

**Figure 2.** Models' performance in monthly scale.

**Figure 3.** Models' performance in seasonal scale.

In summary, the values obtained by the models show that the efficiency of the CanESM5 model in estimating the amount of precipitation was better than the CanESM2 model. In addition, due to the closeness of the indicators that take into account the number of deviations and compare the estimated and actual time series, the mentioned model can detect fluctuations and precipitation trends in the selected Station.

### *3.2. Precipitation Future Trend*

It is more suitable to use non-parametric methods for series that cannot be fitted with a special statistical distribution and have high skewness or elongation. The Mann–Kendall test is one of the most common and widely used non-parametric trend analysis methods of time series. Data changes are identified using the Mann–Kendall method, and their type

and time are determined [12]. According to the essential role of precipitation in providing water resources, it is vital to study the process of its changes in the future. This study will help the authorities with planning and managing water resources. Since international reports have delivered serious warnings about the crisis and even the physical lack of water in the future for different parts of the world, knowing the predicted variability of this important meteorological parameter is essential [13]. Although the evaluation of the models' performance on historical data has shown that the models do not have a high ability to estimate the amount of precipitation, the comparison of the projected and observed time series shows a slight deviation and an acceptable agreemen<sup>t</sup> between the two data sets. Therefore, the future forecast of the precipitation by the selected model (CanESM5) is an effective step in understanding the precipitation pattern in the future.

Table 3 presents the Mann–Kendall parameters (Test Z) for the different time-scale rainfall future trends (from 2020 to 2049) using the best-fit model (CanESM5) for the two socio–economic (SSP) scenarios. The selected SSP scenarios for this study were 4.5 (Pessimistic) and 8.5 (Optimistic). The time scales were Monthly (Jan to Dec), Seasonal (Spring to Winter) and Annual for the selected model's scenarios (4.5 and 8.5). For scenario 4.5, rainfall changes were decreasing for February and June to August, while for other months, these changes were increasing. These outputs can be found based on Test Z values in Table 3. Moreover, rainfall changes in the seasonal scale were also decreasing in the Summer and Fall. Trend analysis results for May, for July in the monthly scale and for Summer and Fall in the seasonal scale showed that the downward trend was more intense since the value of Test Z (Kendall's score) reached more than −1 in this period of time.


**Table 3.** Mann–Kendall test Z values for 4.5 and 8.5 scenarios.

On the other hand, for scenario 8.5 (Table 3), rainfall changes were increasing in most of the months, although, like scenario 4.5, the decreasing trend remained. However, the intensity of the downward trend became less prominent when compared to scenario 4.5. In addition, rainfall changes in the spring tended to decrease, unlike the 4.5 scenario.
