Evaluation of the Performance of CMIP6 Climate Models in Simulating Rainfall over the Philippines

Abstract

The Philippines is highly vulnerable to multiple climate-related hazards due to its geographical location and weak adaptation measures. Floods are the most catastrophic hazards that impact lives, livelihoods, and, consequently, the economy at large. Understanding the ability of the general circulation models to simulate the observed rainfall using the latest state-of-the-art model is essential for reliable forecasting. Based on this background, this paper objectively aims at assessing and ranking the capabilities of the recent Coupled Model Intercomparison Project Phase 6 (CMIP6) models in simulating the observed rainfall over the Philippines. The Global Precipitation Climatology Project (GPCP) v2.3 was used as a proxy to gauge the performance of 11 CMIP6 models in simulating the annual and rainy-season rainfall during 1980–2014. Several statistical metrics (mean, standard deviation, normalized root means square error, percentage bias, Pearson correlation coefficient, Mann–Kendall test, Theil–Sen slope estimator, and skill score) and geospatial measures were assessed. The results show that that CMIP6 historical simulations exhibit satisfactory effectiveness in simulating the annual cycle, though some models display wet/dry biases. The CMIP6 models generally underestimate rainfall on the land but overestimate it over the ocean. The trend analysis shows that rainfall over the country is insignificantly increasing both annually and during the rainy seasons. Notably, most of the models could correctly simulate the trend sign but over/underestimate the magnitude. The CMIP6 historical rainfall simulating models significantly agree on simulating the mean annual cycle but diverge in temporal ability simulation. The performance of the models remarkably differs from one metric to another and among different time scales. Nevertheless, the models may be ranked from the best to the least best at simulating the Philippines’ rainfall in the order GFDL, NOR, ACCESS, ENS, MRI, CMCC, NESM, FIO, MIROC, CESM, TAI, and CAN. The findings of this study form a good basis for the selection of models to be used in robust future climate projection and impact studies regarding the Philippines. The climate model developers may use the documented shortcoming of these models and improve their physical parametrization for better performance in the future.

1. Introduction

Climate change has been identified to pose threat to biodiversity and livelihoods and cause the economy to become volatile in countries that depend mostly on climate-related sectors such as agriculture. Therefore, understanding the past, current, and future climate trends, variations, and changes are vital for devising ways to curb and curtail the negative impacts of climate change. Climate models are important tools in understanding anticipated future climate changes around the globe. Nevertheless, the projection skills of climate models are largely dependent on their ability to simulate the historical climate of a given location. For instance, projections of future extreme El Niño occurrence frequencies are significantly overestimated owing to the systematic biases of current state-of-the-art climate models in simulating the past climate [1].

The global general circulation models (GCMs) have evolved, and their capabilities are improving as the model development moves from one phase to another. Numerous modeling groups around the globe have aided the development of climate models. The CMIP climate group under the World Climate Research Programme (WRCP) has been producing climate simulation models under different phases [2,3]. The current one, CMIP6, is different from the other phases in terms of the reference year and radiative forcing and the projections are grouped into four categories termed socioeconomic pathways (SSP) [4].

Several studies have evaluated the effectiveness of CMIP6 climate models in simulating different climate variables over various parts of the globe. For instance, Oduro et al. (2021) [5] assessed the mean surface temperature over Ghana and documented the better performance of the current CMIP6 models in simulating interannual variability than seasonal variability. Kim et al. (2020) [6] evaluated the teleconnection patterns responsible for heatwaves over Korea and found that the CMIP6 models skillfully simulate the heatwaves and teleconnection patterns better than the models from their predecessor (CMIP5). On the same note, Khadka et al. (2022) [7] compared the performance of CMIP6 and CMIP5 and found that the CMIP6 models could more skillfully simulate the onset, cessation, and rainy-season rain lengths than the CMIP5 models over the South Asia domain. This improved performance was attributed to the enhanced climate modelers’ skills in simulating land–atmosphere interactions [2].

Over China, Wang et al. (2021) [8] documented that some CMIP6 models exhibited dry bias during summer, whereas others showed wet bias during spring. Nevertheless, the models exhibited satisfactory skill in simulating the decadal characteristics of annual rainfall. Similar findings were documented by Ngoma et al. (2021) [9] over Uganda; some CMIP6 historical climate models overestimated/underestimated the annual rainfall cycle over the country, whereas over Nigeria, the models exhibited wet/dry bias in simulating the observed spatial and seasonal amounts of rainfall [10]. Fan et al. (2022) [11] observed that most of the CMIP6 models evaluated over the Pan-Third Pole (PTP) region can proficiently capture the observed spatiotemporal variability of mean and extreme temperature despite inherent dry bias over the Tibetan Plateau.

The Philippines has a mixed-type economy whereby 41% of the land mass is occupied by agriculture. Notably, the country is disaster-prone and is ranked among the top 10 countries (including Myanmar, Vietnam, and Thailand) that have been greatly affected by the negative impacts of climate change during 1997–2016 according to a report by Thomas et al., 2013 [12]. Additionally, the temperature has been rising over recent decades and the warming trend is expected to continue in the future [13]. The Philippines is among five countries around the globe that are projected to be negatively impacted by global warming [14]. This calls for the need for credible future rainfall projections to help devise appropriate mitigation measures to curb and mitigate the impacts of extreme events.

The frequency and intensity of extreme climate events (tropical cyclones, floods, drought, and landslides) are increasing and predicted to intensify in the future [15,16]. Climate change has aggravated these events and climate models that can accurately predict their occurrence are desperately needed to save lives and the Philippines economy at large. Nevertheless, few climate studies have been conducted on the southeast Asia region despite it being identified as one of the regions that is highly susceptible to the negative impacts of climate change.

There is a paucity of comprehensive knowledge about the performance of the current state-of-art climate models in simulating rainfall over the Philippines. For instance, Iqbal et al. (2021) [17] reported that the CMIP6 climate models could reasonably capture the observed spatiotemporal rainfall patterns over the South Asia region but that the performance of the models differs from one location to the other. This calls for the evaluation of climate models specifically tailored for each country and each climate variable. Based on this backdrop, this study objectively aims at ranking 11 CMIP6 models’ historical simulations based on their ability to reproduce the observed rainfall over the Philippines. The reliability of future climate projections is contingent on the effectiveness of the CMIP6 models to simulate the historical climate.

The rest of this paper will be structured as follows: Section 2 will elucidate the data types, the climate characteristics of the study region, and the evaluation techniques deployed. Section 3 will elaborate on the results. The discussion and conclusion will be drawn in Section 4 and Section 5.

2. Materials and Methods

2.1. Study Area

The study area is bounded by latitudes 5°–23° N and longitudes 115°–130° E, as depicted in Figure 1, and is herein referred to as the Philippine domain. The country is characterized by complex topography that ranges from the lowlands to the highest mountain heights of Mt. Apo National Park in Mindanao. The monsoons primarily drive rainfall patterns over Asia. The major monsoon systems include the summer monsoons that occur from mid–late May and extend all the way to September. The summer monsoons are normally termed “Habagat” and the winter monsoons are referred to as “Amihan” [18]. During October to November, the country normally receives some showers that result from retreating monsoons that shift to become winter monsoons. During summer monsoons southwest winds dominate, whereas during the winter monsoon season the prevailing winds shift to become northeasterly and bring cold dry conditions over the country [19]. Winter monsoons originate from Siberia/Mongolia/the South China Sea and are conveyed by the anticyclonic flow to the country. The rainfall-modulating factors over the country include tropical cyclones that originate from the mid-Pacific Ocean, monsoons systems, El Niño southern oscillation (ENSO), and mesoscale systems [20].

2.2. Data Source

The simulated monthly rainfall data from 11 CMIP6 climate models were evaluated against the observed rainfall over the Philippines from 1980–2014. The CMIP6 climate models are publicly available and archived at https://esgf-node.llnl.gov/search/cmip6/ (accessed on 17 December 2022). The CMIP6 models have different historical runs; for the sake of comparison, the first ensemble member (r1i1p1f1) of each model was used. Remarkably, the use of different ensemble members in the analysis showed no significant difference between rainfall amounts simulated by the CMIP6 models over North Africa [21]. The CMIP6 models have different horizontal scales, so all the models’ outputs were re-gridded using the bilinear interpolation method to a common horizontal resolution of 1° × 1°. Notably, the ensemble model (ENS) will be alphabetically represented by “E”. More details of the CMIP6 climate models used in this study are shown in

Table 1. Characteristics of the CMIP6 models used including model name, horizontal resolution, and the modeling institution.

Model Abbreviation	Alphabet Used	Horizontal Resolution	Modeling Centre
ACCESS-ESM1-5	ACCESS (A)	250 km	Commonwealth Scientific and Industrial Research Organization and Bureau of Meteorology, Australia
CanESM5	CAN (B)	500 km	Canadian Centre for Climate Modeling and Analysis (Canada)
CESM2	CESM (C)	100 km	National Center for Atmospheric Research (NCAR) Boulder, CO, USA
CMCC	CMCC (D)	100 km	Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici, Lecce 73100, Italy
FIO-QLNM	FIO (F)	100 km	First Institute of Oceanography, State Oceanic Administration, Qingdao 266061, China), QNLM (Qingdao National Laboratory for Marine Science and Technology, Qingdao 266237, China)
GFDL-ESM4	GFDL (G)	100 km	National Oceanic and Atmospheric Administration, Geophysical Fluid Dynamics Laboratory, Princeton, NJ 08540, USA
MIROC6	MIROC(H)	250 km	JAMSTEC (Japan Agency for Marine-Earth Science and Technology, Kanagawa, Japan), AORI (Atmosphere and Ocean Research Institute, The University of Tokyo, Chiba, Japan), and NIES (National Institute for Environmental Studies, Ibaraki, Japan
MRI-ESM2-0	MRI (I)	100 km	Meteorological Research Institute, Tsukuba, Ibaraki 305-0052, Japan
NESM3	NESM (J)	250 km	The Nanjing University of Information Science and Technology, Nanjing, 210044, China
NorCPM1	NOR (K)	250 km	NorESM Climate modeling Consortium consisting of CICERO (Center for International Climate and Environmental Research, Oslo 0349), MET-Norway (Norwegian Meteorological Institute, Oslo 0313), NERSC (Nansen Environmental and Remote Sensing Center, Bergen 5006), NILU (Norwegian Institute for Air Research, Kjeller 2027), UiB (University of Bergen, Bergen 5007), UiO (University of Oslo, Oslo 0313) and UNI (Uni Research, Bergen 5008), Norway. Mailing address: NCC, c/o MET-Norway, Henrik Mohns plass 1, Oslo 0313, Norway
TaiESM1	TAI (L)	100 km	Research Center for Environmental Changes, Academia Sinica, Nankang, Taipei 11529, Taiwan

.

The Global Precipitation and Climatology Project (GPCP), which is a gauge-based gridded dataset, was used to evaluate the capability of the models to simulate the rainfall amount, seasonality, and interannual variability of rainfall over the Philippines. The datasets are produced for monthly, pentad (five-yearly), and daily precipitation intervals from surface and satellite measurement from 1979–present (or 1997–present for daily). The World Data Center for Meteorology serves as a secondary conduit for this data in conjunction with NASA/GSFC. The GPCP One-Degree Daily Precipitation (1DD) Datasets provide daily, global 1X1-deg gridded fields of precipitation totals for October 1996–present with a delay of about three months. Moreover, the 1DD datasets draw upon several different data sources covering different areas of the globe. The complete record has been made as homogeneous as possible given the different available input sources. The satellite gauge and associated precipitation error estimates are the GPCP’s two primary monthly sources. The version 2.3 monthly product covers January 1979–present, with a delay of two to three months for data reception and processing. In addition, version 2.3, which was employed in the current study, supersedes all previous versions of the GPCP monthly product, including versions 1, 1b, V2x79, 2, 2.2, and 2.1. It should be noted that the previous versions are currently not available from the NOAA/NCEI archive.

2.3. Evaluation Metrics

This study adopted various statistical and geospatial measures. The descriptive and inferential statistics measures used included the temporal mean, Pearson correlation, percentage/relative bias, and normalized root mean square error (nRMSE). The effectiveness of the models was assessed based on their ability to simulate the movements of the Intertropical Convergence Zone (ITCZ), seasonal rainfall amounts, interannual variability, and multi-decadal trends.

This study used the normalized nRMSE to elucidate the relative average oddness of the GCM models from the reference dataset. This method is better than the use of RMSE, since it evaluates the models from the common ground by diving their RMSE, which is a statistically based formula for calculating the standard deviation (Equation (1)) by the mean/standard deviation of the reference data (Equation (2)). In this study, the RMSE was normalized by the mean, where μobs refers to the mean of the observation and is thus considered as a coefficient of variation (Equation (2)).

$$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{\sqrt{\frac{1}{\mathrm{n}-1}\sum _{\mathrm{i}=\mathrm{n}}^{\mathrm{n}}\left({\mathrm{x}}_{\mathrm{i}}-\mathsf{\mu }\right)}2}{\mathrm{n}}$$

(1)

$$\mathrm{n}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}}{{\mathsf{\mu }}_{\mathrm{o}\mathrm{b}\mathrm{s}}}$$

(2)

Relative and percentage bias measures were used to understand the inconsistency among the CMIP6 models in simulating the observed rainfall amounts. The relative bias was divided by the observed mean field to scale all the models and evaluate them on common ground (Equation (3)). The percentage bias can be mathematically expressed as

$$\mathrm{P}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{a}\mathrm{g}\mathrm{e} \mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}=\frac{{\mathrm{C}\mathrm{M}\mathrm{I}\mathrm{P}6}_{\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{s}}-{\mathrm{O}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}}_{\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{s}}}{{\mathrm{O}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}}_{\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{s}}}\ast 100\%$$

(3)

Rainfed agriculture plays a pivotal role in the economy of the Philippines, hence understanding rainfall trends is paramount. In this regard, non-parametric trend detection with a modified Mann–Kendall test was deployed [22,23]. This test signifies the sign of the trend, whereas its magnitude was tested using a non-parametric Theil–Sen slope estimator TSSE [24]. Notably, the use of non-parametric methods has been identified to be more vigorous than the use of their counterparts’ (parametric methods). This is because non-parametric methods are less susceptible to outliers/extreme values that may be present in the data [25]. These methods have been successfully used in previous studies around the globe to understand rainfall trends and magnitudes [26,27].

To accomplish the main objective of this study (ranking the models), skill scores (SS) were used following a study by Fu et al. (2013) [28]. This method combines the skills of the CMIP6 rainfall models at both the seasonal and annual scales. Since different metrics were used, the absolute error of each model in terms of individual metrics is obtained and then they are added up. For instance, in the case of the correlation coefficient, an ideal value (1) is used to capture the nonconformity. Notably, the smaller the SS, the higher the skills of the model in capturing the concerned measures and vice versa.

This study used these scalar metrics to rank the CMIP6 models: the Pearson correlation coefficient (PCC), normalized RMSE (nRMSE), absolute errors in a modified Mann–Kendal (MMK), Theil–Sen Slope Estimator (TSSE), and relative bias. Mathematically, SS can be illustrated as

$$\mathrm{S}\mathrm{S}=\left(1-\mathrm{C}\mathrm{C}\right)+\mathrm{n}\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}+\left|\mathrm{R}.\mathrm{B}\right|+\left|\mathrm{M}\mathrm{M}\mathrm{K}\right|+\left|\mathrm{T}\mathrm{S}\mathrm{S}\mathrm{E}\right|$$

(4)

3. Results

3.1. Annual Rainfall Cycle

The ability of the CMIP6 models to simulate the observed annual rainfall cycle is crucial for agricultural purposes, which are season dependent. According to Sperber and Palmer (1996) [29], a good GCM should skillfully simulate the annual cycle of a given climate variable. The Philippines usually receives one rainy season as a result of monsoonal influence. Based on the observed data, the rainy season usually starts in May and persists until November (MJJASON), with a peak in mid-June (Figure 2).

The CMIP6 climate models simulate the observed annual cycle of rainfall effectively, though some models over/underestimated the observed magnitude (Figure 2). Specifically, six models (CMCC, FIO, GFDL, MIROC, NOR, and TAI) and the multi-model ensemble mean model (ENS) underestimated summer rains, whereas four models (ACCESS, CESM, MRI, and NESM) exhibited a wet bias (Figure 2). Notably, most of the models were unable to simulate the rain peak except for GFDL and FIO. This observed bias necessitates the need to correct the errors, since they are likely to be carried forward to the projections (SSPs) and create high uncertainty in them. Notably, the GPCP data show a close pattern to the observed station data and are hence deemed suitable to be used in the subsequent analysis.

Multiple aspects of the CMIP6 models’ performances in simulating rainfall were assessed and summarized in a Taylor diagram. The results show that the models are skillful in simulating the observed annual cycle, as evidenced by the low spatial variability and the high level of consensus among the models. The best-performing model in this category is CMCC, followed by the multi-model ensemble mean (ENS), whereas the worst-performing model is NESM (Figure 3).

3.2. Latitudinal Spatial Rainfall Distribution (Hovmöller Diagram)

The seasonality of the rainfall over the Philippines is modulated by the movement of the monsoonal systems and the Intertropical Convergence Zone. The GCM has been shown to have the ability to simulate large-scale features more reliably than local features. Additionally, the effectiveness of GCMs for reliably predicting the future climate depend on their ability to capture regional historical teleconnection patterns that modulate rainfall over the area of interest [30]. The ability of the CMIP6 models to effectively simulate the movement of the ITCZ illustrates their ability to simulate teleconnection patterns that influence rainfall patterns over the study domain.

There are numerous parameters (top-cloud height, surface pressure, longwave radiation, and rainfall) that can be used to trace the movement of the ITCZ [31]. Nevertheless, these proxies seldom align; in this study, rainfall was used to trace the migration of the ITCZ. The latitudinal averages of observed rainfall over the Philippines (5°–23° N) show that the country experiences one rainy season (basically summer monsoon) and one dry season (Figure 4). Based on the observations, the rains start synchronously all over the country around mid-May and have a peak in July (Figure 4).

The models show reasonable skills in simulating the rainy-season rains season (May–November) and the dry season (February–May) (Figure 4). Notably, the CMCC simulates rainfall throughout the year at low latitudes but shows a dry condition within a latitude of 15–25° N over the region. The NorCPMI synchronously simulates the driest season during Jan–May all over the country (Figure 4). Remarkably, there is a lack of consistency amongst the CMIP6 models in simulating the onset, peak, and cessation of the rainy season (Figure 4). However, the CMIP6 models could simulate the length of the rainy-season rainfall season close to the reference dataset. These findings are concurrent with a previous study on the ability of the CMIP6 models for simulating rainy-season rains [7].

The documented discrepancies are due to a lack of skills in CMIP6 historical models for simulating the mechanisms that control rainfall over the study domain. For instance, the inability of the CMIP6 models to successfully simulate the movement of the ITCZ has been linked to the documented bias in simulating the seasonal winds over/near the tropical oceans [32]. Therefore, climate modelers are called upon to doublecheck the internal model physics of the models since they discern the model’s capability.

3.3. Spatial Distribution of Rainy-Season Rains and Related Biases in CMIP6

The spatial distribution of rainy-season rains shows that the northwest parts of the country receive the highest rainfall followed by the Visayas in the central region, whereas the southeast region receives the lowest amount of rainfall (Figure 5). This is attributed to the influence of topographical relief features that modulate the rainfall amount and distribution. For instance, as the southwest monsoon winds transport moisture from the Indian Ocean and the South China Sea before they reach the Philippines, the presence of high mountains such the Cordillera Mountain Range lifts the air and produces precipitation over the northern side of the country, leaving the lee side of the mountain with dry conditions.

The ability of the CMIP6 models to simulate the observed rainfall distribution is evaluated using relative bias. The results show that most of the models exhibit a dry bias over the land and a wet bias over the oceans (Figure 5). Based on the relative bias gauging metric, the driest model is CMCC whereas the wettest model is ACCESS (Figure 5). The extent of the relative bias ranged from −3.2 to 1.2 mm day⁻¹. This necessitates the importance of improved parametrization of the CMIP6 climate models to enable them to effectively simulate the topographical features reasonably as they modulate the rainfall distribution over the Philippine highlands and around the globe.

3.4. Descriptive Statistics of the Observed Rainy-Season Rainfall and CMIP6 Models’ Simulations

The ability of the CMIP6 models to simulate the summer mean rainfall characteristics is evaluated in this subsection. The results show that most of the models depict negative bias in simulating the amount of mean summer rainfall (

Table 2. Statistical characteristics of rainfall during May–November both in observed and CMIP6 model data. The bolded numbers depict statistically significant values at $p\le 0.1$.

Class	Mean	stdev	Bias	RMSE	nRMSE	MMK	TSSE
Obs	268.1	17.3				0.16	0.32
ACCESS	302.5	26.1	13.3%	89.7	0.34	0.14	0.6
CAN	278.1	18.2	4.1%	64.1	0.24	0.28	0.72
CESM	267.9	26.5	0.3%	60.9	0.23	−0.12	−0.42
CMCC	255.0	14.5	−4.5%	43.2	0.16	−0.02	−0.02
ENS	261.3	6.2	−2.2%	47.3	0.18	0.15	0.13
FIO	240.1	14.9	−10.1%	31.7	0.12	0.10	0.21
GFDL	250.0	21.0	−6.4%	41.8	0.16	0.16	0.48
MIROC	255.9	22.3	−4.2%	47.1	0.18	0.01	0.005
MRI	271.5	19.2	1.6%	59.6	0.22	−0.04	−0.22
NESM	265.7	16.7	−0.5%	53.0	0.20	−0.03	−0.075
NOR	217.1	19.9	−15.7%	25.5	0.10	0.02	−0.05
TAI	255.7	15.5	−4.3%	45.7	0.17	0.14	0.3

). Based on the models’ abilities to simulate the mean rainfall amounts, the wettest model in this category is ACCESS, with a bias of 13.1%, whereas NOR is the driest bias model among the evaluated CMIP6 models, exhibiting an understating of −15.7% compared with the reference dataset.

Notably, the South Asian region is characterized by high GCM bias in climate simulation, particularly for rainfall during the southwest summer monsoons as compared with the rest of the world [33,34]. This can be linked to the Southern Hemisphere albedo error [35]. This error arises due to the high absorption of short-wave radiation over the oceans in the Global South that leads to prominent biases in rainfall, wind, and sea surface temperatures [36,37]. Therefore, enhancing the simulation of spatial and seasonal features is likely to improve observation and model consistency in both historical simulations and future projections.

The summer rainfall amount insignificantly increased at a rate of 0.16 mm per annum during 1980–2014 (Table 2). The CMIP6 models exhibit reasonable skill in simulating the observed temporal trend, with the majority of the models simulating the correct sign of the trend but either underestimating or overestimating the magnitude. In this case, CAN overestimates the magnitude of the trend, whereas CESM, CMCC, MRI, and NESM produce the wrong sign (negative) for the trend of monsoonal summer rains over the Philippines (Table 2). This shortcomings of these CMIP6 models may be associated with the strong global interannual variability of rainfall [38]. Additionally, only two CMIP6 models (ACCESS and CAN) show credible temporal agreement with the observed interannual variability of the rainy-season rainfall.

3.5. Spatial Dissemination of Annual Rainfall and Associated CMIP6 Relative Biases

On the annual scale, the country observes a similar spatial distribution pattern of annual rainfall as the rainy-season rains. The southwest region observes the highest rainfall amounts annually compared with the rest of the country (Figure 6). The CMIP6 models display reasonable effectiveness for simulating the observed annual rainfall over the land but exhibit considerable bias over the seas. Most of the models exhibit a high bias over the ocean than over the land. The CMIP6 climate models inherent a biases range from −1.6 to 0.8 mm day⁻¹ over the land. Notably, the driest bias of −3 mm day⁻¹ is documented by ACCESS over the South China Sea. The multi-model ensemble documents a wet bias (0–40%) over most parts of the Philippines’ landmass, except for the southern parts of the country.

3.6. Mean Statistical Features of the CMIP6 Climate Models in Comparison with the Observed Annual Rainfall

The trend analysis shows that annual rainfall is insignificantly increasing over the Philippines at a rate of 0.2 mm yr⁻¹ (

Table 3. The statistical characteristics of annual rainfall both in observations and the CMIP6 models. The bolded values depict statistically significant MMK values at p ≤ 0.

Class	Mean	stdev	Bias	RMSE	nRMSE	MMK	TSSE
Obs	206.1	18.4				0.20	0.53
ACCESS	174.5	10.3	−15.3%	40.7	0.20	0.11	0.13
CAN	218.2	18.2	5.9%	118.2	0.57	0.28	0.69
CESM	195.2	23.4	−5.3%	44.9	0.22	0.03	0.08
CMCC	206.5	14.4	0.2%	49.0	0.24	0.01	0.05
ENS	196.3	6.4	−4.7%	38.8	0.19	0.10	0.09
FIO	189.3	11.7	−8.1%	35.3	0.17	−0.04	−0.05
GFDL	186.2	16.3	−9.6%	32.2	0.16	0.06	0.20
MIROC	188.6	25.7	−8.5%	42.1	0.20	−0.03	−0.08
MRI	187.0	17.3	−9.2%	34.2	0.17	0.02	0.02
NESM	193.7	15.9	−6.0%	39.2	0.19	−0.06	−0.07
NOR	194.7	18.1	−5.5%	26.9	0.13	−0.01	−0.05
TAI	211.5	16.9	2.7%	56.2	0.27	0.13	0.21

). Notably, most of the models can capture the sign of the trend correctly, except for FIO, MIROC, NESM, and NOR, which show a negative trend in this category. All the models underestimated the magnitude of the observed annual rainfall except CAN, which had a magnitude of 0.69 mm yr⁻¹; the observed magnitude was 0.53 mm yr⁻¹.

The degree of spread among the evaluated CMIP6 models was assessed through standard deviation and nRMSE. NOR shows a small nRMSE (least deviation from the observation), whereas CAN has the highest nRMSE (largest deviation from the reference dataset). This depicts the existence of intermodal divergence in the CMIP6 models as compared to the observations (Table 3).

Notably, most of the models underestimate the observed annual mean rainfall in reference to the gridded dataset used in this study. Based on the mean statistics, the driest model is ACCESS, whereas the wettest model is CAN (Table 3). This dry bias may be linked to the inadequacy of GCMs to reasonably simulate the cold Arabian Sea surface temperature (SST) biases [39,40], leading to underestimation of observed summer rains over the Philippines by the CMIPs. Empirically, cold Arabian SSTs usually lead to the late onset of Asian summer rains and subsequently reduce the amount of rain according to Levine and Turner, (2012) [41].

3.7. Gauging the Multiple Skills of CMIP6 Models in Simulating Observed Monsoonal Summer Rainfall over the Philippines

Skillfully simulating the interseasonal rainfall variability is essential for the economy of the Philippines, which is highly dependent on rainfed agriculture. Additionally, the country is highly susceptible to extreme climate events, hence effectively simulating the observed rainfall characteristics is crucial for mitigating future extreme weather events. The CMIP6 models exhibit high model discordance, as they are spread apart in the Taylor diagram (Figure 7). This indicates less agreement in simulating the summer rains over the Philippines. Additionally, only a few CMIP6 models (FIO, NOR, TAI, and the multi-model ensemble) could reasonably simulate the amplitude of the rainy-season rains over the country (Figure 7), whereas other models exhibit high spatial variability, as evidenced by a standard deviation of greater than one. In this case, the least performing CMIP6 models based on the combined statistics, as depicted by the Taylor summarizing tool, are CESM and CMCC.

Remarkably, based on this category, the multi-modal ensemble (ENS) model outperforms the individual models, and it is hence considered to be the best-performing model as per this model ranking criterion. Similar findings have been recorded about the capability of the CMIP6 ENS model though using a different set of CMIP6 models [21]. This conclusion about the performance of the CMIPs ENS can be attributed to the fact that the ENS minimizes the noise in the models despite the advocation of “one model one vote”.

3.8. Taylor Diagram of CMIP6 Models Skills in Simulating Observed Annual Rainfall

To summarize different facets of the relationship between the observed annual rainfall as documented by the GPCP and CMIP6 models, a Taylor diagram was used. The statistical intuitive metrics address the spatial pattern, the amplitude, and the RMSE. The RMSE is depicted by the green concentric semi-cycle, whereas the grey radial coordinates indicate the magnitude of the standard deviation (Figure 8). The results show that there is a lack of consensus among the CMIP6 models in terms of rainfall as they have spread apart (Figure 8). Nevertheless, most of the CMIP6 models reasonably simulate the amplitude of the observed patterns but with low spatial variability. The CESM, CMCC, and MRI models overestimated the differences between dry and wet months over the country (Figure 8). MRI showed the least ability for simulating the interannual variability of the annual rainfall. Notably, the ENS model, as represented by the alphabet (E), outperforms all the individual models (Figure 8).

3.9. Skill Score Ranking of CMIP6 over the Philippines

Rating the skills of the GCMs in simulating the observed climate characteristics of a given area is essential for reliable future climate projections. The credibility of GCMs projections is contingent on the capability of the GCMs to reproduce the past climate. In this regard, the performance of the CMIP6 group of models was gauged using different spatial and statistical metrics. Three different metrics were used (monsoonal summer rainfall, annual rainfall amount, and the average of the two). The results show that different models perform differently under certain measures and the performance of models differs in different time scales. For instance, the ENS outperforms all the individual models at the seasonal scale but ranks fourth out of the 11 individual CMIP6 models assessed at both the annual scale and in the overall ranking of the models. On the annual scale, the best-performing model is GFDL, whereas the worst-performing model is CAN (Figure 9).

In the final ranking, the considered CMIP6 models may be listed from the best to worst in term of simulating the observed rainfall patterns over the Philippines in the order: GFDL, NOR, ACCESS, ENS, MRI, CMCC, NESM, FIO, MIROC, CESM, TAI, and CAN. Notably, the choice of the model to be used in the projection should be based on the sole purpose of the use of the model, since different models perform differently at various time scales. Remarkably, there is not an absolutely bad model, since their skills differ in capturing certain atmospheric phenomena that modulate rainfall as the climate is governed by complex processes.

4. Discussion

The present study assessed and ranked the capability of the recent Coupled Model Intercomparison Project Phase 6 (CMIP6) models for simulating the observed rainfall over the Philippines. This was achieved by comparing the GCMs with observational datasets obtained from the Global Precipitation Climatology Project (GPCP) v2.3. The assessment was conducted in terms of spatial patterns and temporal variability for the period 1981–2014, reflecting the long-term climatology as widely advocated by the World Meteorology Organization, which defines a mean climate to be a long-term period of about thirty years. This study employed robust scalar metrics to rank the CMIP6 models; the metrics included the Pearson correlation coefficient (PCC), normalized RMSE (nRMSE), absolute errors in a modified Mann–Kendal test, Theil–Sen Slope Estimator, statistical analysis, and relative bias. The analysis was mainly based on all the models and the MME was derived by computing the simple arithmetic mean of the individual models for any rainy season from May to November.

Our results show that the CMIP6’s historical simulations exhibit satisfactory skill in simulating the annual cycle, though some models display wet/dry biases (Figure 2 and Figure 3). However, most of the models exhibit a dry bias over the land and a wet bias over the oceans (Figure 5). The dry bias over land can be linked to the inability of the CMIP6 climate models to simulate orographic effects that play a key role in modulating the rainy-season rains. The inadequacy of the CMIP models to simulate topographical relief features was observed in CMIP6’s predecessors and even regional downscaled models around the globe [26,27,42,43,44]. The Philippines is a complex archipelagic country with varied, partly mountainous topography. The effects of these important features on the climate of the country are unlikely to be well represented by GCMs. Multi-model dynamically downscaled models could therefore be useful in the analysis of local climate, so as to address the challenges of the coarse resolution of GCMs, which cannot capture the convective processes as a result of complex geomorphology. However, the main challenge of generating climate models using dynamically downscaling techniques is that they are very computationally expensive, especially when running high-resolution climate models such as the HigResMIP historical runs with at least 50 km of atmospheric resolution and 0.25° of oceanic resolution [45]. The HighResMIP experiments offer the opportunity to better understand precipitation fluctuation based on global high-resolution simulation models. Eventually, the hydrological cycle can be better reproduced to mitigate the impacts of precipitation variability on human life and the environment.

Further analysis for annual simulation showed greater bias over the ocean than over the land (Figure 6). The observed substantial dry/wet bias over the ocean/seas may be attributed to the inadequacy of CMIP6 models to effectively simulate the enormous convective activities over the South China Sea and the Indian Ocean. A similar inadequacy to simulate convective rainfall was linked to the poor parameterization of convective schemes in CMIP5 [46]. This claim is supported by Ma et al. [43], who observed that, on the tropospheric column of GCMs, there is huge warming in regions that record downward air motions based on the observations but regions that have enormous updraft activities exhibit slow warming both under greenhouse gas forcing and unified SST intensification experiments [43].

Despite the robust findings of this study, various limitations are noted that influence the current results. Examples include the limited number of models evaluated (i.e., eleven models from CMIP6), the statistical approaches employed, and the restricted number of seasons considered. Nevertheless, the results point to some remarkable findings and future directions. The study identified models that perform robustly over the region that can be used to build an ensemble for future projections. Future studies may consider resolving persistent wet biases noted over the study region. Persistent uncertainty causes various stakeholders such as policymakers and users of climate information to remain in a state of confusion about the future climate during the wet season, since the models’ reliability cannot be wholly trusted. This calls for further investigation and attribution studies into the sources of unyielding systematic biases. The overall performance in the CMIP6 models over the Philippines region calls for the model developers to improve their physical parametrization for better performance in the future.

5. Conclusions

The effectiveness of 11 CMIP6 climate models was assessed for simulating the monsoonal summer rains and annual characteristics of rainfall over the archipelagoes of the Philippines. The gridded dataset from GPCP was used as the reference data in this study. Several geostatistical and statistical metrics such as mean, nRMSE, percentage and relative bias, and skill score were used as the gauging scales. The results show that the country is characterized by a unimodal rainfall regime, termed rainy-season (summer monsoon) rains, with one dry season (winter monsoon). The spatial distribution of both the annual and summer rains shows higher rainfall amounts on the southeastern side of the country. Trend analysis from 1980–2014 shows that the observed rainfall amounts during the summer and annually are increasing insignificantly throughout the country at a rate of 0.16 mm yr⁻¹ and 0.2 mm yr⁻¹, respectively. This can be linked to an increase in greenhouse gases in the atmosphere leading to high convective activity over the land and ocean.

The CMIP6 models show satisfactory effectiveness for simulating the observed summer rainfall over the country, though some models underestimate the monthly amounts within the season. The majority of the models exhibit dry biases over the ocean but show wet biases over the land ocean. Notably, the wettest bias was recorded by ACCESS over the South China Sea. The CMIP6 exhibits unsatisfactory ability for simulating interannual variability, which was characterized by huge divergence among the models. The ENS outperformed the evaluated individual CMIP6 models in simulating summer rains over the Philippines. Remarkably, most of the CMIP6 models could simulate the observed rainfall trends, though they over/underestimated the magnitude of them. Notably, the performance of the models was different at different time scales and under various evaluation metrics. Nevertheless, considering all the evaluation measures and the time scale, this study attempts to rank the evaluated CMIP6 models from the best to worst as GFDL, NOR, ACCESS, ENS, MRI, CMCC, NESM, FIO, MIROC, CESM, TAI, and CAN. Therefore, the selection of the CMIP6 model to be used in future climate projections should be based on the intended use of the model.

The findings of this study are useful in identifying the models that may be used to produce reliable projections of the future climate under different SSPs scenarios. The climate model developers may use the CMIP6 models’ shortcomings that have been identified in this study to improve the model parametrization for better performance in the future. Caution should be exercised when choosing the model to be used in future projections due to the documented differential performance at various time scales of the CMIP6 models. The choice of the model should be based solely on the intended use of the model. Climate scientists should incorporate bias correction when deploying CMIP6 models for impact studies and climate projections due to inherent biases.