Performance Evaluation of Numerical Weather Prediction Models in Forecasting Rainfall Events in Kerala, India

Abstract

Heavy rainfall events are the main cause of flooding, especially in regions like Kerala, India. Kerala is vulnerable to extreme weather due to its geographical location in the Western Ghats. Accurate forecasting of rainfall events is essential for minimizing the impact of floods on life, infrastructure, and agriculture. For accurate forecasting of heavy rainfall events in this region, region-specific evaluations of NWP model performance are very important. This study evaluated the performance of six Numerical Weather Prediction (NWP) models—NCEP, NCMRWF, ECMWF, CMA, UKMO, and JMA—in forecasting heavy rainfall events in Kerala. A comprehensive assessment of these models was performed using traditional performance metrics, categorical precipitation metrics, and Fractional Skill Scores (FSSs) across different forecast lead times. FSSs were calculated for different rainfall thresholds (100 mm, 50 mm, 5 mm). The results reveal that all models captured rainfall patterns well for the lower threshold of 5 mm, but most of the models struggled to accurately forecast heavy rainfall, especially for longer lead times. JMA performed well overall in most of the metrics except False Alarm Ratio (FAR). It showed high FAR, which revealed that it may predict false rainfall events. ECMWF demonstrated consistent performance. NCEP and UKMO performed moderately well. CMA, and NCMRWF had the lowest accuracy either due to more errors or biases. The findings underscore the trade-offs in model performance, suggesting that model selection should depend on the accuracy required or rainfall event prediction capability. This study recommends the use of Multi-Model Ensembles (MME) to improve forecasting accuracy, integrate the strengths of the best-performing models, and reduce biases. Future research can also focus on expanding observational networks and employing advanced data assimilation techniques for more reliable predictions, particularly in regions with complex terrain such as Kerala.

1. Introduction

Heavy rainfall events are one of the primary contributors to flooding, which poses significant risks to life, infrastructure, and agriculture across many regions of the world [1]. Both seasonal-monsoon-driven extreme events and climate-change-induced sea level rise and storm surge are commonly linked with floods and landslides [2]. In several tropical countries including India, where monsoon-driven rainfall is a crucial part of the climate system, accurate forecasting of rainfall and distribution of good quality data are critical for minimizing flood-related impacts and enhancing disaster monitoring and management strategies [3].

The state of Kerala and adjacent areas in the Western Ghats region located on the southwestern coast of India are particularly vulnerable to extreme weather events, including heavy rainfall and landslides, owing to its geographical location and topographical features [4]. These rainfall events often lead to widespread flooding and inundation, necessitating robust forecasting systems capable of predicting such events with high accuracy and reliability. The 2018 flood in Kerala was the most devastating in a century, with the previous major flood in the monsoon of 1924 [5]. Recently, there has been a notable rise in extreme rainfall events around the globe. The key factor in issuing a flood alert is the accuracy of forecasting heavy rainfall events. As such, predicting these events is crucial for minimizing damage and preventing loss of life [6].

Rainfall forecasts are currently provided using conventional methods such as satellite observations, weather radars, and Numerical weather prediction (NWP) models [7,8]. Among these observations, satellites and weather radars provide 41 qualitative forecasts, while NWP models provide quantitative forecasts [9,10]. A preliminary assessment of Global Precipitation Measurement (GPM)-based precipitation estimates over India for the 2014 monsoon season highlights that all products struggle with detecting precipitation in orographic regions like northeast and southeast India [11]. NWP models are physically based models represented by governing equations, processes, and parameters that cover land, atmosphere, and ocean conditions [12], and the outputs from these models are compared and used to assess the impact across multiple spatial and temporal scales [13]. Since the early 1990s, NWP output ensembles have been utilized. It is widely accepted that ensemble forecasts offer probabilistic insights that are more useful for understanding the limitations caused by unavoidable errors in the initial conditions than relying solely on deterministic forecasts [14,15,16].

NWP model verification plays a crucial role in meteorological research and operational forecasting activities [17]. When verification methodologies are carefully designed, the results can effectively address the needs of various stakeholders, including modelers, forecasters, and users of forecast information. Durai et al. evaluated the performance of two National Centers for Environmental Prediction Global Forecast System (NCEP) resolutions, namely, T574 and T382 over India during the 2011 summer monsoon, finding that both models showed skill in capturing heavy rainfall regions, with T574 outperforming T382, though both models exhibited biases in lower and upper tropospheric moisture and circulation [18]. Forecasting heavy rainfall using Quantitative Precipitation Forecasting (QPF) remains a significant challenge, even for advanced high-resolution NWP models [19,20].

Sharma et al. concentrated on evaluating the effectiveness of the United Kingdom Met Office Unified Model (UKMO) in forecasting intense rainfall events across India. The predictions effectively represented the overall characteristics of average monsoon rainfall, particularly highlighting higher precipitation levels along India’s western coast [21]. Ashrit et al. evaluated the accuracy of NWP models utilized by the National Centre for Medium-Range Weather Forecasting (NCMRWF) in forecasting the extreme rainfall that occurred in Kerala in August 2018 [22]. The performance of five global NWP models (National Centre for Unified Modelling (NCUM), UKMO, India Meteorological Department Global Forecast System (IMD GFS), NCEP, and European Centre for Medium-Range Weather Forecasts (ECMWF)) was assessed in forecasting daily rainfall over India during the 2020 monsoon season using both traditional and advanced verification methods. While all models accurately captured large-scale monsoon patterns, forecast accuracy decreased with lead time, with ECMWF performing best overall, though regional performance varied, and models struggled to predict localized, high-intensity rainfall events [23].

The performance of the Global Forecasting System (GFS) and Weather Research and Forecasting (WRF) models for forecasting monsoon rainfall over India during the summer monsoon season of 2014 was evaluated. The GFS model showed reasonable accuracy in predicting large-scale rainfall features and significant improvement from 2013 to 2014, particularly for longer forecast periods. In contrast, the WRF model exhibited a consistent over-prediction in rainfall, highlighting the need for bias correction and improved physical parameterization schemes for better monsoon predictions [24]. ECMWF, NCEP, and UKMO models were assessed for forecasting extreme rainfall over India, and they could predict rainstorms up to five days in advance with biases in spatial distribution and intensity. NCEP showed smaller spread but less accurate averages, and all models exhibited under-prediction and increased errors in longer forecasts [6]. Venkat Rao et al. evaluated the NCEP model for rainfall forecasting in the Nagavali and Vamsadhara river basins and found good performance with correlation coefficients > 0.3 and probability of detection > 0.6 for day-1 and day-3 forecasts. Bias analysis showed a shift from overestimation to underestimation with increasing lead time. Bias correction improved RMSE by >18% for day-1 forecasts, offering insights for flood forecasting, early warning, and water management [25]. The object-based verification method reveals that the WRF model overproduces large rain areas and underestimates the diurnal cycle of rainfall, with a positive size bias, particularly in the afternoon. However, this method is sensitive to object size, which can lead to inaccurate results for smaller-scale events [26]. Another evaluation of the GFS T1534 model for the 2016–2017 monsoon seasons reveals a wet bias over land and overestimation of lighter rainfall, while underestimating heavier rainfall [17]. Singhal et al. conducted an inter-comparison of four gridded quantitative precipitation forecasts—ECMWF, Japan Meteorological Agency (JMA), NCMRWF, and UKMO—over the Ganga River basin in India and found ECMWF to be a suitable substitute for NCMRWF in detecting spatial patterns of extreme precipitation. In terms of NSE, JMA outperforms the other three NWP models. Additionally, JMA exhibited similar patterns of probability of detection when compared to ECMWF [27]. Sonawane et al. evaluated the JMA model’s ability to predict monsoon rainfall over 32 years. The study revealed that the model could effectively represent fluctuations in Indian summer monsoon precipitation, especially when incorporating influences such as El Niño and the Indian Ocean Dipole concurrently [28].

Over the years, a variety of spatial verification methods have been developed to more effectively assess the performance of high-resolution forecasts [29]. Gallus et al. noted that spatial verification provides more detailed and relevant metrics of forecast skill, offering a clearer picture of the accuracy of high-resolution predictions. Neighborhood or fuzzy methods evaluate forecast accuracy within space-time neighborhoods. In these methods, all grid-scale values surrounding an observation are treated as equally plausible estimates of the true value. “Neighborhood verification” assesses forecast skill by varying neighborhood sizes and conducting verification across different spatial scales and intensity thresholds [30]. The Fractions Skill Score (FSS), introduced by Roberts and Lean, exemplifies such fuzzy spatial verification techniques. Instead of a direct grid-to-grid comparison, FSS evaluates forecast accuracy within local neighborhoods of observations [31,32].

The performance of the NWP models can vary depending on several factors, including forecast lead time, precipitation thresholds, and regional geographical characteristics. Kerala, situated along the Western Ghats, experiences complex orographic influences that significantly affect heavy rainfall events. The interaction between the steep terrain and moisture-laden low-level westerly winds from the Arabian Sea enhances precipitation, leading to highly localized and intense rainfall. The intensity and variability of the monsoon flow further influence precipitation patterns, making accurate forecasting particularly challenging. Despite Kerala’s vulnerability to extreme rainfall and associated flood risks, there is a lack of region-specific evaluation of rainfall forecast models, limiting efforts to enhance prediction accuracy under its unique climatic conditions. Given the critical need for reliable heavy rainfall forecasting, this research aims to evaluate the ability of different NWP models in the short-term forecasting of heavy rainfall events in Kerala, India, and to assess their performance using a range of evaluation metrics. By focusing on traditional performance metrics, categorical precipitation metrics, and fractional skill scores, this study provides a comprehensive understanding of the strengths and weaknesses of various models in predicting heavy rainfall across different forecast periods.

The outcomes of this analysis are intended to guide the selection of appropriate NWP models for operational forecasting, particularly in the context of short-term forecasting, which is crucial for timely flood warning systems. Furthermore, the insights from this study are expected to contribute to improve the overall accuracy and reliability of rainfall forecasts, ultimately aiding in the better management of flood risks in vulnerable regions like Kerala.

2. Study Area and Data Sources

Kerala, situated between 8° and 13° N latitude and 74° to 78° E longitude (Figure 1), occupies the southwestern edge of the Indian subcontinent. The state features a highly varied topography characterized by coastal plains, mid-elevation hills, and the rugged Western Ghats, which rise to over 2695 m. This geographical diversity significantly influences Kerala’s climatic and hydrological dynamics. The state experiences a humid tropical monsoon climate dominated by two principal rainfall seasons: the Southwest Monsoon (June–September), contributing over 70% of the annual precipitation, and the Northeast Monsoon (October–December) [33,34,35]. Kerala’s dense river network, comprising 44 rivers, supports its hydrological system. The Western Ghats act as an orographic barrier, intensifying monsoon rainfall and influencing large-scale atmospheric processes like integrated vapor transport and atmospheric river activity. This complex interaction between terrain, monsoon dynamics, and hydrological processes makes Kerala an ideal study area for examining extreme precipitation events, which are important for water resource management in a changing climate.

The IMD and the NCMRWF collaboratively developed a merged satellite–gauge rainfall dataset to improve monsoon rainfall analysis and numerical model validation. This dataset, known as the IMD-NCMRWF merged satellite–gauge product, integrates GPM-based near real-time multi-satellite precipitation estimates with IMD’s dense rain gauge network across India, enhancing accuracy [36,37]. This product is recognized as one of the most reliable gridded rainfall datasets for the Indian region, and the IMD-NCMRWF merged satellite–gauge product is widely utilized in hydro-meteorological research and forecasting applications [38,39,40]. Studies have demonstrated its effectiveness in capturing tropical cyclone (TC) rainfall over India, making it a valuable resource for extreme weather studies [41]. For this study, the gridded rainfall used to validate NWP model rainfall forecasts is the IMD-NCMRWF merged (Satellite + Gauge) dataset for the monsoon season (June–September) from 2018 to 2022, available at a 0.25° × 0.25° grid resolution.

The daily precipitation forecasts from six NWP models were downloaded from the ECMWF TIGGE Data Retrieval website. These models were the NCEP, NCMRWF, ECMWF, China Meteorological Administration (CMA) GFS, UKMO, and JMA. The forecasts cover 1-day, 2-day, and 3-day rainfall predictions, with a 0.25° grid resolution used in this study. The data span the monsoon season (June–September) from 2018 to 2022 and specifically focus on Kerala. Both the observed and past forecast datasets were taken in gridded format, covering the entire Kerala state, and were then considered for further performance evaluation. The dataset consists of a grid of latitude and longitude points ranging from 8° N to 13° N and 74° E to 78° E, with a 0.25° resolution, ensuring consistent spatial coverage across the study region.

3. Methodology

This study evaluated the accuracy of six different operational forecast models: NCEP, NCMRWF, ECMWF, CMA, UKMO, and JMA. The models’ performance was analyzed for their ability to accurately predict rainfall in Kerala during the monsoon season (June–September) from 2018 to 2022.

Table 1. NWP models used in this study.

NWP Model	Forecast Range (Days)	Organization/Center
NCEP	0–16	National Centers for Environmental Prediction (NCEP)
NCMRWF	0–10	National Centre for Medium Range Weather Forecasting (NCMRWF), India
ECMWF	0–15	European Centre for Medium-Range Weather Forecasts (ECMWF)
CMA	0–15	China Meteorological Administration (CMA)
UKMO	0–15	United Kingdom Met Office (UKMO)
JMA	0–11	Japan Meteorological Agency (JMA)

below provides details of the six operational models evaluated in this study (source: https://confluence.ecmwf.int/display/TIGGE/Models (accessed on 21 November 2024)). Model performance was evaluated using several accuracy and skill measures for day-1 to day-3 forecasts of rainfall over Kerala.

The overall methodology for this study is summarized in Figure 2. Different verification methods were used in the past to evaluate the performance of different NWP models. It is crucial not to focus solely on a particular method. Even models with an overall strong performance may not always be the best fit for every specific application or region [23,42]. The Traditional Verification Methods, Categorical Statistics Verification Methods, and Fractional Skill Score (FSS) were used in this study to assess precipitation forecast accuracy of NWP models.

3.1. Traditional Verification Methods

Traditional methods, namely, root mean square error (RMSE), anomaly correlation coefficient (ACC), mean error (ME), and relative bias (RBIAS) quantifying errors and biases, help to identify discrepancies between observed and predicted rainfall. The mathematical equations for RMSE, ACC, RBIAS, and ME are presented in

Table 2. Equations for RMSE, ACC, BIAS, and ME.

Metric	Equation
RMSE	$RMSE=\sqrt{{\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\left(O_i-F_i\right)}^2}$
ACC	ACC $={\displaystyle \frac{{\sum }_{i=1}^n\left(O_i-O_m\right)\left(F_i-F_m\right)}{\sqrt{{\sum }_{i=1}^n{\left(O_i-O_m\right)}^2{\sum }_{i=1}^n{\left(F_i-F_m\right)}^2}}}$
RBIAS	RBIAS $={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\displaystyle \frac{\left(P_i-O_i\right)\times 100}{O_i}}$
ME	ME $={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\left(P_i-O_i\right)$

. RMSE quantifies the absolute average error, giving more weight to larger discrepancies. The ACC measures the similarity of the forecasted precipitation anomalies (deviations from the mean) with the observed anomalies, indicating how well the forecast captures the patterns of precipitation, especially during extreme events [25,43,44]. RBIAS is particularly important for assessing systematic biases and identifying over- or under-prediction trends in precipitation forecasts, and it is essential for flood forecasting and model calibration [25,43,44]. The ME represents the averaged magnitude of differences between the forecasted and reference datasets.

3.2. Categorical Statistics Verification Methods

Categorical methods, namely, Probability of Detection (POD), False Alarm Rate (FAR), Critical Success Index (CSI), and Equitable Threat Score (ETS) focus on identifying rainfall events and false alarms, crucial for operational forecasting and flood preparedness [25,43,45]. These verification methods evaluate the performance of precipitation forecasts by comparing predicted and observed events using contingency tables (

Table 3. Contingency tables.

Predicted\Actual	Rainfall (Yes)	No Rainfall (No)
Rainfall (Yes)	Hits (H)	False alarms (F)
No Rainfall (No)	Misses (M)	True negatives (T)

). These methods categorize forecasts into hits, misses, false alarms, and correct negatives to calculate skill scores that assess forecast accuracy and reliability. POD measures the ability to correctly forecast rainfall events; FAR quantifies over-predictions; CSI represents the proportion of correctly forecasted rainfall events among all occurrences; and the ETS accounts for hits due to chance. Equations for calculating skill scores are presented in

Table 4. Equations for skill score calculation.

Skill Score	Equation	Purpose
Probability of Detection (POD)	$POD={\displaystyle \frac{H}{H+M}}$	Measures the proportion of actual rainfall events correctly predicted
False Alarm Rate (FAR)	$FAR={\displaystyle \frac{F}{H+F}}$	Measures the proportion of predicted rainfall events that were false alarms
Equitable Threat Score (ETS)	$ETS={\displaystyle \frac{H-\mathit{Expected} \mathit{Hits}}{H+F+M-\mathit{Expected} \mathit{Hits}}}$	Evaluates the forecast accuracy while accounting for random chance
Critical Success Index (CSI)	$CSI={\displaystyle \frac{H}{H+F+M}}$	Measures the proportion of correct predictions among all possible outcomes

. These methods typically require a threshold (1 mm/day) to classify events as rain or no-rain, ensuring consistency in verification [25,39,43,45,46,47,48,49,50,51,52].

Here, Hits (H) represents correctly predicted rainfall events; False Alarms (F) indicates instances where rainfall was forecasted but did not occur; Misses (M) refers to observed rainfall events that were not predicted by the forecast; and True negatives (T) denote cases where no rainfall was observed, and the forecast correctly predicted no rainfall.

3.3. Fractional Skill Score (FSS)

Fractional Skill Score (FSS) assesses how well the forecasted rainfall matches the observed rainfall over a specified threshold and spatial domain, valuable for extreme weather and localized rainfall predictions [46,47]. The evaluation focused on three rainfall thresholds (5.0 mm, 50.0 mm, and 100.0 mm). The Fractional Brier Score (FBS) and Worst Possible Fractional Brier Score ${\mathrm{F}\mathrm{B}\mathrm{S}}_{\mathrm{W}\mathrm{O}\mathrm{R}\mathrm{S}\mathrm{T}}$ were computed for each mode and threshold by comparing forecasted and observed rainfall. The Fractional Skill Score (FSS) was then derived to assess model accuracy, with higher FSS indicating better performance. The FSS values range from 0 (for the worst forecast) to 1 for the perfect forecast [46,47,53,54,55].

$$\mathrm{F}\mathrm{S}\mathrm{S}=1-{\displaystyle \frac{\mathrm{F}\mathrm{B}\mathrm{S}}{{\mathrm{F}\mathrm{B}\mathrm{S}}_{\mathrm{W}\mathrm{O}\mathrm{R}\mathrm{S}\mathrm{T}}}}$$

(1)

$$\mathrm{FBS}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^n{\left(O_i-F_i\right)}^2$$

(2)

where $F_i$ and $O_i$ are the forecast and observed fractions, respectively, at each point i and have values between 0 and 1, and N is the number of pixels in the verification area.

$${FBS}_{wWORST}={\displaystyle \frac{1}{N}}\left[{\sum }_{i=1}^N{O_i}^2+{\sum }_{i=1}^N{F_i}^2\right]$$

(3)

4. Results and Discussion

4.1. Traditional Verification Methods

The analysis of forecast performance reveals several key trends and insights across the NWP models. Key metrics such as anomaly correlation coefficient (ACC), root mean square error (RMSE), mean error (ME), and relative bias (RBIAS) are used to evaluate each model’s ability to predict rainfall accurately (Figure 3). Figure 3 compares forecast performance across NWP models using these metrics. A key observation from Figure 3 is that the systematic decline in forecast skill with increasing lead time, reflected in decreasing ACC values and rising RMSE across all models from Day 1 to Day 3. Several studies have reported similar findings, emphasizing the challenges of maintaining forecast reliability as lead time increases [6,25,27,56,57,58]. It may be due to the errors in initial conditions, assumptions in model physical process, and the chaotic nature of the atmosphere, which increase over time, leading to diminished accuracy and reduced agreement between predicted and observed rainfall as lead time increases. Among the models, JMA excels in short-term forecasting, achieving the highest ACC and lowest RMSE, which agrees with the findings of Singhal et al. [27]. ECMWF demonstrates strong performance, though its accuracy declines slightly over time. The first-day average ACC for NCEP is approximately 0.34, which aligns with similar studies conducted in the region [6,25]. An ACC greater than 0.3 is regarded as a reliable indicator of precipitation forecast accuracy [24]. NCMRWF shows lower ACC and higher RMSE, indicating poor forecast skill. NCEP and UKMO show moderate performance considering all the three days of the forecast.

A positive RBIAS indicates overestimation (forecast is higher than observed). All models generally over-predicted precipitation, indicated by their positive RBIAS values. A possible reason for the overprediction of rainfall could be the model overestimating the area experiencing wet conditions, resulting in rainfall being predicted over a larger region than what actually occurred [59,60,61]. Additionally, a significant bias was evident in all models over the Arabian Sea near the Indian west coast [23]. ECMWF had the lowest RBIAS values. NCEP and UKMO showed moderate errors [61]. CMA exhibited the highest RBIAS, indicating significant forecast inaccuracies.

Among the models, JMA performed better, achieving the highest ACC values on all three days (0.394 on Day 1), indicating its superior ability to capture precipitation patterns, which agrees with earlier studies [27,28]. It also achieved the lowest RMSE across all days (14.73 on Day 1 and 15.84 on Day 3), reflecting minimal error magnitudes. While its ME values remained low, highlighting accurate predictions, JMA’s RBIAS, though moderate (38.92 on Day 1, improving to 29.13 on Day 3), suggests some room for bias correction. ECMWF demonstrated superior consistency, making it more reliable for predictions. NCEP and UKMO displayed moderate errors, while CMA and NCMRWF struggled to maintain forecast quality.

Table 5. Ranking of NWP models for the 1st day forecast.

Day 1 Rankings
Model	ACC Rank	RMSE Rank	ME Rank	RBIAS Rank
CMA	6	5	5	4
ECMWF	2	2	2	2
JMA	1	1	1	3
NCEP	4	4	4	6
NCMRWF	3	6	6	5
UKMO	5	3	3	1

,

Table 6. Ranking of NWP models for the 2nd day forecast.

Day 2 Rankings
Model	ACC Rank	RMSE Rank	ME Rank	RBIAS Rank
CMA	6	5	5	6
ECMWF	2	2	2	1
JMA	1	1	1	3
NCEP	4	3	4	4
NCMRWF	3	6	6	5
UKMO	5	4	3	2

and

Table 7. Ranking of NWP models for the 3rd day forecast.

Day 3 Rankings
Model	ACC Rank	RMSE Rank	ME Rank	RBIAS Rank
CMA	4	4	5	6
ECMWF	3	2	1	1
JMA	1	1	2	3
NCEP	2	3	4	4
NCMRWF	5	6	6	5
UKMO	6	5	3	2

present the comparative rankings of NWP models for Day 1, Day 2, and Day 3 forecasts, respectively, based on key performance metrics.

4.2. Categorical Statistics Verification Methods

The performance of NWP models in categorical precipitation verification metrics Probability of Detection (POD), False Alarm Ratio (FAR), Critical Success Index (CSI), and Equitable Threat Score (ETS) for Days 1, 2, and 3 revealed critical insights into their forecasting skill. Figure 4 compares categorical precipitation verification metrics across different NWP models, highlighting variations in performance.

Across all models, a systematic decrease in POD, CSI, and ETS was observed from Day 1 to Day 3, which aligns with the expected decline in forecast skill with increasing lead time due to the chaotic nature of atmospheric dynamics and accumulating uncertainties in the models [24,25,27,49,58]. FAR remained relatively consistent for most models, indicating that false alarms persist at similar levels regardless of lead time, reflecting challenges in refining detection thresholds [27]. ECMWF and JMA showed high POD, CSI, and ETS, which indicated good extreme event detection. The ECMWF model demonstrated a more balanced performance, with consistently high POD values coupled with relatively moderate FAR (0.41 on all days). This balance resulted in competitive CSI scores and ETS scores, which were the highest among all models [27,58,61]. The relatively smaller decline in ETS compared to other models’ further underscores ECMWF’s robustness over increasing lead times. JMA had the highest FAR (0.45–0.47), which reveals that it may predict false rainfall events. The overall CSI value across all models ranged from 0.49 to 0.57 over the three-day period, which is aligned with other studies [25].

A few studies showed a POD above 0.8 for CMA, indicating their strong capability in detecting rainfall events [24,25]. Conversely, in this study, the CMA model consistently underperformed, with the lowest POD values, suggesting limited skill in detecting precipitation events. Its FAR values were comparable to other models, but its CSI and ETS scores were the lowest across all days. CMA faced significant challenges across all metrics, underscoring the need for improvements in its model physics and assimilation methods. The challenges were further compounded in regions with complex topography, where models often fail to capture localized extreme weather events. Similar issues have been reported in other studies conducted in comparable geographical regions [44,61,62,63]. The findings highlight the inherent trade-offs in precipitation forecasting, where achieving a balance between detection, accuracy, and reliability is key to improving forecast skill.

This analysis highlights the variability in model performance, emphasizing the use of data assimilation schemes to forecast inaccuracies. Since each model employs a unique assimilation technique, their effectiveness in representing key atmospheric processes differs. This indicates that the reliability of extreme event predictions depends on how well a model integrates observational data and captures essential physical mechanisms [44,63,64,65]. The poor verification scores, combined with a high False Alarm Ratio (FAR exceeding 0.5), indicate significant challenges in existing microphysics schemes when simulating warm-rain extreme events in weakly forced synoptic conditions. These shortcomings emphasize the necessity for improvements in the representation of cloud and drizzle auto conversion as well as accretion processes [50,66]. The sources of errors in each model vary based on specific weather conditions, influencing their ability to forecast precipitation accurately. For stratiform rainfall, errors often arise from inaccuracies in cloud microphysics and large-scale dynamics, leading to misrepresentation of precipitation intensity and spatial distribution. In the case of convective rainfall, deficiencies in convection schemes can result in poor simulation of localized thunderstorms, causing underprediction or false alarms. Orographic rainfall, which is strongly influenced by terrain interactions, presents challenges due to coarse model resolution and inadequate representation of topographically induced lifting. Seasonal variations also contribute to model biases, as monsoon dynamics, moisture availability, and large-scale circulation patterns affect precipitation predictability. Addressing these errors requires improvements in physical parameterizations, higher-resolution models, and better data assimilation techniques to enhance rainfall forecasting accuracy across different weather conditions.

4.3. Fractional Skill Score (FSS)

Extreme rainfall events can significantly impact model performance. This effect is assessed by computing skill scores across various rainfall thresholds [58]. FSS is calculated using Equations (1)–(3) to evaluate the performance of the six NWP models for rainfall forecasting. The Fractional Skill Score (FSS) values for the first, second, and third forecast days were compared across the models for three rainfall thresholds (5.0 mm, 50.0 mm, and 100.0 mm). Figure 5 illustrates the comparison of FSS across NWP models for three thresholds.

FSS measures the model’s ability to match observed data, with values closer to 1 indicating better skill. In general, an FSS value greater than 0.5 is typically considered a reliable indicator of a forecast’s effectiveness [55]. As the forecast period extends, FSS decreases, reflecting a loss of forecast skill over time. This decline is due to increased uncertainty and model limitations in longer forecast horizons. The decline in FSS over time is due to atmospheric instability, where small initial errors grow, making long-term forecasts less accurate. Lower model resolution and boundary condition error also contribute, as they become more significant with time. Additionally, parameterization of physical processes, like convection, increases uncertainty in longer-range forecasts, reducing model skill. There was a decrease in FSS as the rainfall threshold increased. This pattern was evident across all models in the data, where FSS values were higher at lower thresholds and decreased for higher thresholds [23]. This trend reflects the general difficulty of predicting heavy rainfall events accurately compared to lighter ones, as the complexity of extreme events reduces model performance. Overall, NCEP and UKMO models showed higher forecast skill (FSS), especially for short-term forecasts and lower thresholds. ECMWF and JMA models, however, exhibited a significant decline in forecast skill, particularly for higher thresholds and longer periods [23]. This is due to the inherent unpredictability of atmospheric systems, where small errors grow over time, and model resolution limitations that affect long-term accuracy. The findings highlight the importance of considering both the forecast period and threshold when choosing a model.

As the rainfall threshold decreased, models became more aligned in predicting light to moderate rainfall, since they rely on large-scale atmospheric patterns that are similar across models. This results in FSS values converging, suggesting that river managers can expect consistent forecast dynamics for storms that require less precision in capturing small-scale details [53]. However, at higher thresholds (e.g., 50, 100 mm), small-scale processes like convection and orography become more important, leading to variations in model performance and divergence in FSS values. Therefore, the models’ ability to predict extreme events differs more at higher thresholds, while they are more consistent in forecasting lower-intensity rainfall.

5. Summary

The evaluation of six NWP models, namely, NCEP, NCMRWF, ECMWF, CMA, UKMO, and JMA, reinforces the importance of accurate precipitation forecasts. The analysis of NWP model performance across various metrics, including forecast accuracy, precipitation detection, and forecast skill, reveals key insights into the capabilities and limitations of different models as forecast lead time and rainfall thresholds increase. While all models captured rainfall patterns well, they struggled with accurately forecasting heavy rainfall events, particularly as forecast lead times and rainfall thresholds increased. Heavy precipitation events, especially those influenced by complex terrains such as the Western Ghats along India’s west coast, significantly impact model performance. The interaction between the mountain range and moisture-laden low-level westerly winds from the Arabian Sea enhances rainfall in this region, where the intensity of the large-scale monsoon flow influences both low-level wind patterns and the associated precipitation. This region is crucial for monsoon studies due to its significant role in modulating rainfall patterns.

All models showed declining accuracy as the forecast lead time increased. This pattern is a commonly observed feature of NWP models, which can be attributed to the complex and unpredictable nature of atmospheric processes and the growing uncertainty in the initial conditions as the forecast horizon lengthens. By analyzing traditional performance and categorical precipitation metrics, it was observed that JMA performed best but suffered from false alarms, while ECMWF demonstrated strong overall performance in different metrics. NCEP and UKMO had moderate performance, while CMA and NCMRWF showed very poor performance.

For rainfall thresholds, all models faced challenges as the threshold increased, particularly for extreme rainfall events. JMA and ECMWF showed better performance in traditional and categorical precipitation metrics, while analyzing FSS, ECMWF, and JMA showed a more significant decline in forecast skill as the rainfall threshold increased, particularly for extreme rainfall events.

Incorporating higher-resolution models and advanced data assimilation techniques can significantly improve the accuracy of extreme rainfall forecasts. High-resolution models capture finer-scale atmospheric processes and better represent local terrain influences, which is crucial for regions like Kerala with complex topography. Additionally, data assimilation techniques, such as four-dimensional variational (4D-Var) and ensemble Kalman filtering (EnKF), enhance initial conditions by integrating real-time observational data, leading to more accurate precipitation predictions. Combining these approaches with improved parameterization of convective processes and land–atmosphere interactions can help mitigate model biases and improve performance at higher rainfall thresholds.

Multi-Model Ensembles (MME) improve precipitation forecasting by combining outputs from multiple models to reduce errors and enhance reliability. One key approach is bias correction, where statistical methods like quantile mapping and Bayesian model averaging adjust systematic errors in individual models, leading to more accurate predictions, particularly for extreme rainfall events. Additionally, weighted averaging techniques assign weights to different models based on their past performance. MME also enhance spatial and temporal coverage by blending models that perform well under different conditions, which is particularly beneficial for regions with complex topography like Kerala. Furthermore, machine learning techniques, such as neural networks and random forests, can be employed to intelligently merge model outputs by identifying nonlinear relationships between model biases and observed rainfall patterns. Given Kerala’s highly localized and topographically influenced rainfall patterns, object-based verification methods face challenges in accurately capturing small-scale convective systems and extreme precipitation events. However, integrating object-based evaluation in future studies, along with grid-based assessments, could help provide a more comprehensive assessment by improving the analysis of spatial displacement and structural errors in precipitation forecasts [67].

6. Conclusions

The analysis of forecast performance across multiple NWP models emphasizes the inherent trade-offs involved in precipitation forecasting. JMA and ECMWF performed best, followed by NCEP and UKMO, while CMA and NCMRWF had the lowest accuracy due to higher biases. JMA excelled in short-term detection but struggled with false alarms. ECMWF showed superior consistency and was more reliable for medium-range forecast but showed a greater decline in performance for higher thresholds and longer forecast. NCEP and UKMO were effective for short-term forecasts, particularly for lower rainfall thresholds, but their skill diminished for more extreme events. CMA and NCMRWF faced challenges in precipitation detection and accurate forecasting, with higher biases and lower skill scores across all metrics. These findings underscore the need for selecting models based on the specific forecast horizon, threshold, and required reliability, with ECMWF and JMA being better suited for short- to medium-range forecasts and NCEP and UKMO showing more robustness for lower thresholds and shorter forecast periods. The results highlight the complexity of precipitation forecasting, where achieving a balance between detection, accuracy, and reliability is crucial for improving overall model performance. Using higher-resolution models improves local terrain representation, while data assimilation techniques like 4D-Var and EnKF enhance initial conditions, leading to better extreme rainfall forecasts.