3.2. Comparative Analysis of Nowcasting Performance
To evaluate the performance of EGN-Nowcast in handling complex precipitation evolution, a cross-generational comparison is conducted between EGN-Nowcast and representative models spanning multiple generations within the current field of precipitation nowcasting. The full spectrum, ranging from classic physical extrapolation to cutting-edge large models, is covered by the selected baselines: First, PySTEPS, a traditional physical algorithm based on the optical flow method, is included; second, ConvLSTM, regarded as a foundational work in deep learning spatiotemporal prediction, is selected to represent the performance benchmark of early Recurrent Neural Networks; furthermore, TECO [
31], a video generation model possessing strong spatiotemporal coherence modeling capabilities, is chosen; finally, variants of spatiotemporal large models representing the current SOTA are included, namely Nuwä-EVL [
32]—based on multimodal evolution—along with NowcastingGPT and NGE [
33]. Here, NGE denotes the NowcastingGPT-EVL with the generative evolution setting, which is used as the strongest generative baseline in our comparison.
To improve the reproducibility and fairness of the baseline comparison, all learning-based baselines were evaluated under the same KNMI experimental protocol, including identical preprocessing pipeline, chronological train/validation/test split, input–output configuration, spatial resolution, temporal sampling setting, and evaluation procedure. The architectures and capacity settings of external baselines were based on their original papers or publicly available implementations, with necessary adaptations only for the unified KNMI input–output setting. The optimization configurations, including optimizer, learning rate, batch size, and checkpoint selection strategy, followed the corresponding baseline implementations and were selected based on validation performance under the KNMI validation set. The independent 2025 test set was used only for final performance reporting, and a fixed random seed was used for data loading and model initialization in the reported comparison. PySTEPS was not trained because it is an optical-flow-based extrapolation method, but it was configured and evaluated under the same input–output and verification protocol. Since large generative baselines, including TECO, Nuwä-EVL, NowcastingGPT, and NGE, were evaluated without their original external pretraining data or pretrained weights, the results should be interpreted as a controlled comparison under the KNMI radar setting rather than a universal comparison across all possible pretrained configurations.
By benchmarking against baselines spanning physical extrapolation, recurrent structures, and large-model architectures, we evaluate the relative performance of EGN-Nowcast under a unified KNMI experimental setting. The average performance of each model on the test set is summarized in
Table 2:
The quantitative evaluation in
Table 2 shows that EGN-Nowcast achieves competitive performance in both continuous error metrics and categorical precipitation-event verification. The results indicate that the proposed framework improves event detection while reducing false alarms, particularly under more stochastic precipitation conditions.
Because rare-event metrics can become unstable when the number of observed positive samples is small, we additionally report the number of observed positive pixels at each rainfall-intensity threshold and forecast lead time. For a threshold
and lead time
l, the positive-sample count is computed as
where
denotes the de-normalized observed rainfall intensity for forecast case
n, lead time
l, and grid cell
.
Table 3 shows that the number of observed positive samples decreases sharply as the threshold increases, especially at 8 mm/h. This confirms that heavy-rainfall verification is more sensitive to sampling variability than the 1 and 2 mm/h thresholds, and helps explain why the absolute CSI values at 8 mm/h remain low.
To further quantify the uncertainty of these deterministic metrics,
Table 4 reports forecast-case-level bootstrap confidence intervals for EGN-Nowcast and the strongest baseline, NGE. Different from the standard deviations reported in
Table 2, the uncertainty values in
Table 4 denote the half-width of the 95% bootstrap confidence intervals obtained from forecast-case-level resampling.
Under the moderate-rainfall threshold of 2 mm/h, EGN-Nowcast achieves a CSI of 0.158, compared with 0.120 for NGE, while reducing FAR from 0.710 to 0.590. The bootstrap uncertainty estimates in
Table 4 show non-overlapping 95% confidence ranges for POD, FAR, and CSI between EGN-Nowcast and NGE at 2 mm/h, suggesting statistically stable moderate-rainfall improvements under the current KNMI test-set distribution. This gain suggests that the event-guided spatiotemporal structure predictor helps improve precipitation localization while reducing the dominance of clear-sky background pixels during optimization.
At the more challenging 8 mm/h threshold, EGN-Nowcast achieves a CSI of 0.011 and reduces FAR from 0.52 to 0.42. However, the absolute CSI remains low, and the improvement over NGE is small in absolute terms. As shown in
Table 3, observed positive pixels at 8 mm/h are much fewer than those at the 1 and 2 mm/h thresholds, making heavy-rainfall verification more sensitive to sampling variability. The bootstrap confidence intervals in
Table 4 further show that the 95% confidence ranges of POD and CSI overlap between EGN-Nowcast and NGE, whereas the FAR intervals do not overlap. Therefore, the 8 mm/h results should be interpreted as modest but favorable improvements in false-alarm suppression and localized high-intensity structure refinement, rather than as evidence of complete pixel-level detection of rare high-intensity precipitation. Beyond the overall test-set comparison, we further examine whether this performance pattern is consistent across different seasonal precipitation regimes.
To assess seasonal robustness, we stratify the 2025 test set into summer (JJA, convective-dominant) and winter (DJF, stratiform-dominant). Summer is evaluated at 8 mm/h. Winter 8 mm/h events are too rare for meaningful statistical comparison, so we report 2 mm/h instead. Results are shown in
Table 5.
In summer, EGN-Nowcast reduces FAR from 0.54 to 0.44 while improving CSI from 0.008 to 0.010. In winter, FAR drops from 0.67 to 0.55, and CSI rises from 0.132 to 0.169. These results suggest consistent false-alarm reduction across both convective- and stratiform-dominant regimes, although the summer 8 mm/h gains remain modest because localized convective cores are sparse and sensitive to spatial displacement errors.
To further evaluate the model’s robustness and its performance decay over temporal scales, we visualize the lead-time evolution of CSI and FAR across three precipitation thresholds (1 mm/h, 2 mm/h, and 8 mm/h) in
Figure 6.
The non-monotonic FAR evolution at the 8 mm/h threshold in
Figure 6f should be interpreted cautiously. Since heavy-rainfall pixels above 8 mm/h are extremely sparse, FAR at this threshold is highly sensitive to a small number of false-alarm or missed-event pixels at individual lead times. Therefore, the observed fluctuation mainly reflects rare-event sampling variability and threshold sensitivity, rather than a stable physical trend in model behavior.
To further validate the predictive robustness of EGN-Nowcast over extended horizons,
Figure 7 presents a visual comparison across three critical stages:
T = 30 min,
T = 90 min, and
T = 120 min.
At the early stage (
T = 30 min), all models, including the operational baseline PySTEPS and the generative model NowcastingGPT, exhibit reasonable skill in capturing the initial storm structure (top row in
Figure 7). As the lead time extends to the mid-to-long range (
T = 90 and 120 min), differences among the methods become more evident. PySTEPS shows noticeable spatial displacement and a pixel-scatter effect, leading to a loss of coherent convective structure (third column of
Figure 7). Although NowcastingGPT maintains a relatively clear precipitation morphology, it tends to underestimate the peak intensity of the precipitation centers; the high-intensity precipitation cores observed in the ground truth are predicted as moderate-intensity regions (fourth column of
Figure 7). In comparison, EGN-Nowcast provides improved localization and structural representation of high-intensity precipitation regions at longer lead times (as highlighted by the white dashed ellipses in the second column of
Figure 7). This visual comparison is consistent with the quantitative results, suggesting that event-guided auxiliary supervision helps reduce localization errors, while the Conditional Diffusion Refiner contributes to the recovery of fine-scale precipitation details that are often smoothed in long-lead forecasts. To clarify the case-selection procedure, this case was selected from the independent 2025 test set based on predefined precipitation characteristics, including the occurrence of high-intensity precipitation exceeding the 8 mm/h verification threshold, sufficient precipitation-object coverage, and continuous temporal evolution during the forecast period. The selection was performed independently of model performance and was not based on case-specific verification scores or visually favorable results of EGN-Nowcast. This example is intended only as a qualitative illustration; the overall model comparison is based on the full-test-set quantitative metrics. To further illustrate the limitations of EGN-Nowcast,
Appendix A (
Figure A1) provides a representative failure case. In that case, EGN-Nowcast captures the main precipitation structure and maintains high-intensity precipitation at extended lead times, but exhibits a spatial displacement error, with the predicted convective core lagging behind the observed location. This highlights the remaining challenge of accurately predicting the location and displacement speed of rapidly evolving convective systems at longer lead times.
3.3. Ablation Study
To systematically investigate the contributions of the core components within the EGN-Nowcast framework, a series of ablation variants were designed for comparative experiments. The aim of this section is to examine the role of event-guided auxiliary supervision in high-intensity precipitation representation and the effectiveness of the Conditional Diffusion Refiner in recovering fine-scale precipitation details. We emphasize that the first-stage Transformer uses standard self-attention; therefore, the ablation does not remove a mathematically distinct region-aware attention operator, but evaluates the effect of event-guided supervision and the two-stage refinement strategy.
The following five configurations were compared on the same test set to evaluate the contributions of event-guided auxiliary supervision and the Conditional Diffusion Refiner. In this study, event-guided optimization refers to the training strategy implemented through the auxiliary classification loss, rather than an independent architectural component. (1) Baseline: A deterministic Transformer backbone without auxiliary event supervision and without the Conditional Diffusion Refiner. The deterministic backbone prediction is directly used as the final output. (2) R1: A variant in which the event-guided training strategy is removed from the first-stage predictor, and the coarse prediction is generated using only the standard reconstruction objective. This variant evaluates the influence of event-guided optimization on the deterministic precipitation representation. (3) R2 (w/o Conditional Diffusion Refiner): the first-stage structure predictor is retained, but the diffusion model in the second stage is removed, and the deterministic prediction from the first stage is directly used as the final output. (4) w/o Aux: The complete two-stage architecture is retained, while only the auxiliary classification loss is disabled by setting its weight to zero. This variant isolates the contribution of the auxiliary supervision signal within the event-guided training strategy. (5) EGN-Nowcast: The complete model combining event-guided auxiliary supervision and conditional diffusion refinement.
The performance differences in various variants on key metrics are presented in
Table 6.
The quantitative results in
Table 6 demonstrate the distinct and complementary roles of the proposed components. Compared with the Baseline, which only employs the deterministic Transformer backbone, the ablation variants reveal the contributions of event-guided auxiliary supervision and conditional diffusion refinement. The effect of event-guided auxiliary supervision varies across evaluation metrics, indicating its influence on precipitation localization and intensity representation under severe class imbalance. When the event-guided optimization strategy is removed in R1, the CSI at 2 mm/h decreases from 0.158 to 0.128, accompanied by an increase in FAR from 0.59 to 0.69. This suggests that event-guided optimization contributes to precipitation localization under background-dominated conditions. The comparison between R1 and
w/
o Aux further reveals the different roles of the event-guided training strategy and the auxiliary classification objective. R1 evaluates the effect of removing the event-guided optimization strategy from the first-stage predictor, whereas
w/
o Aux retains the two-stage forecasting framework and disables only the auxiliary classification loss. Therefore, R1 reflects the contribution of the overall event-guided training strategy, while
w/
o Aux isolates the specific effect of the auxiliary classification signal within the complete forecasting framework. Overall, the ablation results indicate that the proposed components contribute to different aspects of precipitation forecasting rather than uniformly improving all evaluation metrics.
It is worth noting that R2, which removes the Conditional Diffusion Refiner, achieves lower MSE and MAE than the Baseline but shows a lower CSI at the 8 mm/h threshold. This behavior reflects the mismatch between pixel-level average-error metrics and high-intensity precipitation verification under severe class imbalance. Since MSE and MAE are dominated by clear-sky and weak-to-moderate precipitation pixels, a deterministic regression model can reduce average errors by producing smoother and more conservative precipitation fields. However, this smoothing tendency suppresses localized high-intensity convective cores, which are critical for high-intensity precipitation representation. As a result, R2 improves global pixel-level accuracy relative to the Baseline, but loses skill in high-intensity precipitation representation compared with the full EGN-Nowcast model. This result supports the role of the Conditional Diffusion Refiner in recovering high-frequency precipitation structures that may be smoothed by deterministic prediction. It should be noted that EGN-Nowcast does not uniformly improve all categorical metrics. For example, the deterministic Baseline achieves a higher CSI at the 2 mm/h threshold than the full model. This reflects the trade-off between categorical pixel-level detection and diffusion-based structural refinement. Therefore, the contribution of EGN-Nowcast is mainly reflected in improving localized precipitation structure representation and reducing false alarms, rather than maximizing every individual verification metric.
The auxiliary-loss ablation further clarifies the role of event-aware supervision. Here, the Event-Aware Auxiliary Loss should be interpreted as a data-driven intensity-aware regularizer rather than as a physical constraint. It does not explicitly enforce mass conservation, moisture conservation, storm-motion constraints, or atmospheric governing equations. Instead, it increases the optimization emphasis on sparse high-intensity precipitation regions during training. Compared with the full EGN-Nowcast model, removing the Event-Aware Auxiliary Loss leads to moderate degradation in MSE and MAE, from 3.22 to 3.38 and from 0.58 to 0.60, respectively. However, the degradation is more evident for high-intensity precipitation verification: CSI at 8 mm/h decreases from 0.011 to 0.009, while FAR at 8 mm/h increases from 0.42 to 0.52. These results suggest that the auxiliary loss mainly provides an additional intensity-aware optimization signal for sparse high-intensity regions and contributes to reducing false alarms under high-intensity precipitation conditions, rather than uniformly improving all precipitation metrics.
Finally, the full EGN-Nowcast model combines event-guided supervised structure prediction with conditional diffusion refinement. Accurate spatiotemporal structural representations are enhanced through event-guided auxiliary supervision, while high-frequency textures and localized precipitation details are recovered via the diffusion model. Consequently, EGN-Nowcast achieves a favorable empirical trade-off among pixel-level, categorical, and high-intensity precipitation verification metrics.
3.4. Probabilistic Forecast Verification
To further evaluate the probabilistic forecasting capability of EGN-Nowcast, we conduct probabilistic verification using the Continuous Ranked Probability Score (CRPS) and a reliability diagram for heavy-precipitation events. During inference, EGN-Nowcast generates
stochastic realizations for each input sequence through multi-realization diffusion sampling. The ensemble size of
is adopted to balance probabilistic verification with inference efficiency under the current computational budget. We note that this ensemble size provides an initial estimate of predictive uncertainty, but remains limited for robust estimation of tail probabilities associated with rare heavy-rainfall events. Therefore, the probabilistic results, especially those at the 8 mm/h threshold, should be interpreted cautiously. The final deterministic forecast used for MSE, MAE, PCC, CSI, and FAR is obtained as the ensemble mean of these generated samples:
where
denotes the
n-th stochastic prediction. Before deterministic verification, the ensemble-mean prediction is converted back to the physical rainfall-intensity domain for comparison with the ground-truth rainfall fields. For event-based probabilistic verification, the exceedance probability at rainfall threshold
is computed as the fraction of generated samples exceeding
:
where
i denotes a spatial grid point.
The overall predictive distribution is evaluated using the Continuous Ranked Probability Score (CRPS). For an ensemble predictive distribution
F and observation
y, CRPS is defined as
For an ensemble forecast with
N members, the empirical CRPS is computed as
CRPS is computed on the normalized predictive distribution and averaged over all spatial locations and forecast lead times; therefore, the reported CRPS is dimensionless, and lower values indicate better probabilistic forecast performance. Through multi-realization stochastic sampling, EGN-Nowcast achieves a lower mean CRPS of 0.142 than NowcastingGPT (0.168), indicating improved predictive distribution quality. This result suggests that the conditional diffusion refinement stage contributes not only to localized spatial texture recovery but also to probabilistic forecast skill. Since deterministic regression variants do not generate multiple stochastic realizations, their probabilistic spread is not directly comparable in this CRPS evaluation.
To further examine probability calibration for heavy precipitation, we construct a reliability diagram for the precipitation ≥ 8 mm/h event, as shown in
Figure 8. The forecast probability at each grid point is computed as the fraction of
stochastic samples exceeding the threshold. The reliability curve shows a clear monotonic relationship between forecast probability and observed event frequency, indicating that the diffusion ensemble provides informative probabilistic ranking for heavy precipitation. Meanwhile, its deviation from the perfect-calibration line, with most populated bins lying above the diagonal, suggests that the raw ensemble probabilities are somewhat conservative for this rare event. The lower histogram reports the number of grid points in each forecast-probability bin on a logarithmic scale, highlighting the strong imbalance between low- and high-probability bins.
Because the reliability diagram pools grid-point probabilities across space and forecast lead times, the samples exhibit spatial and temporal dependence. Therefore, the reliability curve is interpreted as a calibration diagnostic rather than an independent-sample statistical test. For the 8 mm/h reliability diagram, because the 10-member ensemble produces exactly eleven discrete exceedance probabilities, namely for , the reliability diagram is constructed using these eleven discrete probability categories rather than continuous probability intervals. The sample counts for the categories are 1,685,290,121, 412,543, 185,312, 94,271, 52,834, 31,419, 18,922, 11,283, 6514, and 6989, respectively, and the category has a count of zero in the 2025 test set. These counts refer to all grid-point forecast-probability samples, rather than observed positive pixels. Most samples are concentrated in the lowest-probability category, confirming that the 8 mm/h reliability curve is strongly affected by class imbalance and should therefore be interpreted cautiously. The Brier Score for the 8 mm/h exceedance event was computed from the ensemble exceedance probability and the binary observed event indicator on the independent 2025 test set, yielding a value of . Given the low frequency of this rare precipitation event, the absolute Brier Score is strongly affected by the large number of non-event samples. Therefore, the climatological reference score and Brier Skill Score are calculated to provide a more meaningful probabilistic assessment.
The event climatological frequency is defined as:
where
denotes the number of observed grid points exceeding the 8 mm/h threshold, and
denotes the total number of evaluated grid-point samples across all forecast cases and lead times in the independent 2025 test set.
Therefore, BS, BSS, reliability analysis, and categorical metrics are jointly considered for evaluating probabilistic performance under severe event imbalance. As an ensemble-dispersion diagnostic, the correlation between ensemble spread and ensemble-mean absolute error is computed over forecast cases. The ensemble spread is calculated from the 10 stochastic forecast members, and the resulting spread–error correlation is , indicating that ensemble dispersion contains information related to forecast uncertainty.
To further evaluate temporal consistency, we perform an object-based trajectory analysis of high-intensity precipitation objects defined by an 8 mm/h verification threshold. Connected-component labeling is applied to binary rainfall masks generated from the predicted and observed precipitation fields. Eight-neighbor connectivity is adopted for precipitation object extraction. To reduce isolated noisy detections caused by radar uncertainty and stochastic diffusion sampling, objects with an area smaller than 5 connected pixels are removed before object matching. Since the final verification grid has a spatial resolution of approximately 2 km per pixel, this filtering corresponds to an effective minimum object area of approximately 20 km2. The object-based verification is conducted on the final forecast grid (128 × 128 pixels), which is identical to the resolution used for quantitative forecast evaluation. Predicted and observed precipitation objects are associated between consecutive forecast frames using a one-to-one centroid-based matching strategy. A predicted object is considered matched with an observed object when the centroid distance is within 10 pixels. This analysis is used to evaluate precipitation localization and structural evolution rather than as an independent object detection framework. The Mean Absolute Displacement Error (MADE) between predicted and ground-truth object centroids is used to quantify trajectory deviation. EGN-Nowcast achieves a lower MADE of 0.83 pixels/frame compared with 1.47 pixels/frame for NGE, indicating improved consistency in the spatial evolution of precipitation objects during forecast sequences.
Object Detection Rate (ODR) measures the fraction of observed heavy-rainfall objects matched by predicted objects, while area bias measures the ratio between the total area of matched predicted and observed objects. Unmatched observed objects are counted as missed detections, while unmatched predicted objects contribute to false object occurrences. Object-based metrics are reported as point estimates calculated on the independent 2025 KNMI test set. These metrics are used to evaluate precipitation-object localization and structural evolution consistency rather than to provide probabilistic uncertainty estimates.
As shown in
Table 7, EGN-Nowcast achieves higher ODR, lower MADE, and an area bias closer to 1 than NGE, suggesting modest improvements in heavy-rainfall localization and object-size representation.