Evaluation of the UMASEP-10 Version 2 Tool for Predicting All >10 MeV SEP Events of Solar Cycles 22, 23 and 24

Abstract

The prediction of solar energetic particle (SEP) events may help to improve the mitigation of adverse effects on humans and technology in space. UMASEP (University of Málaga Solar particle Event Predictor) is an empirical model scheme that predicts SEP events. This scheme is based on a dual-model approach. The first model predicts well-connected events by using an improved lag-correlation algorithm for analyzing soft X-ray (SXR) and differential proton fluxes to estimate empirically the Sun–Earth magnetic connectivity. The second model predicts poorly connected events by analyzing the evolution of differential proton fluxes. This study presents the evaluation of UMASEP-10 version 2, a tool based on the aforementioned scheme for predicting all >10 MeV SEP events, including those without associated flare. The evaluation of this tool is presented in terms of the probability of detection (POD), false alarm ratio (FAR) and average warning time (AWT). The best performance was achieved for the solar cycle 24 (i.e., 2008–2019), obtaining a POD of 91.1% (41/45), a FAR of 12.8% (6/47) and an AWT of 2 h 46 min. These results show that UMASEP-10 version 2 obtains a high POD and low FAR mainly because it is able to detect true Sun–Earth magnetic connections.

1. Introduction

Eruptive events in the solar corona are associated with the acceleration of protons and ions. These particles propagate along the interplanetary magnetic field lines, and may reach the near-Earth environment [1]. A solar energetic particle (SEP) event takes place when the integral proton flux in the near-Earth environment surpasses a threshold e.g., J (E > 10 MeV) > 10 particle flux units (1 p.f.u. = 1 particle cm⁻² s⁻¹ sr⁻¹) [2]. Forecasting SEP events helps to improve the mitigation of adverse effects on humans and technology in the near-Earth and space missions. Solar energetic particles, especially protons and heavy ions, can cause single event upsets (SEUs) and space radiation for space missions [3] and on passengers and flight crews on polar airline routes [4,5]. One of the most important objectives in space weather is the prediction of SEP events; however, it is also very important for space agencies not to manage a large false alarm ratio [2,6]. For this reason, solar radiation event forecasts should be accurate enough to take important decisions in satellite and space mission operations.

SEP event prediction models may be physics-based or empirical. They should predict two important event characteristics: the occurrence (i.e., the time at which the integral proton intensity surpasses a specific proton flux threshold) and the intensity profile. Physics-based SEP models, e.g., SEPMOD [7,8] and SOLPENCO [9,10], are better suited than empirical models to predict the SEP intensity profile due to their knowledge about the dynamics of the physical phenomena and to their integration into other models that predict solar wind evolution, CME and shock propagation, and particle injection and transport; however, the processing time required by physics-based models to make predictions is the main current limitation for these models to predict the occurrence of energetic SEP events and GLEs in real time; for this reason, we may say that, nowadays, empirical SEP event prediction models, which may make predictions shortly after analyzing real-time data, are better suited than physics-based models to make real-time SEP event occurrence predictions.

Empirical SEP event prediction models may be divided into three categories, depending on the type of input data that they analyze: solar data only (e.g., early observations of solar eruptive signatures), in situ particle data only (e.g., observations of incoming proton and electrons), or both solar and in situ data:

• Analysis of solar data only: This is the most common empirical approach, in which SEP event predictions rely on early observations of solar eruptive signatures [11,12,13,14,15,16,17,18,19,20,21,22,23,24,25];
• Analysis of in situ data only: These approaches take benefit on the early arrival of electrons and protons to predict the occurrence of SEP events. Posner [26] proposed the RELEASE model [26,27] to predict 30–50 MeV SEP events by analyzing electron data only, and Núñez [28] proposed the model UMASEP/Poorly Connected Prediction (PCP) model, which analyzes proton data only for predicting poorly connected >10 MeV SEP events;
• Analysis of both solar and in situ data: The UMASEP/Well-Connected Prediction (WCP) model correlates soft X-ray (SXR) and proton fluxes to predict the occurrence of well-connected SEP events [28,29,30]. Another approach in this category was proposed by Boubrahimi et al. [31], which trained a machine learning model with SXR and proton observations to predict >100 MeV SEP events.

Empirical approaches have also been proposed to predict the SEP intensity profile: the FORSPEF tool [32] analyzes a database with the characteristics of the parent solar flare and pre-calculated SEP characteristics (peak-flux and fluence) of hundreds of SEP events in several integral energy channels (E > 10; >30; >60; >100 MeV) [20]; the database is contrasted with the observed/predicted flare and SEP event occurrence to statistically calculate the SEP-projected characteristics (e.g., maximum of the peak flux, time of maximum of the peak flux, duration, and fluence). On the other hand, the SEPFLAREs tool [6] supports the SEP intensity profile prediction on the characteristics of the parent solar flare, the predicted shock propagation using the SARM model [33] on a static interplanetary magnetic field and the predicted peak intensity of the UMASEP-10 version 1 model [28].

This study presents the evaluation of the empirical model UMASEP-10 version 2 tool to predict the occurrence of all the >10 MeV SEP events for the period 1987–2019, that is, three solar cycles (SC). This paper is organized as follows: Section 2 presents the main components of the tool; Section 3 presents the performance forecasting results of UMASEP-10 version 2 for predicting all >10 MeV SEP events in SC 22–24, taking into account the revised times and events reported by Bain et al. [2] and a comparison with other SEP models; and Section 4 presents the conclusions.

2. Materials and Methods

UMASEP-10 version 2 is based on two prediction models: WCP2 and PCP. Section 2.1 describes WCP2 for the first time, and Section 2.3 summarizes the PCP model, which is described in detail in [28].

The prediction models developed for this study were constructed and evaluated by using 33 years of continuous 5 min soft X-ray and proton observations carried out by the GOES 07–15 satellites from 1987–2019 (i.e., solar cycles 22–24). The proton data are from seven differential energy channels of the instruments Electron, Proton, Alpha Detector (EPEAD) and the High Energy Proton and Alpha Particles Detector (HEPAD) aboard the aforementioned GOES satellites.

2.1. The WPC2 Model

The WCP2 model and its previous version (called here WCP1) predict well-connected events by making a lag-correlation of the slopes of the electromagnetic (EM) flux with the slopes of in situ particle fluxes. WCP1 used several combinations of solar EM and particle fluxes: it correlated proton fluxes with SXR fluxes [28,29,30], microwave (MW) flux density at 5 and 9 GHz [34] and EUVs [35]. Regarding in situ particle data, the WCP1 correlated near-relativistic electrons with SXR flux [36]. The WCP2 model, introduced in this paper, uses a novel approach for making the aforementioned lag-correlation.

The first step of the WCP2 is the transformation of the time series (EM and particle fluxes) to bit-based sequences. A “1” in a sequence indicates an extreme slope in the corresponding time series; a “0” indicates that no extreme slope takes place in the time series. An extreme slope is the slope (i.e., the difference between the flux at time t and the flux at the previous time step, that is, t − 5 min) that surpasses a percentage p of the maximum slope in the present sequence of size L, beyond which no influence is assumed in the well-connected SEP event to be predicted. If WCP2 encounters that there is a lag-correlation (explained below) between the EM-based sequence and each particle-based sequence, they assume that there is evidence that a Sun–Earth magnetic connection is taking place. If there is evidence of a magnetic connection and an EM flare has recently taken place (during the recent L-size interval) whose SXR peak has surpassed a threshold f, a well-connected SEP event prediction is issued. To avoid false alarms due to relatively strong slopes during periods of low solar activity, a threshold d is necessary as a minimum solar EM flux, which is the minimum value that is needed to consider it an extreme slope (i.e., a “1”). See [29] for more details about how these bit-based sequences are obtained.

The second step (i.e., the lag-correlation analysis) is carried out by WCP2 by using novel approach. While WCP1 assumed one-to-one causal links between extreme slopes, WCP2 assumes one-to-many causal links. That is, in WCP1 a solar EM-related “1” is associated with only 1 particle-related “1”. In contrast, WCP2 assumes that an EM-related “1” is related with many particle-related “1”s; that is, the new model, WCP2, assumes that an extreme EM-related slope may be associated with several extreme proton flux enhancements (i.e., that is, several bunches of protons at several velocities). As a consequence of this assumption, WCP2 builds a causal hypothesis from the recent L-size sequences and tries to estimate the maximum number of possible cause-consequence pairs, called here cause–consequence pairs. If the hypothetical maximum number of pairs is larger than a threshold n, WCP2 concludes that there is a magnetic connection and some particles are arriving along that magnetic connection.

In WCP2, a hypothetical cause–consequence pair is discovered if some criterion is met. The criterion for searching cause–consequence pairs may be summarized as follows: let us say that a subsequence i of two consecutive EM-related “1s”, separated by d_i time steps, is followed after dp time steps by a subsequence x of two consecutive particle-related “1s” separated by d_x time steps. We say that the i − x is a valid pair if d_x = d_i (see Figure 1).

In the third and final step, WCP2 checks whether the number of found cause–consequence pairs is greater than the threshold n (which provides empirical evidence of the existence of a Sun–Earth magnetic connection) and whether a flare has recently taken place (during the recent L-size interval), whose SXR peak has surpassed a threshold f. The (true) prediction of a well-connected SEP event is issued if the two conditions are met. If the two aforementioned conditions do not meet, WCP2 predicts an “all clear” situation.

2.2. The PCP Model

The goal of the PCP model [28] is to predict poorly connected SEP events. PCP does not analyze solar EM data; it makes its predictions by analyzing the evolution of very gradual rises of proton fluxes of these poorly connected events by learning from historical data. After processing the data, the algorithm automatically discovers temporal patterns in the data (if any) and creates a model that may determine which label should be given to new data based on the discovered patterns. In our case, the algorithm constructed an ensemble of regression trees [37,38,39]. Each regression tree was trained from the differential proton fluxes that took place in the beginning phases of past >10 MeV integral proton enhancements. To rapidly predict poorly connected events, the UMASEP-10 version 1 and 2 tools implement a simplified version of the resulting PCP model. For more information about the PCP model, please consult [28].

2.3. Optimization of the Performance of the UMASEP-10 Version 2 Tool

The most common metrics for measuring the performance of event predictors are the probability of detection (POD), the false alarm ratio (FAR) and the average/median warning time (AWT/MWT). These metrics, widely used for evaluating event-oriented prediction models (e.g., [2,15]) use the following variables: number of correct forecasts or hits, H (an event was predicted and one occurred); the number of false alarms, F (an event was predicted but none occurred); the number of missed events, M (no event was predicted but an event did occur); and, the number of correct nulls, N (no event was predicted and none occurred). Then, POD = H/(H + M) and FAR = F/(F + H). The warning time is the temporal distance between the time when the prediction is issued and the time when the observed target events occurs.

The input GOES SXR and proton data used during the optimization and testing phases were not cleaned or corrected, which allows the tool to better simulate real-time operations. In those periods when two GOES satellites were operative at the same time, any single-satellite-based SEP event prediction becomes a UMASEP-10 prediction. Since UMASEP-10 is mainly based on first derivatives of fluxes (i.e., slopes) rather than flux values, its forecasted results are not affected much by the accuracy of the proton observations. Regarding the SXRs, its forecasts may be affected by the accuracy of SXR observations, only if the SXR peak of the associated flare is near the f threshold.

Section 2 mentions several user-defined thresholds (L, p, n and f). The purpose of the optimization of the WCP2 algorithm is to maximize the POD and minimize the FAR. As the optimization function, we use an index called critical success index (CSI), which is a combination of POD and FAR as follows: CSI = (POD⁻¹ + (1 − FAR)⁻¹ − 1)⁻¹. CSI is a commonly used performance metric in atmospheric forecasting studies. The US National Weather Service has used this measure for decades to assess predictors because the CSI is an unbiased verification statistic appropriate for predicting rare events and for this reason is used to assess severe weather predictors [2,40]. We searched for the set of the aforementioned thresholds that produces a very high CSI, not necessarily the highest one. A CSI of 100% is the indication of an excellent predictor with POD = 100% and FAR = 0%. As a result of the model optimization of UMASEP-10 version 2.2 (the most recent version of the presented tool), the obtained values for the thresholds were: L was 7 h, p was 91%, n was 8 and f was 1 × 10⁻⁶ Watt m⁻².

The aforementioned setting of the threshold f means that the WCP2 model is able to trigger predictions of SEP events associated with ≥C1 flares. A well-connected SEP event associated with small flares (e.g., C1–C3 class flares) is not frequent. The prediction of these rare SEP events requires a successful identification of magnetic connections; otherwise, the number of false alarms would be very high. The forecasting results presented in the next section show that a better detection of magnetic connections is an important advantage to reduce the false alarm ratio while predicting well-connected SEP events associated with larger flares.

3. Results

In this section we present the forecasting results of UMASEP-10 version 2 for predicting all types of >10 MeV SEP events (i.e., including those associated with C1–C3 class flares, those whose associated flare took place behind the limb and those with no clear flare association) that took place in the period 1987–2019. These results were obtained using 33 years of continuous 5 min soft X-ray and proton data (i.e., solar cycles 22–24). During these solar cycles, 213 SEP events took place. This section compares these forecasting results with those of other tools as follows: Section 3.1 and Section 3.2 present the >10 MeV SEP events during the solar cycles 22–24 and the corresponding performance forecasting results of UMASEP-10 version 2; Section 3.3 presents a comparison of the UMASEP-10 version 1 and 2 for each solar cycle; Section 3.4 presents a comparison of UMASEP-10 version 2 with other empirical SEP event prediction models for the periods reported by the corresponding SEP modelers; and Section 3.5 shows two forecasts of SEP events associated with <C4 flares.

3.1. SEP Events for the Period 1987–2019

Table 1. Forecast results of UMASEP-10 version 2.2 (the most recent version of this tool) for each event of solar cycles 22–24 using the SWPC threshold on the GOES satellite data. Start times of SEP events of solar cycles 23 and 24 are presented according to Bain et al. [2]. The events of solar cycle 22 are extracted from the SWPC SEP event list in https://umbra.nascom.nasa.gov/SEP (accessed on 1 December 2021).

SEP Event Date 1	Time (ST)	Associated Flare Class	UMASEP-10 Version 2.2 Forecast Result 2	Warning Time 3
8 November 1987	2:00	M1	Hit	2 h 45 min
2 January 1988	23:25	X1	Hit	10 min
25 March 1988	22:25		Hit	5 min
30 June 1988	10:55	M9	Hit	5 min
26 August 1988	0:00	M2	Miss
12 October 1988	9:20	X2	Miss
8 November 1988	22:25	M3	Hit	8 h 10 min
14 November 1988	1:30	M3	Hit	30 min
17 December 1988	6:10	X1	Miss
17 December 1988	20:00	X4	Hit	13 h 40 min
4 January 1989	23:05	M4	Hit	1 h 5 min
8 March 1989	17:35	X15	Hit	9 h 5 min
17 March 1989	18:55	X6	Hit	5 min
23 March 1989	20:40	X1	Hit	10 min
11 April 1989	14:35	X3	Miss
5 May 1989	9:05	M5	Miss
6 May 1989	2:35	X2	Hit	16 h 40 min
23 May 1989	11:35		Miss
24 May 1989	7:30	M5	Hit	18 h 25 min
18 June 1989	16:50	C4	Hit	55 min
30 June 1989	6:55	M3	Miss
1 July 1989	6:55		Miss
25 July 1989	9:00	X2	Miss
12 August 1989	16:00	X2	Hit	5 min
4 September 1989	1:20	X1	Miss
12 September 1989	19:35	M5	Hit	6 h 5 min
29 September 1989	12:05	X9	Hit	5 min
6 October 1989	0:50		Miss
19 October 1989	13:05	X13	Miss
9 November 1989	2:40		Miss
15 November 1989	7:35	X3	Hit	5 min
27 November 1989	20:00	X1	Miss
30 November 1989	13:45	X2	Miss
19 March 1990	7:05	X1	Hit	35 min
29 March 1990	9:15	M4	Hit	1 h 20 min
7 April 1990	22:40	M7	Miss
11 April 1990	21:20		Hit	10 min
17 April 1990	5:00	X1	Hit	2 h 15 min
28 April 1990	10:05		Hit	10 min
21 May 1990	23:55	X5	Hit	50 min
24 May 1990	21:25	X9	Miss
28 May 1990	7:15		Hit	6 h 5 min
12 June 1990	11:40	M6	Hit	2 h 55 min
26 July 1990	17:20		Hit	10 h 45 min
1 August 1990	0:05	M4	Hit	2 h 25 min
31 January 1991	11:30	X1	Hit	3 h 50 min
25 February 1991	12:10	X1	Hit	50 min
23 March 1991	8:20	X9	Hit	25 min
29 March 1991	21:20		Miss
3 April 1991	8:15	M6	Hit	3 h 55 min
13 May 1991	3:00	M8	Hit	30 min
31 May 1991	12:25		Hit	2 h 55 min
4 June 1991	8:20	X12	Miss
14 June 1991	23:40	X12	Miss
30 June 1991	7:55	M6	Hit	5 h 45 min
7 July 1991	4:55	X1	Miss
11 July 1991	2:40	M3	Hit	1 h 45 min
11 July 1991	22:55		Miss
26 August 1991	17:40	X2	Hit	6 h 25 min
1 October 1991	17:40	M7	Miss
28 October 1991	13:00	X6	Hit	15 min
30 October 1991	7:45	X2	Hit	20 min
7 February 1992	6:45	M4	Hit	4 h 35 min
16 March 1992	8:40	M7	Hit	2 h 10 min
9 May 1992	10:05	M7	Hit	35 min
25 June 1992	20:45	X3	Hit	10 min
6 August 1992	11:45	M4	Hit	1 h 25 min
30 October 1992	19:20	X1	Hit	15 min
4 March 1993	15:05	C8	Hit	1 h 40 min
12 March 1993	20:10	M7	Hit	1 h 5 min
20 February 1994	3:00	M4	Hit	30 min
20 October 1994	0:30	M3	Hit	1 h 50 min
20 October 1995	8:25	M1	Hit	45 min
4 November 1997	8:45	X2	Hit	1 h 30 min
6 November 1997	13:05	X9	Hit	20 min
20 April 1998	14:00	M1	Hit	1 h 55 min
2 May 1998	14:20	X1	Hit	10 min
6 May 1998	8:35	X2	Hit	5 min
24 August 1998	23:55	X1	Hit	55 min
25 September 1998	0:10	M7	Hit	2 h 10 min
30 September 1998	15:25	M2	Hit	60 min
8 November 1998	2:45		Hit	13 h 5 min
14 November 1998	8:10	C1	Hit	1 h 15 min
23 January 1999	11:05	M5	Hit	12 h 10 min
24 April 1999	18:40		Hit	1 h 35 min
5 May 1999	18:20	M4	Hit	55 min
2 June 1999	2:45		Hit	2 h 10 min
4 June 1999	9:25	M3	Miss
18 February 2000	11:30	M1	Hit	1 h 15 min
4 April 2000	20:55	C9	Hit	2 h 55 min
7 June 2000	13:35	X2	Hit	2 h 50 min
10 June 2000	18:05	M5	Hit	20 min
14 July 2000	10:50	X5	Hit	22 h
22 July 2000	13:20	M3	Hit	25 min
28 July 2000	10:50		Hit	7 h
11 August 2000	16:50		Hit	1 h 25 min
12 September 2000	15:55	M1	Hit	1 h 5 min
16 October 2000	11:25	M2	Hit	3 h 5 min
26 October 2000	0:45	M2	Hit	9 h 10 min
8 November 2000	23:50	M7	Hit	10 min
24 November 2000	15:20	X2	Hit	5 h 55 min
28 January 2001	20:25	M1	Hit	2 h 30 min
29 March 2001	16:35	X1	Hit	2 h 40 min
2 April 2001	23:40	X20	Hit	10 min
10 April 2001	8:50	X2	Hit	14 h 45 min
15 April 2001	14:10	X14	Miss
18 April 2001	3:15	C2	Hit	4 h
28 April 2001	4:30	M7	Miss
7 May 2001	19:15		Hit	5 h
15 June 2001	17:50		Hit	1 h 5 min
10 August 2001	10:20	C3	Hit	1 h 45 min
16 August 2001	1:35		Hit	5 min
15 September 2001	14:35	M1	Hit	1 h 15 min
24 September 2001	12:15	X2	Hit	25 min
1 October 2001	2:55	M9	Miss
19 October 2001	22:25	X1	Hit	18 h 5 min
22 October 2001	19:10	X1	Hit	40 min
4 November 2001	17:05	X1	Hit	10 min
19 November 2001	12:30	M2	Hit	2 h 55 min
22 November 2001	23:20	M9	Hit	60 min
26 December 2001	6:23	M7	Hit	18 min
29 December 2001	5:10	X3	Hit	19 h 15 min
30 December 2001	2:45		Hit	20 h 55 min
31 December 2001	0:15		Hit	2 h 35 min
10 January 2002	20:45	C9	Hit	2 h 25 min
15 January 2002	14:35	M4	Hit	1 h 15 min
20 February 2002	7:30	M5	Hit	10 min
17 March 2002	8:20	M2	Hit	45 min
18 March 2002	13:00		Hit	4 h 25 min
20 March 2002	15:10	M1	Miss
22 March 2002	20:20	M1	Hit	1 h 40 min
17 April 2002	15:30	M2	Hit	2 h 25 min
21 April 2002	2:25	X1	Hit	30 min
22 May 2002	17:55	C5	Hit	3 h 25 min
7 July 2002	18:30	M1	Hit	4 h 55 min
16 July 2002	17:50	X3	Hit	2 h 20 min
19 July 2002	10:50		Miss
22 July 2002	6:55	X3	Hit	40 min
14 August 2002	9:00	M2	Hit	1 h 5 min
22 August 2002	4:55	M5	Hit	1 h 20 min
24 August 2002	1:40	X3	Hit	5 min
7 September 2002	4:40	C3	Hit	13 h 35 min
9 November 2002	19:20	M4	Hit	2 h 10 min
28 May 2003	23:35	X3	Hit	6 h 5 min
31 May 2003	4:40	M9	Hit	50 min
18 June 2003	20:50	M6	Hit	4 h 10 min
26 October 2003	18:25	X1	Hit	15 min
28 October 2003	12:15	X17	Hit	30 min
2 November 2003	11:05		Miss
4 November 2003	22:25	X28	Miss
21 November 2003	23:55	M5	Miss
2 December 2003	15:05	C7	Hit	25 min
11 April 2004	11:35	C9	Hit	3 h 15 min
25 July 2004	18:55	M1	Hit	1 h 30 min
13 September 2004	20:11	M4	Miss
19 September 2004	19:25	M2	Hit	50 min
1 November 2004	7:03		Hit	8 min
7 November 2004	19:10	X2	Hit	15 min
16 January 2005	2:10	X2	Hit	17 h 10 min
14 May 2005	5:25	M8	Hit	3 h 15 min
16 June 2005	22:00	M4	Hit	20 min
14 July 2005	2:45	M5	Hit	2 h 20 min
27 July 2005	23:00	M3	Hit	8 h 10 min
22 August 2005	20:40	M5	Hit	17 h 30 min
8 September 2005	2:15	X17	Hit	1 h 20 min
14 September 2005	1:00	X17	Hit	1 h 25 min
6 December 2006	15:55	X9	Miss
13 December 2006	3:10	X3	Hit	5 min
14 August 2010	12:30	C4	Hit	45 min
8 March 2011	1:20	M3	Hit	1 h 35 min
21 March 2011	19:50		Hit	10 h 40 min
7 June 2011	8:05	M2	Hit	40 min
4 August 2011	6:35	M9	Hit	1 h 55 min
9 August 2011	8:45	X6	Hit	15 min
23 September 2011	22:55	X1	Hit	14 h
23 October 2011	15:05	X1	Miss
26 November 2011	11:25	C1	Hit	15 min
23 January 2012	5:30	M8	Hit	45 min
27 January 2012	19:05	X1	Hit	10 min
7 March 2012	5:10	X5	Hit	1 h 10 min
13 March 2012	7:45	M7	Miss
13 March 2012	18:10	M7	Hit	10 min
17 May 2012	2:55	M5	Hit	50 min
27 May 2012	5:05		Hit	4 h
16 June 2012	19:55	M1	Hit	2 h 45 min
7 July 2012	4:00	X1	Hit	40 min
9 July 2012	1:30	X1	Hit	21 h 40 min
12 July 2012	18:35	X1	Hit	30 min
17 July 2012	17:15	M1	Hit	50 min
23 July 2012	15:45		Hit	4 h 50 min
1 September 2012	13:35	C8	Hit	1 h 5 min
28 September 2012	3:00	C3	Hit	1 h 20 min
16 March 2013	19:40	M1	Hit	9 h 50 min
11 April 2013	10:55	M6	Hit	1 h 15 min
15 May 2013	13:35	X1	Hit	40 min
22 May 2013	14:20	M5	Hit	15 min
23 June 2013	20:10	M2	Miss
30 September 2013	5:05		Hit	2 h 20 min
28 December 2013	21:50	C9	Hit	15 min
6 January 2014	9:15		Hit	45 min
7 January 2014	19:30	X1	Miss
20 February 2014	8:50	M3	Hit	30 min
25 February 2014	13:55	X4	Hit	2 h 10 min
18 April 2014	15:25	M7	Hit	1 h 30 min
11 September 2014	2:40	X1	Hit	1 h 45 min
18 June 2015	11:35	M1	Hit	5 h 15 min
21 June 2015	20:35	M2	Hit	5 min
26 June 2015	2:30	M7	Hit	8 h 10 min
29 October 2015	5:50		Hit	30 min
2 January 2016	4:30	M2	Hit	3 h 50 min
14 July 2017	9:00	M2	Hit	3 h 35 min
5 September 2017	0:38	M5	Hit	18 min
10 September 2017	16:45	X8	Hit	10 min

¹ NOAA defines the start of a proton event to be the first of three consecutive data points with fluxes (E > 10 MeV) greater than or equal to 10 pfu [2]. ² Hits are those SEP events forecasted with a warning time greater than or equal to one minute. Misses are those events that were not anticipated. ³ The warning time is the temporal difference between the start time of the SEP event, ST, and the time at which the forecast was issued.

lists all the >10 MeV SEP events of solar cycles 22–24 and the forecast results of UMASEP-10 version 2.2 (the last version of this tool). The start times of the events occurred during solar cycles 22 are the same listed in the SWPC SEP event list in https://umbra.nascom.nasa.gov/SEP (accessed on 1 December 2021); the start times of the events occurred during the solar cycles 23 and 24 were extracted from Bain et al. [2], which presents some minor updates of the aforementioned SEP event list. From left to right, the columns show the following:

• Start date and times (ST) of SEP events;
• Class of the associated flare, according to the aforementioned SWPC SEP event list.
• Forecast results: Hits are those SEP events forecasted with a warning time greater than or equal to one minute. Misses are those events that were not anticipated;
• The warning time is the temporal difference between the start time of the SEP event, ST, and the time at which the forecast was issued.

3.2. Performance Results of UMASEP-10 Version 2 for the Solar Cycles 22, 23 and 24

The UMASEP-10 version 2 tool described in Section 2 was optimized for maximizing the CSI of the solar cycles 23 and 24 by using all >10 MeV SEP events reported by Bain et al. [2]. The resulting model was tested with out-of-sample SEP events of the solar cycle 22 taking into account the aforementioned SWCP SEP event list. By using SC23 and SC24 data for training, we may guarantee that the model is trained with data of the highest quality, so the strengths and weakness of the model may be better observed. On the other hand, the use of deficient quality data for testing (SC22) has the advantage of better simulating the performance of the model when there are deficiencies in real-time data.

Figure 2a shows the distribution of hits and misses presented in Table 1, as well as false alarms of UMASEP version 2.2 in the years of the solar cycles 22, 23 and 24. Figure 2b presents the summary of the counters presented in Figure 2a in terms of all-type POD, FAR, AWT and CSI. All-type POD is the percentage of all >10 MeV SEP events (according to Bain et al. [2] and the SWPC SEP event list) whose occurrence were correctly predicted. The CSIs for solar cycles 23 and 24 were 76.4% and 80.4%, respectively. The CSI for solar cycle 22 was much lower, mainly due to the lower quality of the soft X-ray and proton data in terms of gaps, spikes and accuracy; however, it is important to say that the resulting out-of-sample all-type POD (68.5%) and FAR (35.1%) of SC22, although low, are similar to those obtained by UMASEP-10 version 1, which used this period (SC22) for optimizing the tool [28]. For the 3 solar cycles 22–24, the all-type POD was 82.16% (175/213), the FAR was 21.52% (48/223) and the AWT (MWT) was 3 h 15 min (1 h 20 min).

In order to observe the forecasting performance of UMASEP version 2 for periods of high solar activity (i.e., 1989–1995, 2000–2004, and 2011–2015), the CSI was 66.67%; for the rest of years (i.e., those of lower solar activity), the CSI was 68.63%, which is very similar. From the above, we may say that the performance results of UMASEP-10 version 2 (and consequently, of its sub-models WCP2 and PCP) are not very sensitive to the level of solar activity.

Figure 3 shows the CSI for each half of the three solar cycles. This figure shows that there is no noticeable difference between the forecasting performance in terms of CSI between the first and the second half within each solar cycle.

The large range of warning times in Table 1 (i.e., from 5 min to ~22 h) is the result of using two models with different prediction temporal horizons: WCP2 was designed to make predictions of well-connected events that might arrive in minutes, so their warning times may be as short as 5 min; on the other hand, PCP was designed to make predictions of very gradual SEP events whose integral proton flux could surpass the SEP threshold (i.e., 10 particle flux units) several hours after the first enhancement, so their warning times may be as large as 24 h.

Figure 4 shows the distribution of hits and misses per class range of the associated flares of all SEP events in Table 1. It is interesting to note that the worst performance is obtained predicting SEP events with no flare associations (i.e., those whose flare association was not clearly identified). The best performance was obtained predicting SEP events associated with C1–C9 flares; in this class range, UMASEP-10 version 2 predicted all >10 MeV SEP events.

3.3. Comparison with the UMASEP-10 Version 1 Tool

The UMASEP-10 version 1 [28] was developed by optimizing the data of solar cycles 22 and 23. The version 1 functioned in real time during the period 2010–2020, and its forecasts are available in NASA’s ISWA web site https://ccmc.gsfc.nasa.gov/iswa (accessed on 1 December 2021). Since this model was an operational product, in order to improve its forecast performance, the model parameters of the WCP1 and PCP models were updated every year of real-time operations, by adjusting the model thresholds (mentioned in Section 2) for making a better post-event prediction of the events of the previous year. In 2020, the last version was 1.7 and its forecasting results are presented in this section. In other words, version UMASEP-10 version 1.7 was optimized with data of three solar cycles.

Table 2. Forecasting performance of UMASEP-10 version 1.7 for predicting all >10 MeV SEP events in Bain et al. [2] for the solar cycles 23 and 24, and the events in https://umbra.nascom.nasa.gov/SEP (accessed on 1 December 2021) for the solar cycle 22.

	Number of SEP Events	All-Type POD	FAR	AWT	CSI
SC24	45	82.2% (37/45)	26.0% (13/50)	3 h 15 min	63.8%
SC23	95	82.1% (78/95)	19.6% (19/97)	4 h 41 min	68.4%
SC22	73	72.6% (53/73)	33.8% (27/80)	4 h 34 min	53.0%

shows the resulting performance of UMASEP-10 version 1.7 after processing 33 years of 5 min data. Regarding the CSI obtained for the solar cycles 23 and 24, UMASEP-10 version 2 was better (80.4% and 76.4%, respectively) than the CSI of UMASEP-10 version 1 (63.8% and 68.4%, respectively). The CSI obtained for the solar cycle 22 of both tools were similar (50.0% for the version 2 and 53.0% for the version 1). As a conclusion to this subsection, we may say that the new model UMASEP-10 version 2.0, which was trained with two solar cycles, obtained a better overall performance than the model UMASEP-10 version 1.7, which was trained with three solar cycles.

3.4. Comparison with Other Models for Predicting All >10 MeV SEP Events

Humans and equipment in space may be affected by any of the >10 MeV SEPs, so space weather users would need to know the POD for predicting all types of >10 MeV SEP models (i.e., all-type POD). The prediction of every single SEP event may be used to make important decisions to improve the mitigation of its adverse effects [3,4,5]. For this reason,

Table 3. Comparison of UMASEP-10 version 2.2 (the most recent version of this tool) with: (a) Protons, (b) PPS, (c) ESPERTA, (d) ESPERTA/SMOTE and (e) UMASOD tools.

(a)
	SWPC SEP Events (1986–2004)
	All-Type POD 1	FAR
PROTONS	57% 2	55%
UMASEP-10 7	78.34% (120/157)	29.71% (52/175)
(b)
	SWPC SEP Events (1997–2001)
	All-Type POD 1	FAR
PPS	40% (18/42) 3	50% (18/36)
UMASEP-10 7	89.58% (43/48)	21.82% (12/55)
(c)
	SWPC SEP Events (2006–2014)
	All-Type POD 1	FAR
ESPERTA	52.77% (19/36) 4	29.62% (8/27)
UMASEP-10 7	91.04% (33/36)	8.33% (3/36)
(d)
	SWPC SEP Events (1995–2017)
	All-Type POD 1	FAR
ESPERTA/SMOTE	55.8% (77/138) 5	39% (~49/126) 5
UMASEP-10 7	89.13% (123/138)	17.45% (26/149)
(e)
	SWPC SEP Events (1997–2014)
	All-Type POD 1	FAR
UMASOD	55.3% (73/132) 6	40.2% (49/122)
UMASEP-10 7	88.64% (117/132)	13.97% (19/136)

¹ All-type POD is the percentage of all >10 MeV SEP events (according to Bain et al. [2] and the SWPC SEP event list) whose occurrence were correctly predicted. ² The reported all-type POD and FAR of PROTONS correspond to the performances of the automatic empirical system at SWPC. The final yes/no predictions of SWPC NOAA are made by a human expert. ³ Although the PPS system runs with >M5 flares, Kahler et al. [14] reported the all-type hits and all-type misses counters (for all sizes of flares); for this reason, we show their all-type POD. ⁴ During 2006–2014, 36 >10 MeV SEP events took place according to Bain et al. [2]. Since Alberti et al. [11] predicted 19 events, the all-type POD was 52.77% (19/36). ⁵ During January 1995–April 2017, 138 >10 MeV SEP events took place according to Bain et al. [2] and the SWPC SEP event list. Since Stumpo et al. [24] reported a POD of 83% of a dataset of 92 events associated with ≥M2 flares, the number of predicted events was 77 events, and therefore, the all-type POD was 55.8% (77/138). ⁶ During January 1997–2014, 132 >10 MeV SEP events took place according to Bain et al. [2]. Since the UMASOD model [18] predicted 77 events, the all-type POD was 55.3% (77/132). ⁷ UMASEP-10 version 2.2 is the last version of the tool explained in this paper. The version 2.0 version had obtained a higher FAR.

shows a comparison of forecasting performance in terms of all-type POD and FAR of UMASEP-10 version 2.2 (the most recent version of this tool) and PROTONS [12], PPS [22], ESPERTA [11,15], ESPERTA/SMOTE [24] and UMASOD [18] for the periods reported in the corresponding studies.

Regarding the AWT (MWT), the PROTONS, PPS, ESPERTA, ESPERTA/SMOTE and UMASOD tools provide a very satisfactory AWT in the range 7–9.5 h (2–6 h). The AWT (MWT) of UMASEP-10 version 2 is in range 2.5–4 h (1 h 10 min–1 h 30 min), which is much lower (i.e., worse). The UMASEP-10 component models (i.e., WCP2 and PCP) require processing proton fluxes, whose first enhancements are observed tens of minutes or hours after the flare peak. Therefore, the drawback of the UMASEP-based models is the delay for waiting for the first observations of proton enhancements, which negatively affect the warning times.

From all the above, we conclude that UMASEP-10 version 2 is better in terms of CSI and worse in terms of AWT than the PROTONS, PPS, ESPERTA, ESPERTA/SMOTE and UMASOD tools.

3.5. Forecasts of SEP Events Associated with <C4 Flares

During the period 1987–2019, 82.6% (176/213) of the >100 MeV SEP events were associated with ≥C4 class flares, according with the flare associations in the aforementioned SWPC SEP list. The prediction of a well-connected SEP event associated with a <C4 flare requires an accurate identification of a magnetic connection. Note that the false alarm ratio of UMASEP-10 version 2 is very low (e.g., 12.4% for SC24); if the empirical magnetic connection were wrongly estimated by UMASEP version 2, the use of C1–C3 class flares would have triggered a very high number of false alarms. This section shows some specific examples of well-connected SEP event predictions that are associated with <C4 flares; these specific SEP events are missed by all prediction models that rely on solar data.

Figure 5 and Figure 6 show the forecast graphical output of the UMASEP-10 version 2 tool. The upper time series of these figures shows the observed >10 MeV integral proton flux. The current flux is indicated below the label “now” in each image. To the right of this label, the forecasted integral proton flux is presented. The yellow/orange-colored band indicates the expected evolution of the integral proton flux derived from the prediction of the proton flux. The central time series displays the SXR flux, and the lower time series shows the magnetic connectivity estimation with the best-connected CME/flare process zone. When a forecast is issued, the graphical output also shows the inferences about the associated flare, heliolongitude and active region.

Figure 5a,b present two successful well-connected SEP event prediction triggered by a C3 and C2.2 flares, respectively. Figure 6 presents a SEP event that is associated to a behind-the-limb C1.1 flare that was successfully predicted by the PCP model (which is not triggered by flares) of the UMASEP-10 version 2.

4. Conclusions

This paper presents the UMASEP-10 version 2 tool. This tool makes predictions of the time interval within which the >10 MeV proton flux is expected to surpass 10 particle cm⁻² sr⁻¹ s⁻¹ (i.e., the SWPC threshold [2]).

The UMASEP-10 version 2 tool has two component models: WCP2, which is an improved well-connected prediction model that makes an empirical detection of the Sun–Earth magnetic connectivity by correlating GOES SXR fluxes with differential proton fluxes and is able to make predictions from ≥C1 flares; and, PCP, which tries to predict poorly connected events by analyzing the evolution of gradual proton flux enhancements.

The UMASEP-10 version 2 tool was optimized for maximizing the CSI of the solar cycles 23 and 24 by using all >10 MeV SEP events reported by Bain et al. [2]. The resulting model was tested with the SEP events of the solar cycle 22. By using SC23 and SC24 data for training, we may guarantee that the model is trained with data of the highest quality, so the strengths and weakness of the model may be better observed. The use of deficient quality data for testing (SC22) allowed us to better simulate the performance of the model when there are deficiencies in real-time data.

For the three solar cycles, the all-type POD was 82.16% (175/213), the FAR was 21.52% (48/223) and the AWT was 3 h 15 min. The best performance of UMASEP-10 version 2 was achieved for the solar cycle 24 (i.e., 2008–2019), obtaining an all-type POD of 91.1% (41/45), a FAR of 12.8% (6/47) and an AWT of 2 h 46 min.

The ability for predicting well-connected SEP events associated with <C4 flares provides a few additional hits and allows making predictions a few minutes before the flare reaches the peak. Since the SEP events associated with small flares (e.g., C1–C3 class flares) are not frequent, this ability does not seem to represent an important advantage; however, this goal requires a successful identification of magnetic connections; otherwise, the number of false alarms would be very high. This challenge forced us to modify the well-connect prediction model (WCP2) to better detect magnetic connections, which is an important advantage to reduce the false alarm ratio while predicting well-connected SEP events associated with larger flares.

In order to observe the forecasting performance of UMASEP version 2 for periods of high solar activity (i.e., 1989–1995, 2000–2004 and 2011–2015), the CSI was 66.67%; for the other years (i.e., those of lower solar activity), the CSI was 68.63%, which is very similar. From these results, we can affirm that UMASEP-10 version 2 is not very sensitive to the level of solar activity.

Figure 5 shows two examples of well-connected SEP event predictions associated with C1–C3 flares, which are missed events by all the current models that rely on solar data (see Section 1), including UMASEP-10 version 1, which requires ≥C4 flares to trigger well-connected event predictions.

The worst prediction performance of UMASEP version 2 was obtained predicting SEP events with no flare associations (i.e., those whose flare association was not clearly identified). The best performance was obtained predicting SEP events associated with C1–C9 flares. In this class range, UMASEP-10 version 2 predicted all >10 MeV SEP events.

This study compares this tool with its previous version [28], PROTONS [12], PPS [22], ESPERTA [11,15], ESPERTA/SMOTE [24] and UMASOD [18], for predicting all >10 MeV events. The prediction of every single SEP event may be used to make important decisions to improve the mitigation of its adverse effects. For this reason, this study makes the comparison in terms of all-type POD, FAR and CSI. In this study, we conclude that UMASEP-10 version 2 is better than these models in terms of CSI, and worse than the aforementioned tools in terms of AWT.

These results show that UMASEP-10 version 2 obtains a high all-type POD and very low FAR, mainly because it is able to better detect true Sun–Earth magnetic connections than current SEP event prediction tools.