A Proton Flux Prediction Method Based on an Attention Mechanism and Long Short-Term Memory Network

Round 1

Reviewer 1 Report (Previous Reviewer 2)

Comments and Suggestions for Authors

Dear Editor,

 

I have reviewed the first revision of the manuscript

 

“A proton Flux Prediction Method Based on Attention Mechanism and Long Short-Term Memory Network”

 

By Zhang et al.

 

The authors have removed the seemingly accidental suggestion that the model predicts space weather events, thus removing the major problem that I considered in the first version.

 

The authors have also improved the figures following suggestions, making the discussed results easier to evaluate. Discussion of the results and description of the cases has also improved.

 

In the light of being able to see the results more clearly, I can now comment on the results and their interpretation.

 

The authors say in the original and revised manuscript that the “model can effectively predict the trends in proton flux” in the non-SAA regions (line 423). However, the Figure 11 (a) demonstrates a problem in this interpretation: In relative terms, the TPA model (red curve) underestimates the observations (black curve) by up to 2 orders of magnitude (for example between peaks 1 and 2). This is seen also in Figure 12, where 5% trend lines are shown in panel (a): At low intensities, the predicted points seem to have a spread significantly wider than the 5% trends. Thus, in relative terms, the model actually performs worse at the non-SAA areas than in the SAA areas.

 

I believe that the issue is in the metrics: For one, the MAE is defined in absolute flux terms. This means that a larger flux with similar relative error will result in substantially larger MAE, as the absolute values are 3-4 orders of magnitude higher. As a direct consequence, the MAE values in SAA are much larger than in the non-SAA areas. The fact that the difference is only about 1 order of magnitude for both TPA-LSTM and AP8 when the fluxes are 3-4 orders of magnitude larger indicates that the model actually performs better in SAA than in non-SAA.

 

I am not sure how this reflects to the R^2, however it might be a good idea to consider metrics better suited to data that varies several orders of magnitude. These issues should also be discussed in the manuscript.

 

Other issues:

----------------

 

- Figure 12 needs improving: the points are almost invisible, and there is a lot of empty space between the panels, so the panels could be made much larger. This is the main result of the manuscript, it should be more clearly shown.

- Add orbital period to table 1

- Line 151: “26 distinct cases” is not clear, please refer to for example “26 distinct time periods”. You should also state the duration of each period, and whether all “cases” have a SAA crossing, or if you had cases with no SAA crossings. In short: what is the selection criterion of the cases?

- Lines 447-455: there are several errors in writing about the MAE value: mostly you should say “increases” when you say “decreases”.

- “Certain degree”, “certain predictive performance” can still be found on lines 106, 450, 468

- Table A1: the date range format is unclear, please change to the one used in Table A2: YYYY/MM/DD. Also, the “No” and “Case No” columns are the same, drop one.

- Table A2: Are you really defining the window duration with 3 decimal accuracy? That is, with 0.06-second resolution? What resolution is the data given at?

- line 412: Reference to the AP-8 model is still missing

- On line 513, you state that “forecasting space weather events may require additional feature inputs”. There is no “may” in there: you cannot forecast space weather events with just spacecraft latitude and longitude as inputs, when the driver of the events is solar activity! Please rephrase.

 

 

 

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report (Previous Reviewer 1)

Comments and Suggestions for Authors

The authors have responded in my comments raised during the previous review. In order the paper to be accepted, it would be necessary to update few sentences in the manuscript to avoid the creation of serious misunderstandings to the readers.

It should be clearly stated - where suitable - that actually this paper proposes a prediction method for the time-stable proton flux profiles measured by a satellite at the orbit of the considered one - during its crossings through the SAA.

This of course has to due with the LAT/LON related training for the given orbit/altitudes.

To this respect the statements in the Lines: 15-16, 48-49, 100-101, 109, 361, 490 should be slightly updated accordingly.

In addition, the statement in the Line 422 should also change as it reads: "Firstly, we observe that in regions where proton flux remains relatively stable". The authors confuse the term stable. SAA profiles are stable everywhere. The satellite crossings produce varying fluxes but this is due to spatial variation and not to temporal variation.

In addition, the authors should clarify what "normal conditions" means under their statement at line 425.

Last, the authors should clarify in the conclusion section - at least - that the proposed prediction method is applicable for the orbital characteristics of the selected satellite the data of which were used for the model training.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report (Previous Reviewer 2)

Comments and Suggestions for Authors

Dear Editor,

I have reviewed the second revision of the manuscript

“A proton Flux Prediction Method Based on Attention Mechanism and Long Short-Term Memory Network”

By Zhang et al.

The authors have improved their manuscript, however a couple of issues remain.

The authors have addressed well my question about the metrics, and the accuracy of the model in relative terms as opposed to absolute terms. Their tabulated data (Tables 3 and 4) of the logarithmic metrics show that the model LSTM-TPA predicts, in relative terms, the SAA regions much better than the non-SAA regions. This is well reflected in the revised text on pages 16 and 17.

However, strangely the page 15 revisions still state that the model “provides accurate predictions of proton flux in non-SAA regions” (line 433). The authors also say that “in some predictions of non-SAA regions, there may be a deviation of 1-2 orders of magnitude … However, in most cases ... within same order of magnitude …”.

If we look at Figure 11 (a), 11 of the 15 minima between the SAA peaks in the case #21 have the predicted minimum flux over an order of magnitude smaller than the observed one. This does not coincide with the wording “accurate predictions”.

The text on page 15 contradicts the text on pages 16 and 17. Further, the text on pages 16 and 17 discuss the same matter as the text on page 15 (albeit with better correspondence to your results). I suggest that you remove the three revised paragraphs on page 15 completely, to remove the conflict.

I also noticed that you are talking about other models as in LSTM and AP-8, in comparison to TPA-LSTM, on line 439. However, you have dropped all results of the LSTM-only model (already in the previous revision, I missed that): If you don’t show the LSTM results, you cannot comment on its accuracy!

Other issues:

----------------

- Abstract line 16: I’m curious about the appearance of the term “time-stable” in the manuscript (also “stable time series” on line 52). Is there a necessity for starting to use the term? There are some rigorous methods for determining stability of time series, have you analysed the data? It might be prudent to drop the term.

- Abstract line 23: “high level” is not much better than “certain level”. Please be specific, for example “better than AP-8”, likewise on line 517.

- Line 441, you emphasise the latitude and longitude features, but surely you are using latitude and longitude also for AP-8? The LSTM and TPA methods are the ones that are the significant difference to the AP-8.

 

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper is not suitable for publication as it has major issues. I will not focus on the prediction methods the authors have developed, but on the variables the author predict and the added value they claim that their method can give to the community.

The authors base all their work on 1 year measurements obtained by a LEO satellite using a single flux time series of protons with energies 100-275 MeV. From the orbital characteristics of the satellite, and the measurements presented it is evident that the satellite cross the SAA and the detector on-board measures the proton trapped radiation. It is known that the SAA is relatively stable and does not suffer from temporal variations of significance within a time of period of 1 year. To this respect, the methods employed by the authors, in the best case, present actually a reconstruction of the SAA based on the measured profiles and utilizing the position of the satellite using the LAT and the LON information.  Note that SAA is stable and we do not expect drastic changes within a year. To this respect, it is fully expected that the "predicted profiles" match better with the actual measurements compared to AP8 model since the latter is a long-term specification model. I can fully agree that the comparisons between AP8 and the actual measurements are useful but this is not the focus of this work. Moreover, we do not have any additional information about the detector in order to evaluate the measurements of the single channel provided. It is not even clear if the measurements used are proton differential fluxes (cf. bin: 100-275 MeV) or proton integral fluxes (cf. proton flux units in the figures in the article.)

In addition, the authors include a list of Space Weather Events such as SXRs, REEs and a single SPE and they claim that their model maintain a certain level of predictive accuracy. I cannot understand how their model was trained to predict a single SPE with such a high proton energy signature- given the limited number of SPEs that have taken place within the year and especially given the absence of any model input related to the solar proxies. In addition, it is not clear from the Figure 12 how they defined the start/end times of this SPE characterized by protons within 100-275 MeV. Similarly, it is not clear how the model was trained to predict electron flux enhancments, or signatures of SXRs. The authors do not mention the prediction time window for these sporadic events the prediction of which cannot be based on the LAT LON information. 

Comments on the Quality of English Language

Minor issues

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Editor,

 

I have reviewed the manuscript

 

“A proton Flux Prediction Method Based on Attention Mechanism and Long Short-Term Memory Network”

 

By Zhang et al.

 

The manuscript presents a method for predicting proton fluxes in the Earth’s magnetosphere, based on neural network approach that uses Temporal Pattern Attention (TPA) and Long Short-Term Memory (LSTM) network. Such research is important as proton fluxes pose a hazard to satellite technology and humans in space. Further, a working neural network predicting timeseries could be used for forecasting space weather in general.

 

The manuscript is interesting, and does warrant for publication, however I have are several reservations to this. The paper is unevenly written: the methods are well presented, however the results are not: often it is difficult to see how the authors reach a particular point. In particular the claim that the model can predict space weather events is not well founded, and not credible. The following issues should be addressed before the manuscript can be published.

 

Presentation of test results

--------------------------------

 

The presentation of results is not clear. The left panels in Figure 11 and all panels in Figure 12 are unclear, all we see is very sharp peaks with nothing in between.

 

On lines 418, the authors say that “We observe that in non-SAA regions are capable of effectively predicting the general variation trends of proton flux”. How do we see this? This is very unclear. Presumably the peaks in the figures are mostly SAA crossings? This is in no way made clear in the manuscript, but calculating the orbital period of order 4 hours, this is roughly the temporal spacing of the peaks.

 

In that case the non-SAA areas are the areas between the peaks, but we cannot see any variation in those areas as the y-axis is linear.

 

Please change the y-axis scaling to logarithmic, and show examples of the non-SAA and SAA regions separately explaining which is which, or alternatively a time period that spans a couple of SAAs and shows the non-SAA region in between.

 

Also, please mark the areas where the spacecraft crosses the SAA in the figures, or explain in the text that the peaks are due to such crossings.

 

For Figure 13, again please use logarithmic axes (both x- and y-axes). Also, include similar comparison for the AP-8 model.

 

The full evaluation of the results cannot be made before these changes are done.

 

Predicting space weather events

---------------------------------------

 

On line 453 the authors claim that “even during occurrences of space weather events, our model maintains a certain level of predictive accuracy.” This would be very strange: how would a model with only past proton flux, latitude and longitude as inputs be able to forecast flares, solar proton and relativistic electron events? To make such claim, much more rigorous analysis must be made. Why do you not show a blow-out image of these events, such as you do in Figure 11? Why do you not include the AP-8 in this comparison?

 

The readers are not given information about the SAA crossings, but judging from the Figure 12 panels, it looks like the peaks labelled REE and SFX peaks are seen roughly at the time when the spacecraft would be at SAA (for example in panel c, compare the spacing between the first 6 peaks: the REE sits perfectly well in the series). Also the spacings between the SPE event peaks, and the spacings between the peak groups, are similar to the SAA pattern.

 

Please analyse whether the successful space weather predictions are just peaks that take place when the spacecraft crosses the SAA, or alternatively remove the claims that the model predicts space weather events.

 

 

Other issues:

----------------

 

Overall, please remove the phraseology “certain level of”: it is meaningless.

 

Line 164: “The result is a reliable and accurate processed time series dataset” is vague: how do you define “reliable” and “accurate”? Please remove the sentence, or at least replace the words “reliable” and “accurate”.

 

Line 340: States first that the data predominantly occupies “Three quadrants” , then gives only one longitude and latitude range.

 

Line 356, Figure 8: The data differs from Figure 7, which demonstrates the data set. Where are the figure 8 panels from? Does it use the same data? What does the “3D global map” mean? Is the height of the flux determined somehow? How is it presented? The whole discussion related to Figure 8 is vague.

 

Line 391: You use the phrase “each operating condition”, however you do not explain what the operating conditions are.

 

Line 400, Figure 10: There seems to be an area with high R^2 at around 400 hidden states, why did you not consider that number?

 

Line 407: The testing scheme is not explained: how long a period of observations is given to the model to predict the future value? For example Wei et al 2018 use 300 previous days to predict the next day. Is the input data in the testing phase “raw data”, that is without the wavelet decomposition which (presumably) requires analysis of “future” data points?

 

Line 409: Include a reference and explanation for AP-8 model.

 

Line 429: The “same order of magnitude” cannot be seen from the current figures, the logarithmic y-axis is the only way to see this.

 

Line 496: The moving average wavelet method does demonstrate that it can retain trend information and denoise the dataset. However, you do not investigate if it affects the final prediction results.

 

Line 510: The method is interesting and could possibly be used in space weather prediction, however please emphasise that for such purpose, much more inputs than just the flux, latitude and longitude would be needed. If you include the solar proton events, include also example of inputs that would be needed for such forecasts.

 

Appendix A (Appendix X on Line 152, please correct): the table A1 header states that Space weather events include solar x-ray flare, relativistic electron enhancement, solar proton event, however, this is misleading, and the table is confusing:

• Time range is confusing: Include day, month and year to both start and end times

• The acronyms are not explained

• Some events have multiple start times and no end times, they should be explained

• The “None” is not explained. What does this mean?