A Machine Learning Approach for the Tune Estimation in the LHC

Round 1

Reviewer 1 Report

This manuscript provides a very interesting and novel approach to estimate the LHC tunes from the BBQ system spectra using artificial neural networks and to improve simulated data using a modified GAN to include real data artifacts and noise on the simulations, therefore improving the performance of the machine learning models. However, I believe there are severe problems in the presentation and methodology of the work presented that must be addressed before this manuscript is considered for publication:

• In lines 30-32 studies on the presence of 50Hz lines in the BBQ spectrum are mentioned but no reference to these studies is provided.
• The caption of Figure 1 must be improved. It shows the actual spectra of a BBQ acquisition. The title of the image must also be explained or removed.
• Figure 3 suggestion: Provide the meaning of each of the lines in the caption, so the reader does not have to find the citation in the text.
• Lines 107-108 question and suggestion: Is there any study on how often the detected peak in the BBQ spectra is wrong? If it happens very rarely in the training datasets, these peak detection errors could be considered measurement noise. This is likely not the case, but an indication of the amount of peak detection errors can be interesting.
• The term FLATTOP is mentioned several times in the text, but it is never introduced.
• The title of Figure 4 is also not explained in the text.
• In Section 4 a discussion on the size of the training and validation sets is missing. Also, would be convenient to mention here the type of loss function and gradient descent algorithm and parameters used to train the models.
• In Section 4.1, the introduction to dense layers in neural networks is not necessary, together with Figure 6. There is plenty of literature on this available and the image displayed does not even represent any of the actual models used in the study. A straightforward discussion on the topology and hyperparameters of the best neural networks designed would be much more interesting for the reader.
• Same applies to Figure 8. The reviewer thinks the discussion on CNNs is also redundant, but as this is a more complicated model architecture, removing it is just suggested.
• In both sections 4.1 and 4.2, the results of the different neural network topologies attempted during the design of the models is not interesting for the reader. Only the best models obtained should be presented. Therefore, Figures 7 and 9 can be removed and replaced with a comparison of the best ML models versus the different classical models. This would also make the comparison mentioned in lines 189-193 much clearer. A quantitative comparison of the accuracy and precision between the classical and ML algorithms is also missing.
• In the paragraph starting in line 225 and Figure 10: “a model having an architecture similar to that illustrated in Figure 6” is mentioned. Why is not just shown for training and validation of the best dense model? The reviewer also thinks this discussion fits better in the 4.1 section and a similar discussion (or comments on the topic) is missing for the CNN model that would fit in section 4.2.
• Line 312: I suggest providing the actual topology and hyperparameters of the designed SimGAN.
• Line 320, question: Why does each refiner produce a specific type of artifact? Shouldn’t it learn to produce different artifacts from the measured BBQ data?
• In section 5.1 it is missing a quantitative discussion on the training performance of the SimGAN. Figure 13 provides several good examples of the refiner performance, but this is not, in any case, an indication on the actual performance of the refiner and discriminator networks.
• Section 6 question: Why is the refined dataset not used to assess the performance of the different tune estimation algorithms and it is just used to improve the training performance of the ML models? This dataset provides both realistic spectrum data and ground-truth labelling, doesn’t it?
• Line 337: The term “tune traces” is not introduced. Also, it is not described here what kind of data is being used? (measurement data)
• In section 6 it is mentioned that the performance of each tune estimation algorithm “can only be assessed qualitatively by visualising the tune estimate traces over the real spectra”. The reviewer finds this severely incorrect, as the text and subsequent figures proceed to describe 2 different quantitative metrics: using the mean and standard deviation of the predicted tune over time (Figure 14) and the stability metric used by the tune feedback system (Figure 15). If these metrics can be used qualitatively, they can also be used quantitatively (they could even be used as target for the different ML models).
• The title of Figure 14 is also not explained.
• Line 373: A single time series of real data is mentioned, it should be made clear that this “single sample” is many individual inputs to the tune estimation algorithms. Also, drawing conclusions from a single time series does not seem scientifically adequate, as this time series could have been cherry-picked to obtain the best performance. It is also not indicated if this time series is the real data that has been used to train the SimGAN discriminator and refiner networks, as this could significantly bias the results presented in Section 6.

Author Response

Attached please find my replies to each of your comments. The results have been revisited, plots updated and conclusions redrawn. The figure captions now contain more details, and the experimental setup and data sources are made clear. Figure 10 was updated since the x-axis was not Epochs but number of training steps. The stability metric in Equation (7) was also updated after finding a mistake in the previous version. Some new references were added in response to some of the comments.

Author Response File: Author Response.pdf

Reviewer 2 Report

English has to be revised since redundances decrease the text understanding. Punctuation has to be improved. I suggest the revision by a native English-speaking person for catching minor mistakes. I find some points unclear and hard to understand. The figure captions should contain more information to explain the figures, especially figure 4 and 14. Figure 14 should have the heat map with some transparency level. Thus the prediction and confidence region lines can be more visible. Marks showing the specific points discussed in the text can significantly improve the understanding of the discussion arguments.
It was not precisely clear to me if the SimGAN predictions are the parameters used to generate the simulated data detected as similar to the experimental or if the simulated data approved by the discriminator passed through the tune detection in the same way as the conventional approaches.
Since simulations generate the reference data, I would find interesting a discussion of the SimGAN sensitivity to the Gaussian noise level in light of the stability criterion.

Author Response

Attached please find my replies to your comments. The language used in the paper was corrected and made clearer. The results have been revisited, plots updated and conclusions redrawn. The figure captions now contain more details, and the experimental setup and data sources are made clear. Figure 10 was updated since the x-axis was not Epochs but number of training steps. The stability metric in Equation (7) was also updated after finding a mistake in the previous version. Some missing references were added.

Author Response File: Author Response.pdf