Machine Learning Applications for Jet Tagging in the CMS Experiment
Round 1
Reviewer 1 Report
The article gives a summary of the new machine learning algorithms used
by the CMS collaboration to perform jet tagging and data analysis.
There are few general comments:
• Check from a native English speaker is suggested.
• Figures description should be improved, both in the caption and in
the main text.
• All the acronyms should be used only after the complete name is
given (e.g. LSTM at line142).
And some line by line comments:
• L.24: resulting of LHC -> resulting from LHC.
• L.40: information order -> information of the order.
• L.44: “The physicists solve them” sounds too informal.
• L.51: a perfect soil -> a perfect testing ground
• L.59-64: too long and a little confused sentence, with repetition of
the word originated/originating/originate.
• L.72 and others: describe better R (why it is without units).
• L.82-88: this part is not very clear. There are many algorithms
presented, but only two are discussed. In addition, the algorithm
DeepJet is not presented, even if it i described in this section.
• L.90: The authors are assuming that the reader already knows what
the CSV algorithm is.
• L.93: must explicitly told that η is the pseudorapidity.
• L.101: no need of word “respectively” at the end of the sentence.
• Figure 2: describe the nature of the peaks at 0.
• L.112: starts -> goes.
• Pag.4: the large blank space should be avoided.
• L.137-139: splitting it into two sentences would make text more
readable.
• L.140: add a comma: [... of the four groups listed above, all, except
...] or rephrase [... all of the four groups listed above, except ...].
• L.146-148: The sentence should be rewritten since it is not
completely clear. Do not use the letter "x" for the product.
• L.150-152: Plots show background misidentification efficiency vs
the signal efficiency, not the opposite.
• Formula 3: make explicit N.
• L.170: tau -> τ.
• L.175-176: after “other” rewrite like “methods that exploit more
efficiently the information coming from the CMS detector have been
developed”.
• L.178: well -> better.
• L .184: explicitly repeat that is the energy to be normalised.
• Figure 5: figure text is too small.
• L.190: be come -> come.
• L.196: explain where these numbers come from.
• Figure 6: there are other taggers than the ones indicated in the text.
Remove the not necessary ones or, at least, report their presence
in the text (e.g. “are reported in Fig.6(a), compared to other
algorithms”).
• L.229: in other parts of the text ADAM is lowercase.
• L.235: remove “for”.
• L.237: DeepAK-MD -> DeepAK8-MD.
• L.255-257: split in two sentences. AUC should be defined.
• Figure 7: AUC numbers in figures are different from the ones given
in the text.
• L.274-277: the text is not clear.
• L.282: improve -> improves.
• L.305: p^{miss}_{T} -> missing P_{T}.
• L.309-311: these two sentences should be rewritten in a better
english.
• L.319: it shows -> it also shows.
• L.330 and following: for an improved formal correctness it is better
to write H->b b-bar instead of H->bb.
Comments for author File: 
Comments.pdf
Author Response
Dear Reviewer,
thanks a lot for the very detailed and accurate review. Here you find replies line by line to the comments.
There are few general comments:
- Check from a native English speaker is suggested.
- Figures description should be improved, both in the caption and in the main text.
- All the acronyms should be used only after the complete name is given (e.g. LSTM at line142).
→ fixed
And some line by line comments:
- L.24: resulting of LHC -> resulting from LHC.
→ accepted
- L.40: information order -> information of the order.
→ accepted
- L.44: “The physicists solve them” sounds too informal.
→ Such issues are tackled by reducing the dimensionality…
- L.51: a perfect soil -> a perfect testing ground
→ accepted
- L.59-64: too long and a little confused sentence, with repetition of the word originated/originating/originate.
→ accepted… Sentence split and repetition of originate reduced.
- L.72 and others: describe better R (why it is without units).
→ R is now defined. It is an adimensional variable.
- L.82-88: this part is not very clear. There are many algorithms presented, but only two are discussed. In addition, the algorithm DeepJet is not presented, even if it is described in this section.
→ Here the aim is to cite the main algorithms developed by the collaboration but to focus only on the latest applied.
We added a clarification at the beginning to clarify the logical flow, and explain how several algorithms have been developed over the years. Finally, we added a sentence to introduce also deepjet in this context.
Now this part reads:
Two different algorithms are used to find the secondary vertices: the adaptive vertex
reconstruction (AVR) and the inclusive vertex finding (IVF).
After secondary vertex reconstruction is performed, dedicated algorithms have been developed by the CMS collaboration in order to perform heavy-hadron jets tagging based on the properties of secondary vertices from which particles in the jets originated. An example is the Combined Secondary Vertex algorithm, developed in Run-I, that combines the variables of secondary vertexes in a likelihood ratio discriminant [cite] . In Run-II, two new algorithms were developed for heavy-hadron tagging starting from the CSV and making use of ML techniques: the CSVv2 and the DeepCSV.
Ultimately a more sophisticated technique, DeepJet, has been developed that makes use of many variables both of high and low level.
- L.90: The authors are assuming that the reader already knows what the CSV algorithm is.
→ We have modified lines 83-88, where the CSV was mentioned briefly, and added a corresponding reference to the implementation, so that at L90 CSV has already been defined in the context of this paper and the reader is aware from the previous paragraph.
- L.93: must explicitly told that η is the pseudorapidity.
→ now it is defined in line 70
- L.101: no need of word “respectively” at the end of the sentence.
→ done
- Figure 2: describe the nature of the peaks at 0.
→ These are jets without a selected track and secondary vertex are assigned a negative discriminator value. The first bin includes the underflow entries.
→ Explanation added to figure caption.
- L.112: starts -> goes.
→ accepted
- Pag.4: the large blank space should be avoided.
→ Yes we will fix this once the paper will be ready for publication
- L.137-139: splitting it into two sentences would make text more readable.
→ done
- L.140: add a comma: [... of the four groups listed above, all, except ...] or rephrase [... all of the four groups listed above, except ...].
→ rephrased
- L.146-148: The sentence should be rewritten since it is not completely clear. Do not use the letter "x" for the product.
→ X is now fixed
- L.150-152: Plots show background misidentification efficiency vs the signal efficiency, not the opposite.
→ fixed
- Formula 3: make explicit N.
→ Now the N variable is defined and we added the reference to the original paper where the N-subjettiness was introduced.
- L.170: tau -> τ.
→ fixed
- L.175-176: after “other” rewrite like “methods that exploit more efficiently the information coming from the CMS detector have been developed”.
→ done
- L.178: well -> better.
→ done
- L .184: explicitly repeat that is the energy to be normalised.
→ done
- Figure 5: figure text is too small.
→ we enlarged the figure size. We hope this is enough…
- L.190: be come -> come.
→ fixed
- L.196: explain where these numbers come from.
→ we removed the numbers and added an explanation of the decorrelation procedure
- Figure 6: there are other taggers than the ones indicated in the text. Remove the not necessary ones or, at least, report their presence in the text (e.g. “are reported in Fig.6(a), compared to other algorithms”).
→ we added “compared to other algorithms” in the text
- L.229: in other parts of the text ADAM is lowercase.
→ done
- L.235: remove “for”.
→ done
- L.237: DeepAK-MD -> DeepAK8-MD.
→ fixed
- L.255-257: split in two sentences. AUC should be defined.
→ defined
- Figure 7: AUC numbers in figures are different from the ones given in the text.
→ fixed
- L.274-277: the text is not clear.
- L.282: improve -> improves.
→ fixed
- L.305: p^{miss}_{T} -> missing P_{T}.
→ done
- L.309-311: these two sentences should be rewritten in a better english.
→ Rephrased. Now it reads as follows: The final state consists of a dilepton pair with opposite charge and missing pT. A parametric DNN algorithm is used to increase the sensitivity to the signal against the main SM background represented by t ̄t events. A total of 11 kinematic variables are used for the training with the addition of two parameters: the top squark and neutralino masses. The choice of the network parameters strongly depends on the masses of the new particles, and so a specific model is adopted for each signal point.
- L.319: it shows -> it also shows.
→ done
- L.330 and following: for an improved formal correctness it is better to write H->b b-bar instead of H->bb.
→ fixed this and other occurrence (also for ccbar and qqbar)
Please let us know if you have further comments.
Kind Regards,
the authors
Reviewer 2 Report
Machine learning application to jet tagging is widely used in collider experiments. I consider this is the future of our field. I enjoy reading this review article on this topic and certainly like to see it in the publication in this journal. However, I will request the authors to include a small section on the future advancement in this direction and a bit of information on other collaborations in LHC like ATLAS, ALICE, etc. That would be helpful for the readers. With this request, I recommend publishing this review in this journal.
Author Response
Dear Reviewer,
thanks a lot for reviewing our paper. In the following you find our reply.
Future advancements we are aware of are briefly described in the Conclusion section. Other collaborations apply similar techniques but the specific algorithms could differ since they strongly depend on the detector characteristics. The scope of this paper is to focus on the CMS side, so we just added a reference for similar results for Atlas, which has the closest comparison to what is presented here.
We enriched the description in the Conclusion by including the following sentences and citing some of the work ongoing for the ATLAS Collaboration since it is the nearest to the scope of this review.
The development of these technologies is still ongoing both in CMS as well as other LHC experiments like ATLAS[1,2,3]. Among the future developments, it is possible to extend the deep learning to other decay chains of SM particles, for example including leptons in the final state, and to particle beyond SM.
1 http://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2016-039/
2 http://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-PHYS-PUB-2017-010/
3 https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PAPERS/FTAG-2018-01/
Best Regards,
the authors
Reviewer 3 Report
Dear authors,
the paper is really well written and it was a pleasure to read it.
I have only some minor remarks:
- I imagine that described wok is being presented on behalf of the CMS experiment, shouldn't this be written in the authors field ?
- please change in line 32 from "build " to reconstruct
- sentence in line 57-58 is redundant with sentence in lines 64-65. I would suggest keep the one in lines 64-65 and remove the other one
- in lines 59 to 64 there is a very long sentence. For the sake of being more clear I would suggest to divide it. Additionally in this sentence word "originate" is repeated three times.
- line 193-194, please change ": this" to ". This"
- on page 4 there is a huge gap between the text and next figure. Would it be possible to edit the article such it would be removed ?
In the text it is mentioned that " The jet pT and η distributions are reweighted in order to have the same spectrum for all the jet flavours in the training, therefore avoiding that the discrimination is based
on the spectrum of these variables, which would introduce a dependence on the sample used. " but later it is never stated or shown that this is really the case, that algorithms efficiencies are independent from η or pT.
Does number of interaction per bunch crossing has any influence on the jet identification efficiency ?
Kind regards
Author Response
Dear Reviewer,
thanks a lot for your detailed and accurate review and for your interest in our work. In the following you find line-by-line replies to your comments.
I have only some minor remarks:
- I imagine that described work is being presented on behalf of the CMS experiment, shouldn't this be written in the authors field?
R: The aim of this work is just to do an external review of the activities and the materials that are publicly available up to the date, and is not done on behalf of the experiment.
- please change in line 32 from "build " to reconstruct.
→ accepted
- sentence in line 57-58 is redundant with sentence in lines 64-65. I would suggest keep the one in lines 64-65 and remove the other one
→ accepted
- in lines 59 to 64 there is a very long sentence. For the sake of being more clear I would suggest to divide it. Additionally in this sentence word "originate" is repeated three times.
→ accepted… Sentence splitted and repetition of originate reduced.
- line 193-194, please change ": this" to ". This"
→ accepted
- on page 4 there is a huge gap between the text and next figure. Would it be possible to edit the article such it would be removed?
→ Yes we will fix this once the paper will be ready for publication
In the text it is mentioned that " The jet pT and η distributions are reweighted in order to have the same spectrum for all the jet flavours in the training, therefore avoiding that the discrimination is based on the spectrum of these variables, which would introduce a dependence on the sample used. " but later it is never stated or shown that this is really the case, that algorithms efficiencies are independent from η or pT.
→ This sentence is referred to the CSVv2 tagger and you can see in Figure 2 that the output of the discriminator is the same whatever the pt and eta of the jet is. The only requirement is that the jet pt is greater than 20 GeV (as stated in the plot) but this comes from reconstruction quality requirements.
Does number of interaction per bunch crossing has any influence on the jet identification efficiency?
→ Yes it has. There are algorithms of pile up removal to take into account this effect. This enters in the efficiency of the reconstruction algorithms but has a smaller impact on the jet type identification.
Kind Regards,
the authors
