Modernising Receiver Operating Characteristic (ROC) Curves^†

Round 1

Reviewer 1 Report

The authors propose using a combination of measurement system analysis and item response theory for receiver operating characteristics (ROC), among the limitations used in the past, to modify receiver operating characteristics (ROC). The following are a few suggestions for revision.

1.     The abstract must be supplemented with specific quantitative analysis results and conclusions. The wording "... should enable more reliable decision-making in conformity assessment in many scenarios..." is too vague.

2.     In line 482, on Measurement uncertainties, explain U(z) definition.

3.     In line 486, determine whether the formula is correct.

4.     The measurement uncertainties are based on the original data, and the classification model is evaluated using the ROC method. It is generally assumed that the data have a certain quality, i.e., that the measurement error is acceptable before building the model. The goal of model building is to minimize error (including measurement errors), so it is suggested that additional information be provided regarding whether the proposed method provides sufficient information for determining the contribution in practice.

Author Response

R1: The authors propose using a combination of measurement system analysis and item response theory for receiver operating characteristics (ROC), among the limitations used in the past, to modify receiver operating characteristics (ROC). The following are a few suggestions for revision. Thank you, referee 1 for your constructive comments

R1:1 The abstract must be supplemented with specific quantitative analysis results and conclusions. The wording "... should enable more reliable decision-making in conformity assessment in many scenarios..." is too vague. We've added a few words (lines 21 - 25) to reinforce our claims.

R1:2 In line 482, on Measurement uncertainties, explain U(z) definition. Explanation added.

R1:3 . In line 486, determine whether the formula is correct. The formula is correct.

R1:4 The measurement uncertainties are based on the original data, and the classification model is evaluated using the ROC method. It is generally assumed that the data have a certain quality, i.e., that the measurement error is acceptable before building the model. The goal of model building is to minimize error (including measurement errors), so it is suggested that additional information be provided regarding whether the proposed method provides sufficient information for determining the contribution in practice. 

As is written in the first sentence at line 514, measurement uncertainties in the ability and difficulty values are estimated from logistic regressions of the Rasch formula to the raw data (the footnote gives a reference to the WINSTEPS program which has made these estimates). We make no assumptions about the data having a "certain quality", instead we experimentally estimate these. "Model building" cannot minimise the measurement uncertainties - the latter are what they are. (Please note that measurement errors are not generally synonymous with measurement uncertainties. )

The purpose of this section is to judge whether differences between ROC curves are significant considering the size of the logistic regression uncertainties. In the text after figures 8, it is clearly stated how well the scatter in the ROC curves compares with the measurement uncertainties.

 

Reviewer 2 Report

Notes:

- Please improve the overall quality of the figures

- Missing a comparison with other performance metrics

- Where did your method succeed, and where did the original ROC curve fail?

- Comparison with what? What did you obtain? What did you improve?

Abstract:

- "SL"? Is it an acronym?.

- It needs to be modernized, ok. But give examples of situations you are trying to tackle if it is extensively used because it is not so bad.

- Comparison with what? What did you obtain? What did you improve?

- Comparison with other performance metrics?

Keywords:

- Please do not use acronyms in the keywords.

Introduction:

- Please define correctly the acronyms e.g. True Positive Rate (TPR).

- Missing citations e.g. traceability.

- Missing a clear paragraph describing the article innovation - what is old and what is new.

- Missing a paragraph describing the article organization "In Section XX, it will..."

Mote: Missing an important Related Work section

Materials and Methods:

- "...screening tests for pregnancy..." Why? A difficult problem? A problem that can represent a generalization?

- Please improve the Figures' quality.

Results and discussions:

- Please start the section by describing its contents.

- Please improve the quality of the figures.

Author Response

R2: Notes:

• Please improve the overall quality of the figures. Most of the figures have been modified to make them more readable, even in monochrome. This includes making markers larger and more distinct. We have added an explanation (lines 433 ff) that there are several cases where the closeness of the curves, particularly in figure 5a, indeed makes it difficult to distinguish the symbols denoting them: this is a result, not a shortcoming in the figure quality. The authors in any case hope that the figures satisfy the journal's recommendations.

 

R2:2 Keywords:

• Please do not use acronyms in the keywords. OK, keywords modified

 

R2:3 Introduction:

- Please define correctly the acronyms e.g. True Positive Rate (TPR).

- Missing citations e.g. traceability.

- Missing a clear paragraph describing the article innovation - what is old and what is new.

- Missing a paragraph describing the article organization "In Section XX, it will..." 

- Lines 47 - 8: TPR and other ROC abbreviations are clearly defined in Appendix 1 - as is already stated (line 46). Or does the referee mean that we have defined TPR incorrectly?

- Line 52: Metrological quality assurance - Traceabillity and Measurement uncertainty - is explained in the reference [4].

- What is old and what new: see response to referee R1:1: We've added a few words (lines 21 - 25) to reinforce our claims.

• Article organisation: new section 1.1 added

R2:4 Note: Missing an important Related Work section. The original section 2.4 was in fact substantially the Related Work section. This has been moved to a new section 2.

 

R2:5 Materials and Methods:

- "...screening tests for pregnancy..." Why? A difficult problem? A problem that can represent a generalization?

- Please improve the Figures' quality. 

The English word "exemplifying" means: showing a potential generalisation by example.

The reviewer comment about "screening" is correct. Pregnancy tests are usually not used for screening, they are used instead after indication, suspicion, or expectation of the condition! Text at beginning of section 3 has been modified accordingly.

"Figure quality" - see our response to R2 Notes.

 

R2:6 Results and discussions:

- Please start the section by describing its contents.

- Please improve the quality of the figures. 

- Text added.

• Quality of figures: see our response to R2: Notes

 

R2: Notes:

- Missing a comparison with other performance metrics

- Where did your method succeed, and where did the original ROC curve fail?

- Comparison with what? What did you obtain? What did you improve?

 

Text added in Conclusions.

 

 

 

 

 

 

 

 

Reviewer 3 Report

The authors proposed a modernised approach to ROC curves with a combination of measurement system analysis and item response theory. However the problem is well-formulated and motivations are not clarified clearly. Novelty of the work is not highlighted properly. I would suggest to significantly improve introduction and methodology to make the paper coherent and useful to the reader. English language need significant improvement as well. 

 

Author Response

R3: The authors proposed a modernised approach to ROC curves with a combination of measurement system analysis and item response theory. However the problem is well-formulated and motivations are not clarified clearly. Novelty of the work is not highlighted properly. I would suggest to significantly improve introduction and methodology to make the paper coherent and useful to the reader. English language need significant improvement as well. 

 

The other referees have been more explicit about what they see as shortcomings in the manuscript and we have modified the paper in response to these.

This includes adding a few words (lines 21 - 25) to reinforce our claims + Text added in Conclusions.

For your information, the corresponding author is in fact a native Englishman. Some readers might perceive the language as difficult, owing perhaps to the novelty of the subject matter. We have therefore carefully reviewed and amended the language throughout.

 

Round 2

Reviewer 2 Report

- The article has improved since the last submission. 

- Still missing a clear sentence in the introduction describing the innovation of the article. What is old, and what is new? This is essential to understand the article's contribution to the field.

- The article acronyms should be defined in the text when used for the first time, not only in Appendix 1.

- The Figures' quality (resolution) is still low.

- I didn't see a comparison with other state-of-the-art methods, which is essential to validate your method better.

Author Response

230502

Dear Editors,

On behalf of my co-authors, I am happy to submit a revised manuscript, entitled: “Modernising Receiver Operating Characteristic (ROC) curves”.

• Please check that all references are relevant to the contents of the manuscript. All references have been checked.
• Any revisions to the manuscript should be marked up using the “Track Changes” function if you are using MS Word/LaTeX, such that any changes can be easily viewed by the editors and reviewers. Track changes are included in the revised manuscript, as requested. (Previous resolved comments and responses have been closed.)
• Please provide a cover letter to explain, point by point, the details of the revisions to the manuscript and your responses to the referees’ comments:

R2:1       The article has improved since the last submission. Thank you.

R2:2       Still missing a clear sentence in the introduction describing the innovation of the article. What is old, and what is new? This is essential to understand the article's contribution to the field. We have revised the text at lines 62 - 67. Perhaps that new text will help the reader understand the novelty (beyond the existing text already in the Introduction (see section 1.1, lines 76 ff).

R2:3       The article acronyms should be defined in the text when used for the first time, not only in Appendix 1.  See response (IV) below.

R2:4       The Figures' quality (resolution) is still low. See response (IV) below.

R2:5       I didn't see a comparison with other state-of-the-art methods, which is essential to validate your method better. Two major sections of the existing manuscript in our opinion already give a comparison with other state-of-the-art methods: see in particular lines 143 ff and lines 628 ff.

• If you found it impossible to address certain comments in the review reports, please include an explanation in your appeal:

R2:3       The article acronyms should be defined in the text when used for the first time, not only in Appendix 1.  The following article acronyms are indeed already defined in the main body of the manuscript when used for the first time:

Line 41: SL

Line 44: ROC

Line 47: TPR

Line 47: FPR

Line 55: AUC

Line 56: CTT

Line 80: MSA

Line 80: IRT

Line 129: sROC

Line 139: hCG

Line 158: IU/L

Line 265: GLM

We may have missed an acronym: If the referee can specify exactly what is missing, we would be happy to remedy accordingly.

R2:4       The Figures' quality (resolution) is still low.

Admittedly the figures are quite small as they appear in the Word manuscript. But the resolution of each figure is not poor in our opinion. A full set of graphic files (*.png) is already up-loaded to the journal website. Since the resolution is adequate, it is difficult for us to make any changes which would satisfy the referee. Our response to the referees in the first revision on this point was:

‘Most of the figures have been modified to make them more readable, even in monochrome. This includes making markers larger and more distinct. We have added an explanation (lines 433 ff) that there are several cases where the closeness of the curves, particularly in figure 5a, indeed makes it difficult to distinguish the symbols denoting them: this is a result, not a shortcoming in the figure quality. The authors in any case hope that the figures satisfy the journal's recommendations.’

We would be happy to make any changes to the figures are felt necessary to improve their quality, but we do need a specific suggestion.

Round 3

Reviewer 2 Report

The article has improved a lot based on the reviewers' comments.

Please submit better-resolution figures. The figure quality and resolution are very low in the provided document.

Author Response

File attached.

Author Response File: Author Response.pdf