The Consistency and Quality of ChatGPT Responses Compared to Clinical Guidelines for Ovarian Cancer: A Delphi Approach

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Careful comparison of outputs from ChatGPT and other AI technologies to expert clinical guidelines is a worthwhile exercise, as more patients are using these technologies to ask questions about their disease and care.

I would strongly suggest the following edits to improve the quality of presentation:

1.      Table 2 + Figure 2 – as the mean scores are presented in both, I would suggest limiting Table 2 to the survey assessment questions (i.e. take out numbers). Figure 2 could then include the mean scores + 95% CI, with statistically significant differences indicated directly on the graph.

2.      What are the eight clinical questions that were asked? These should be included for completeness.

3.      The conclusion as written is very long and repetitive and should be revised to be more concise.

Comments on the Quality of English Language

This manuscript would benefit from language editing. Two notable passages that are very unclear and should be re-written include:

line 191 - "performed severely"

line 198 - "possibility of hallucinations"

Author Response

Dear reviewer, thanking you for your valuable suggestions, we reply below to your comments by submitting an improved version of the manuscript:

 

 

Comments and Suggestions for Authors

Careful comparison of outputs from ChatGPT and other AI technologies to expert clinical guidelines is a worthwhile exercise, as more patients are using these technologies to ask questions about their disease and care.

I would strongly suggest the following edits to improve the quality of presentation:

1. Table 2 + Figure 2 – as the mean scores are presented in both, I would suggest limiting Table 2 to the survey assessment questions (i.e. take out numbers). Figure 2 could then include the mean scores + 95% CI, with statistically significant differences indicated directly on the graph.

THE AUTHORS AGREE ON THE REDUNDANCY BETWEEN TABLE 2 AND FIGURE 2. HOWEVER, WE PREFER TO REMOVE FIGURE 2 FROM THE NEW DRAFT AND KEEP TABLE 2 INTACT. THIS WAY THE INFORMATION IS SUPPOSED TO BE RENDERED IN A MORE COMPLETE AND UNDERSTANDABLE FORM.

2. What are the eight clinical questions that were asked? These should be included for completeness.

TABLE AD HOC ADDED

3. The conclusion as written is very long and repetitive and should be revised to be more concise. – REPHRASED AS SUGGESTED

 

Comments on the Quality of English Language

This manuscript would benefit from language editing. Two notable passages that are very unclear and should be re-written include:

line 191 - "performed severely" – REPHRASED AS SUGGESTED

line 198 - "possibility of hallucinations" – REPHRASED AS SUGGESTED

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript by Piazza used the Delphi method to compare the responses from ChatGPT and physicians following AIOM guidelines. They also compared the responses from different versions of ChatGPT. They verified that AI technologies such as ChatGPT still have certain gaps in knowledge in certain professional fields. There are some concerns about the manuscript:

1.      There are no descriptions of the 8 questions in the methods section.

2.      The responses to 8 questions given by physicians and those given by AI can be compiled into supplementary for better understanding.

 

3.      The author only compared ChatGPT with professionals. Different AI models, such as Gemini, Copilot, and Grok, should be included since the goal of this manuscript is to verify the use of AI models in clinical.

Comments on the Quality of English Language

No comments.

Author Response

Dear reviewer, thanking you for your valuable suggestions, we reply below to your comments by submitting an improved version of the manuscript:

Comments and Suggestions for Authors

The manuscript by Piazza used the Delphi method to compare the responses from ChatGPT and physicians following AIOM guidelines. They also compared the responses from different versions of ChatGPT. They verified that AI technologies such as ChatGPT still have certain gaps in knowledge in certain professional fields. There are some concerns about the manuscript:

1. There are no descriptions of the 8 questions in the methods section.

TABLE AD HOC ADDED

 

2. The responses to 8 questions given by physicians and those given by AI can be compiled into supplementary for better understanding.

ADDED AS SUPPLEMENTARY MATERIAL

3. The author only compared ChatGPT with professionals. Different AI models, such as Gemini, Copilot, and Grok, should be included since the goal of this manuscript is to verify the use of AI models in clinical. - we thought we would limit the comments to just the ChatGPT tool because it is currently the most widely used, as well as the one that offers easy access to the potential user audience

 

Comments on the Quality of English Language

No comments.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for addressing my concerns/suggestions. Congrats on a great paper!

Author Response

Thank you for your kind words and for taking the time to read our work!

Reviewer 2 Report

Comments and Suggestions for Authors

The limitation part in the Discussion Section needs to include the limitations of this study. Some minor errors, such as  "each model." in Line 126 should be deleted.

Comments on the Quality of English Language

No comm.ents

Author Response

Thank you once again for the valuable suggestions and for taking time to read our work.
We have implemented the limitations paragraph by including those biases and critical issues related to the conceptualization of the study and the used tools.
We have also edited and fixed the text for minor errors other than the one pointed out in line 126.

best regards