Evaluating a Machine Learning Tool for the Classification of Pathological Uptake in Whole-Body PSMA-PET-CT Scans

Round 1

Reviewer 1 Report

Brief Summary:

The aim of the study by Erle & Moazemi et al., was to assess the feasibility and applicability of computer-aided diagnosis (CAD) based on machine learning (ML) algorithms that analyze PSMA-PET-CT image prostate cancer patient data. The authors applied their bioinformatic approach on PSMA-PET-CT scans from established cohort of prostate cancer patients, some of which (n=72) were used as the training dataset and others (n=15) as the validation dataset. The ML algorithm was able to identify hotspots predominantly in prostate cancer metastatic sites with good accuracy. However, the ML algorithm was unable to distinguish between physiological and pathological PSMA uptake in the small glands. Moreover, it appears that the prostate did not score high in hotspot identification, hindering the significance of the study. This manuscript presents interesting data on the diagnostic potential of CAD PSMA-PET-CT image diagnostics and could serve as the basis of more comprehensive studies to elucidate the impact of PSMA uptake on prostate cancer diagnosis. However, some issues need to be addressed for the manuscript to be warrant further consideration, particularly the major weaknesses pointed out.

 

Strengths of the study:

• Establishment and validation of new non-invasive ML algorithm for prostate cancer diagnostic purposes
• Potential repurposing of PSMA-PET-CT imaging to supplement histopathological tumor/disease grading

 

Weaknesses of the study:

• Specificity of the ML algorithm is lacking when it comes to PSMA uptake in small glands
• The prostate did not score high on PSMA uptake compared to other organs hindering the reliability of this CAD approach

 

Comments

Abstract:

• Too technical at some points. Consider rephrasing some parts. [minor]

 

Introduction:

• Well-constructed and concise, pointing out the potential significance of the study in the clinical field, but consider rephrasing sentence in lines 71-73, since clinical diagnosis is also considered “an important task”. [minor]

Materials and Methods:

• Were the PSMA hotspots identified and analyzed by the ML algorithm in each patient individually and then pooled together for the mean? If so, the authors should provide a supplementary table with all the individual patient readings. Otherwise, they should clarify how the analyses were performed in a clearer fashion. [major]
• Anatomically annotate Figure 1 with specific organ sited shown. [minor]
• Table 2 should not be split between two pages. Hard to read. [minor]

 

Results:

• How did the authors classify hotspots as physiological or pathological? Were the hotspot hits confirmed to be pathological by histopathological assessment? This is unclear. [major]
• Why was the prostate not amongst the top hotspot hit for PSMA accumulation? The prostate is not even present in Table 5. This should be mentioned and discussed in the manuscript, since it is a major limitation of the approach if there is no explanation provided. [major]
• Figure 2 and Table 4 could be merged to avoid redundancy. [minor]

 

Discussion:

• See comment above about the lack of PSMA hotspot identification in the prostate. [major]
• Other limitations are nicely discussed, along with potential future work.
• The paragraph in lines 261-266 is confusing. The point the authors wish to make is unclear and they should consider rephrasing. [minor]

 

Data availability:

• It is unclear why the authors claim that any of their data cannot be made available by appropriate request. It is at the discretion of the authors not to share any of the data, but they should clarify to which German regulations they are referring to that won’t allow data sharing. [minor]

Author Response

Brief Summary:

The aim of the study by Erle & Moazemi et al., was to assess the feasibility and applicability of computer-aided diagnosis (CAD) based on machine learning (ML) algorithms that analyze PSMA-PET-CT image prostate cancer patient data. The authors applied their bioinformatic approach on PSMA-PET-CT scans from established cohort of prostate cancer patients, some of which (n=72) were used as the training dataset and others (n=15) as the validation dataset. The ML algorithm was able to identify hotspots predominantly in prostate cancer metastatic sites with good accuracy. However, the ML algorithm was unable to distinguish between physiological and pathological PSMA uptake in the small glands. Moreover, it appears that the prostate did not score high in hotspot identification, hindering the significance of the study. This manuscript presents interesting data on the diagnostic potential of CAD PSMA-PET-CT image diagnostics and could serve as the basis of more comprehensive studies to elucidate the impact of PSMA uptake on prostate cancer diagnosis. However, some issues need to be addressed for the manuscript to be warrant further consideration, particularly the major weaknesses pointed out.

 

Strengths of the study:

• Establishment and validation of new non-invasive ML algorithm for prostate cancer diagnostic purposes
• Potential repurposing of PSMA-PET-CT imaging to supplement histopathological tumor/disease grading

> Thank you for mentioning the strengths

Weaknesses of the study:

• Specificity of the ML algorithm is lacking when it comes to PSMA uptake in small glands
• The prostate did not score high on PSMA uptake compared to other organs hindering the reliability of this CAD approach

 

> Thank you very much for these comments and the extensive review. As you describe completely correctly, the algorithm has weaknesses in small glands. For the small glands we think this problem is limited as the area of the head is not often the critical area in which to decide for metastases but in cases with very intensive tumor load. We added a sentence in the discussion (page 12, lines 302-304). Concerning the prostate, we identified a limited number of hotspots (14) only in the training cohort. That is why we did not analyze the prostate in table 6 (originally table 5). However, as will be discussed, the ML classifiers performed well in classification of prostate hotspots as pathological in the newly reported cross-validation step (page 10, lines 258-261).

Comments

Abstract:

• Too technical at some points. Consider rephrasing some parts. [minor]

> revised as requested (page 1, Abstract).

 

Introduction:

• Well-constructed and concise, pointing out the potential significance of the study in the clinical field, but consider rephrasing sentence in lines 71-73, since clinical diagnosis is also considered “an important task”. [minor]

> revised as requested (newly, page 12, lines 319-325).

Materials and Methods:

• Were the PSMA hotspots identified and analyzed by the ML algorithm in each patient individually and then pooled together for the mean? If so, the authors should provide a supplementary table with all the individual patient readings. Otherwise, they should clarify how the analyses were performed in a clearer fashion. [major]

> the PSMA hotspots the supplementary table is added as requested (table 3).

• Anatomically annotate Figure 1 with specific organ sited shown. [minor]
  > the figure caption has been revised as requested
• Table 2 should not be split between two pages. Hard to read. [minor]
  > revised as requested. However, it may change due to structural reisions by the editors. We will take care of this issue for the final version.

Results:

• How did the authors classify hotspots as physiological or pathological? Were the hotspot hits confirmed to be pathological by histopathological assessment? This is unclear. [major]
  > In general, rather than primary uptake in prostaten, metastases of prostate cancer are especially found in bone and lymph nodes, so tracer uptake in these specific locations is clearly pathological. Uptake in glands and kidneys is physiological, as the tracer uptake here does not show any form of metastasis but the excretion via urine and saliva. Due to the high number of hotspots (around 35 per patient), histopathological assessment of the tissues was ethically questionable. This topic is discussed on page 12, lines 319-325.
• Why was the prostate not amongst the top hotspot hit for PSMA accumulation? The prostate is not even present in Table 5. This should be mentioned and discussed in the manuscript, since it is a major limitation of the approach if there is no explanation provided. [major]

> Nearly all of our patients had total prostatectomy at the beginning of the disease and PET/CT was performed later in the course of diseases, so pathological prostate hotspots were only present in 14 patients of the training cohort and the test cohort patients did not feature any such. Thus, unfortunately, we cannot provide such analysis for the test cohort. However, in the cross-validation steps taken to identify the best ML classifier on the training cohort, pathological uptake in the prostate was classified with high accuracy. This information is added to results (page 10, lines 258-261) and discussion (page12, lines 306-309) sections.

• Figure 2 and Table 4 could be merged to avoid redundancy. [minor]

> revised as requested (figure 2 is replaced with ROC diagrams for classifier comparison)

 

Discussion:

• See comment above about the lack of PSMA hotspot identification in the prostate. [major]

> as already discussed, the topic is further discussed in results (page 10, lines 258-261) and discussion (page12, lines 306-309) sections

• Other limitations are nicely discussed, along with potential future work.
  > thanks for the nice comment
• The paragraph in lines 261-266 is confusing. The point the authors wish to make is unclear and they should consider rephrasing. [minor]
  > revised as requested (newly, page 12, lines 319-325)

 

Data availability:

• It is unclear why the authors claim that any of their data cannot be made available by appropriate request. It is at the discretion of the authors not to share any of the data, but they should clarify to which German regulations they are referring to that won’t allow data sharing. [minor]
  > Thank you for bringing up this point, according to German “Datenschutz Grundverordnung 2016/679” patient data including images or part of images can only be used/seen within the hospital and by personnel of these hospitals if the patient has not agreed specifically to publish it. As this is a retrospective analysis we have no agreement of the patients that we can make the data available openly, just that the data can be evaluated for studies. This information is added to the data availability statement (page 13, lines 359-364).

 

 

Author Response File: Author Response.docx

Reviewer 2 Report

The paper entitled "Evaluating a Machine Learning Tool for the Classification of
Pathological Uptake in Whole-Body PSMA-PET-CT Scans " utilizes the machine learning algorithm for the classification of CT scans. However, the reviewer suggests using the following contributions for further improvement.

1. Use of multiple machine learning algorithms for the classification
2. The addition of motivation in the introduction (e.g., gaps in the existing approach) clearly
3. The comparison of existing state-of-the-art methods could strengthen the paper

Author Response

The paper entitled "Evaluating a Machine Learning Tool for the Classification of

Pathological Uptake in Whole-Body PSMA-PET-CT Scans " utilizes the machine learning algorithm for the classification of CT scans. However, the reviewer suggests using the following contributions for further improvement.

1. Use of multiple machine learning algorithms for the classification

> thanks for the suggestion. We have analyzed and compared alternative ML classifiers and revised methods (page 5 and 6, lines 171-190 and 211-215) and results (page 10, lines 240, 241 and 246, 247) sections have been revised and figure 2 is added accordingly.

1. The addition of motivation in the introduction (e.g., gaps in the existing approach) clearly

> the motivations of the study are extended in the introduction. Specifically added: if trained properly, ML based methods can correct for inconsistencies arised by subjective inter-observer bias [1]. This topic is added to introduction (page 2, lines 56-58)

1. The comparison of existing state-of-the-art methods could strengthen the paper
   > as described in point 1., we have added the analysis of state-of-the-art supervised ML classifiers (SVM and Random Forest) and revised methods (page 5 and 6, lines 171-190 and 211-215) and results (page 10, lines 240, 241 and 246, 247) sections have been revised and figure 2 is added accordingly.

[1] Sollini, M.; Bartoli, F.; Marciano, A. et al. Artificial intelligence and hybrid imaging: the best match for personalized medi-cine in oncology. European J Hybrid Imaging 4, 24 (2020). https://doi.org/10.1186/s41824-020-00094-8

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

The authors have addressed the majority of my concerns adequately and the manuscript has significantly improved.

Reviewer 2 Report

Given the author's effort to address the reviewer's concern, the reviewer accepts the manuscript now.