Real-World Treatment Patterns and Clinical Outcomes among Patients Receiving CDK4/6 Inhibitors for Metastatic Breast Cancer in a Canadian Setting Using AI-Extracted Data

Round 1

Reviewer 1 Report

In this manuscript, the authors show real-world treatment patterns and clinical outcomes in a Canadian group of patients who received CDK4/6 Inhibitors for Metastatic Breast Cancer by using AI-extracted data. In general, this manuscript is well written, and the topic is interesting. Also, the patients could greatly benefit from these results.

The introduction section is well written and provides enough information about the testing the authors want to perform.

The results section provides well-thought-out tables but the description of the results in this section is a bit sparse. Actually, the discussion section contains some parts that describe the results much better. Please consider improving this section.

The discussion section discusses in depth authors´ results but doesn’t make many comparisons with results from other authors. The discussion would greatly improve if the authors could describe other reports made with the same technology but applied on other types of cancer/disease or in another patient group / another country, for example. Do they find the same types of limitations/advantages or are they different. What could this be contributed to….etc.

The final conclusion of this manuscript could be improved as it is the highlight of this entire paper and also the most important message the authors want to communicate.

Author Response

Reviewer 1

In this manuscript, the authors show real-world treatment patterns and clinical outcomes in a Canadian group of patients who received CDK4/6 Inhibitors for Metastatic Breast Cancer by using AI-extracted data. In general, this manuscript is well written, and the topic is interesting. Also, the patients could greatly benefit from these results.

 The introduction section is well written and provides enough information about the testing the authors want to perform.

The results section provides well-thought-out tables but the description of the results in this section is a bit sparse. Actually, the discussion section contains some parts that describe the results much better. Please consider improving this section.

The discussion section discusses in depth authors´ results but doesn’t make many comparisons with results from other authors. The discussion would greatly improve if the authors could describe other reports made with the same technology but applied on other types of cancer/disease or in another patient group / another country, for example. Do they find the same types of limitations/advantages or are they different. What could this be contributed to….etc.

• Thank you for your thorough review of the manuscript, and the insightful feedback provided. We are delighted to learn that you found the introduction to be well written and the tables in the results section to be well thought out.
• Further, we appreciate your suggestion to improve the description of the results and include a comparison of similar results from other authors. In response to your suggestions, we have updated the results section to include more detailed description of the results:
• Lines 159–173: “An F1 score (harmonic mean of precision and recall) of 1.00 was achieved for three features: histology, ER receptor status and PR receptor status, and an overall accuracy (number of correctly identified predictions) of above 90% was achieved for all AI-extracted features. These results are consistent with previous validations of DARWEN^TM [19, 24]. Radiation treatment, date of ABC/MBC diagnosis and treatment start/stop date) were extracted manually, due to limitations imposed by the data captured in the EHR. Radiation treatment is administered at sites outside of Sinai Health, therefore information on patient’s radiation therapy was not consistently captured in the Sinai Health patient EHR. Date of ABC/MBC diagnosis is also often inconsistently reported in patient’s EHR, with ABC/MBC diagnosis often being reported as suspicious but not confirmed. Additionally, patients were often diagnosed with ABC/MBC at other sites and referred to Sinai Health. Prescription information is not stored electronically in the EHR system at Sinai Health, but rather in paper format, dictated into clinical notes. Before data extraction using either method, the pre-defined rules and definitions for each clinical feature were finalized with the Sinai Health PI (Supplementary Table S1).”
• Additionally, we have updated the discussion section to include comparisons to other reports with the same technology in terms or limitations/advantages:
• Line 281–285: “While AI holds immense promise in improving cancer diagnosis, treatment, and outcomes, it is important to recognize challenges and limitations of the technology, specifically related to accuracy and precision. AI algorithms are only as reliable as the data they are trained on, and biases in training data can lead to inaccurate outcomes, particularly in underrepresented populations.”
• Line 300–303: “These limitations are consistent with previous applications of AI tools for the extraction of oncology EHR data, but it is important to note these limitations also impact manual cu-ration of data, highlighting a broader limitation in generating RWE from EHR systems [19, 46].”
• Line 328–332: “Additionally, validation metrics for AI-extracted data are consistent with previous validations of DARWEN^TM, which has been evaluated against manual abstraction for the same clinical features in breast cancer [26], lung cancer [24, 31–35], ambulatory care diseases [24], and dermatology [29] at multiple Canadian institutions.”

The final conclusion of this manuscript could be improved as it is the highlight of this entire paper and also the most important message the authors want to communicate.

• Thank you, we have updated the conclusion to include the most important messages from the study:
• Line 333–341: “This study highlights the validity of AI technology in identifying patients with HR+/HER2- ABC/MBC and generating RWE including treatment patterns and clinical outcomes for patients. This type of technology allows more efficient, consistent, and scalable extraction of data from EHR systems. AI was used to extract nine crucial features from the patient EHR, which were validated and reviewed by a breast cancer expert, and accuracy metrics were consistent with previous validations of the AI technology. Results from this study demonstrate the effectiveness of CDK4/6i+ET, in the Canadian real-world 1L, with most patients receiving palbociclib as CDK4/6i in 1L, over a longer follow-up period than previous real-world Canadian studies.”

Reviewer 2 Report

1) Give a more detailed description of dataset

2) It is recommended to add a comparison with previous studies (in tabulated form) by the end of the discussion section

Author Response

Reviewer 2

• Give a more detailed description of dataset

• Thank you for your comment. We have updated the results section accordingly, incorporating a more detailed description of the dataset:
• Line 180–193: “Baseline characteristics for the 48 included patients can be found in Table 1. In this cohort, median age was 60.5 years. The majority of patients (70.8%) had recurrent ABC/MBC and 29.2% had de novo disease; 66.7% of patients had ductal carcinoma and 18.8% were pre-menopausal. 31.2% of patients presented with bone-only metastases at ABC/MBC diagnosis. 39.6% of patients had lung metastases during the study period and 37.5% had liver metastases during the study period. 45.8% of patients had 1 metastatic site during the study period. Of patients with reported Eastern Cooperative Oncology Group (ECOG) performance scores at diagnosis (22/48), the majority had an ECOG score of 0/1 (18/22 [81.8%]). At ABC/MBC diagnosis, the most common comorbidity was hypertension (37.5%) followed by diabetes (14.6%). Tumour grade at ABC/MBC diagnosis was not consistently reported across patients, with 29/48 (60.4%) missing tumour grades at ABC/MBC diagnosis. Of the 48 patients, 38 received a CDK4/6i in the 1L setting. Baseline demographics for the 38 patients who received a CDK4/6i in 1L were similar to the full patient cohort (Table 1).”

It is recommended to add a comparison with previous studies (in tabulated form) by the end of the discussion section

• Thank you for this suggestion. As per this suggestion and the other reviewer suggestions, we have updated the discussion section to include a more detailed review of the literature in terms of AI in healthcare, and compared the findings presented in this study to other applications of this technology.

Reviewer 3 Report

This study utilized artificial intelligence (AI) to extract data on patient treatment patterns and outcomes for those receiving CDK4/6 Inhibitors for Metastatic Breast Cancer in Canada. While I advocate for the utilization of AI in cancer therapy analysis, I do harbor a few minor concerns regarding this study:

1. Given the novelty of AI in the realm of cancer therapy and research, it would be beneficial for the authors to provide a brief introduction highlighting recent advancements in AI's application in radiotherapy. Incorporating references such as Siddique et al (Prep Pract Oncol Radiother 2020;25:656) could enrich the discussion.

2. The study should address the limitations associated with AI employment in cancer research, including considerations of accuracy, precision, and ethical implications, within the context of the current investigation.

No problem for the English.

Author Response

Reviewer 3

This study utilized artificial intelligence (AI) to extract data on patient treatment patterns and outcomes for those receiving CDK4/6 Inhibitors for Metastatic Breast Cancer in Canada. While I advocate for the utilization of AI in cancer therapy analysis, I do harbor a few minor concerns regarding this study:

Given the novelty of AI in the realm of cancer therapy and research, it would be beneficial for the authors to provide a brief introduction highlighting recent advancements in AI's application in radiotherapy. Incorporating references such as Siddique et al (Prep Pract Oncol Radiother 2020;25:656) could enrich the discussion.

• We completely agree with your suggestion to incorporate a brief overview of recent advancements in AI in healthcare, including radiotherapy and have updated the discussion section accordingly:
• Line 263–271: “Recently, much progress has been made in the implementation of AI tools in healthcare, including assisting radiologists in detecting abnormalities and disease from X-rays, MRIs, and CT scans, personalized medicine and predicting which treatments are likely to benefit a patient, clinical decision support systems and AI-remote monitoring and telemedicine platforms.[38–40] Additionally, AI tools for extraction of clinical text can make sense of and analyze vast amounts of unstructured clinical text from pathology reports, clinical notes, and radiology reports. These tools, such as DARWENTM, are being used for patient and disease identification, pharmacovigilance, and development of learning health systems [24–29].”

The study should address the limitations associated with AI employment in cancer research, including considerations of accuracy, precision, and ethical implications, within the context of the current investigation.

• Thank you, we acknowledge the importance of open and transparency in relation to the limitations of AI technology in cancer research, and appreciate the suggestion to include these in this manuscript, as such we have updated the discussion section:
• Line 281–285: “While AI holds immense promise in improving cancer diagnosis, treatment, and outcomes, it is important to recognize challenges and limitations of the technology, specifically related to accuracy and precision. AI algorithms are only as reliable as the data they are trained on, and biases in training data can lead to inaccurate outcomes, particularly in underrepresented populations. In the context of this study…”
• Line 300–303: “These limitations are consistent with previous applications of AI tools for the extraction of oncology EHR data, but it is important to note these limitations also impact manual cu-ration of data, highlighting a broader limitation in generating RWE from EHR systems [19, 46].”

Round 2

Reviewer 2 Report

Thank you for your response

Reviewer 3 Report

I am satisfied with the modifications from the authors as per my comments.

no comment