An Institutional Febrile Neutropenia Protocol Improved the Antibacterial Treatment and Encouraged the Development of a Computerized Clinical Decision Support System

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Introduction:

1. The authors should more extensively describe the limitations of the existing guidelines, the reasons to implement the CDSS and its potential benefits.

2. line58: By the term "inappropriate test demands" do you mean tests not recommended by the international guidelines? How the CDSS would address this problem? Please explain more extensively

Line 82-83: "The study was approved by the Ethical Committee of Hacettepe University School of Medicine (Approval number and date: GO 20/445; dated 22.05.2020)." It should be clearly stated that patients' consent was sought for their participation in the study and the patient data were anonymized before analysis.

Methods

2.3Statistics

Line 121-123: "The primary outcome measure was the achievement of a 20% increase in the rate of appropriate empirical treatment of FN in bacteremic patients." The authors should explain why the specific threshold was chosen .

Results

The results are clearly reported. However, some results should be further explained, e.g in lines 132-133 "The demographic characteristics of the patients were similar except the duration of neutropenia". What are the possible implications of this difference in the outcomes?

Discussion

1. Lines 206-207 "In this study, we observed that the quality of antibacterial treatment for FN was improved after the introduction of a guideline provided in the local language (Turkish)".Please elaborate the reasons behind this improvement.

2. Potential limitations and biases should be more extensively analyzed.

3. Lines 249-250 "It is possible that some confounding variables may have been underestimated due to the retrospective design". It would be useful to present some confounding variables that could have affected the results.

Conclusions

Clearly presented

Comments on the Quality of English Language

Minor grammatical and syntax errors should be corrected.

Author Response

Dear Reviewer, thank you for your review and comments.

Below, you will find your comments followed by our responses in order.

Introduction:

Comments 1: The authors should more extensively describe the limitations of the existing guidelines, the reasons to implement the CDSS and its potential benefits.

As stated in the revised text (page 2, lines 78-122), while existing international guidelines are beneficial, they may be hindered by language barriers for some users. Furthermore, because these guidelines are developed without considering local epidemiological data, we believe that developing institution-specific guidelines—such as those implemented in our hospital—could lead to improvements. CDSS, on the other hand, may offer the potential to minimize errors that could arise from individuals misunderstanding the guidelines or from a lack of experience with the subject. If CDSS is integrated into the hospital electronic system, it has the potential to prevent inappropriate test demands, such as those where an unexperienced junior doctor might order all tests due to fear of missing necessary investigations for febrile neutropenia.

line58: By the term "inappropriate test demands" do you mean tests not recommended by the international guidelines? How the CDSS would address this problem? Please explain more extensively

Response: We mean tests not recommended by our local guideline (Such as galactomannan antigen from a patient under mold prophylaxis without suspicion of invasive aspergillosis or CMV PCR in all patients with neutropenia). We propose that CDSSs, which can be integrated into hospital electronic systems to guide test requests by unexperienced doctors, could be highly beneficial.

Line 82-83: "The study was approved by the Ethical Committee of Hacettepe University School of Medicine (Approval number and date: GO 20/445; dated 22.05.2020)." It should be clearly stated that patients' consent was sought for their participation in the study and the patient data were anonymized before analysis.

Response: We have stated and included this information (page 6, lines 198-203). The sentences we added to the text are also present below.

Data were used anonymously at all stages of the study. Informed consent was not required due to the retrospective nature of the study, in line with local regulations.

Methods

2.3Statistics

Line 121-123: "The primary outcome measure was the achievement of a 20% increase in the rate of appropriate empirical treatment of FN in bacteremic patients." The authors should explain why the specific threshold was chosen .

Response: It has been added to the relevant section in the revised text (page 10,  lines 350-354).

Results.

The results are clearly reported. However, some results should be further explained, e.g in lines 132-133 "The demographic characteristics of the patients were similar except the duration of neutropenia". What are the possible implications of this difference in the outcomes?

Response: It was discussed in the discussion section of the revised text ( page 15, lines 467-480).

Additionaly, ıf the demographic characteristics of the patients had differed, for example, if fluoroquinolone prophylaxis had been more common or if there had been a previous history of ICU admission, this would have increased the rate of broad-spectrum antibiotics. It would therefore be possible to ascertain whether the increased rate of appropriate empirical antibacterial treatment was a consequence of the local protocol or simply indicative of an increased use of broad-spectrum antibiotics.

Discussion

1. Lines 206-207 "In this study, we observed that the quality of antibacterial treatment for FN was improved after the introduction of a guideline provided in the local language (Turkish)".Please elaborate the reasons behind this improvement.

Response:  It was added to the discussion. The improvement was possibly inspiration of the guideline in the local language with continuous discussion as well as easy access from the hospital information system as we stated on page 4 (lines 123-131 and 151-159) and pages 14-15 (lines 444-480)

2. Potential limitations and biases should be more extensively analyzed.

Response: It was added to the limitations subsection (pages 15-16, lines 507-531).

3. Lines 249-250 "It is possible that some confounding variables may have been underestimated due to the retrospective design". It would be useful to present some confounding variables that could have affected the results.

Response: It was added to the limitations subsection (pages 15-16, lines 509-531).

 Since the study was retrospective, complete data for some patients were not accessible, and there is a possibility of inaccurate records. The number of patients included in the evaluation was affected by missing data in the assessment of adherence to antibiotic and antifungal prophylaxis, as well as by the fact that some patients were already under treatment when they developed FN, which may have influenced the adherence results. It is not possible to ascertain compliance with the local protocol with 100% accuracy since we were unable to include all patients with FN in the study period. The primary objective of the protocol was to enhance the quality of care when an ID consultant could not be present at the patient’s bedside. However, due to the limitations of the study, it was not feasible to categorize the treatment selections made at the bedside or during on-call shifts during night and weekend periods. We did not have sufficient data on antibiotic side effects across the different periods, which is one of this study's limitations. Local pathogen profiles and resistance profiles can vary between different hospitals and wards. This may influence the appropriateness of empirical treatment. Since this was a study conducted at a single center, the results cannot be extrapolated to other centers and settings. Additionally, blood culture volumes and the number of blood culture sets obtained are directly associated with the detection of bacteremia. The CDSS recommends performing two sets of blood cultures in accordance with the local guidelines, which were developed based on international guideline recommendations. Although it was recommended to obtain two sets of blood cultures in patients with FN, in most cases, only one set of blood cultures was obtained. Therefore, optimal blood culture efficacy might not have been achieved.

 

Conclusions

Clearly presented

 

Comments on the Quality of English Language

Minor grammatical and syntax errors should be corrected.

Response:  English editing services were obtained through MDPI, and the necessary revisions were made.

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for submitting your manuscript entitled "Developing an institutional febrile neutropenia protocol: A pathway to a computerised clinical decision support system." This work is timely and could significantly impact infection control practices within the field of infectious diseases.

 

Specific Comments

==============

Major Comments

---------------------

[Overall]

- The manuscript would benefit greatly from structural improvements. I recommend utilizing the STROBE checklist or similar frameworks to enhance clarity and organization. Almost all my comments would be resolved to use these checklists.

 

[Title]

- The title should be revised to better reflect the study design.

 

[Abstract]

- Please specify the release date of the CDSS along with the local guideline implementation.

 

[Introduction]

- The introduction requires significant expansion:

  - Include a more detailed historical overview of CDSS applications in managing infectious diseases and febrile neutropenia (FN).

  - Discuss international guidelines on FN to provide a comparative context.

  - Clearly articulate the knowledge gaps your study addresses and define specific aims of your research.

 

[Methods]

- Define "junior doctors" to clarify the roles and responsibilities within the study context.

- Mention the specific date of CDSS and local guideline implementation.

- Elaborate on data collection methods, particularly how empirical antibacterial treatments and infectious diagnoses were evaluated and validated.

- Include detailed inclusion and exclusion criteria, ideally presented in a PRISMA-like flowchart.

- Explain the local guidelines for international readers to understand the context better.

- Detail the usage frequency of the CDSS and methods used to assess how clinicians applied the recommendations.

- Define the primary outcomes of the study.

- Provide specifics on how blood cultures were performed, the devices used, and criteria used to distinguish between contamination and true bacteremia.

 

[Results]

- Discuss any side effects related to antibiotics used between two phases of the study (P1 and P2); if data are unavailable, acknowledge this as a limitation.

 

[Discussion]

- Justify why the CDSS recommends performing two sets of blood cultures.

 

Minor Comments

---------------------

[English Proficiency]

- The manuscript requires thorough proofreading to correct grammatical errors and enhance the readability. An example to consider revising for clarity and grammatical accuracy is: "Only 2 patients in P1 (2/91, 2.2%) had two sets of blood cultures, whereas this rate was significantly higher in P2 (34/136, 25.0%) (p<0.001)."

 

These comments aim to assist in refining your manuscript for potential publication. Please consider these suggestions carefully to ensure a comprehensive and clear presentation of your study's findings.

Comments on the Quality of English Language

Please refer previous quality of English language and my comments for authors.

Author Response

Dear Reviewer, thank you for your review and suggestions. 

Below, you will find your comments followed by our responses in order.

 

Comments and Suggestions for Authors

 

Thank you for submitting your manuscript entitled "Developing an institutional febrile neutropenia protocol: A pathway to a computerised clinical decision support system." This work is timely and could significantly impact infection control practices within the field of infectious diseases.

 

Specific Comments

==============

Major Comments

---------------------

[Overall]

- The manuscript would benefit greatly from structural improvements. I recommend utilizing the STROBE checklist or similar frameworks to enhance clarity and organization. Almost all my comments would be resolved to use these checklists.

 

[Title]

- The title should be revised to better reflect the study design.

Response: The title was revised according to your suggestion.

 

[Abstract]

- Please specify the release date of the CDSS along with the local guideline implementation.

Response: This was added to the abstract (page 2, lines 21-25) section.

[Introduction]

- The introduction requires significant expansion:

  - Include a more detailed historical overview of CDSS applications in managing infectious diseases and febrile neutropenia (FN).

Response: Efforts were made to expand the Introduction section (pages 3-4, lines 88-122).

  - Discuss international guidelines on FN to provide a comparative context.

Response: Comparing the current guidelines in all aspects of management of FN is out of the scope of this study. We tried to show some samples of the differences in the guidelines (Pages 1-2, lines 43-77). We referred to a recent review that can provide the scope you mentioned for readers(Keck et al, reference number 1).

 

  - Clearly articulate the knowledge gaps your study addresses and define specific aims of your research.

Response: In this study, we aimed to investigate the impact of a written guideline in the local language (Turkish) on the quality of the management of febrile neutropenia. We then investigated the applicability of the CDSS in the management of febrile neutropenic patients, focusing on those with bacteraemia.

As stated in the revised text (page 2, lines 78-122), while existing international guidelines are beneficial, they may be hindered by language barriers for some users. Furthermore, because these guidelines are developed without considering local epidemiological data, we believe that developing institution-specific guidelines—such as those implemented in our hospital—could lead to improvements. CDSS, on the other hand, may offer the potential to minimize errors that could arise from individuals misunderstanding the guidelines or from a lack of experience with the subject. If CDSS is integrated into the hospital electronic system, it has the potential to prevent inappropriate test demands, such as those where an inexperienced junior doctor might order all tests due to fear of missing necessary investigations for febrile neutropenia. 

[Methods]

- Define "junior doctors" to clarify the roles and responsibilities within the study context.

Response: We mentioned this on page 4,  lines 140-153.

- Mention the specific date of CDSS and local guideline implementation.

Response: This was added to the abstract (page 2, lines 21-25) and methods ( page 4, lines 154-155  page 8, lines 290-291) sections.

- Elaborate on data collection methods, particularly how empirical antibacterial treatments and infectious diagnoses were evaluated and validated.

Response: In the Introduction and Materials and Methods section, the process of data collection, as well as the criteria used in diagnosis and treatment, were detailed (page 5, lines 182-197 and pages 6-7, lines 205-283)

- Include detailed inclusion and exclusion criteria, ideally presented in a PRISMA-like flowchart.

Response: Therefore, Figure 1 was created and added to the revised text (page 5).

- Explain the local guidelines for international readers to understand the context better.

Response: Detailed additions have included on pages 6 and 7 in the revised text.

- Detail the usage frequency of the CDSS and methods used to assess how clinicians applied the recommendations.

Response: The CDSS was not used in the daily practice. The number of access to the CDSS for the trial was 50 for each participant (pages 9-10, lines 325-337)

- Define the primary outcomes of the study.

Response: The primary outcome was defined as 20% increase in the rate of appropriate empirical antibacterial treatment  (page 10, lines 350-354)

- Provide specifics on how blood cultures were performed, the devices used, and criteria used to distinguish between contamination and true bacteremia.

Response: It has been added to the revised text (page 7, lines 272-283).

During the study period, blood cultures were performed by using Bactec Plus Aerobic/F bottles (aerobic bottles) and Bactec Lytic/10 Anaerobic/F bottles (anaerobic bottles) incubated in the Bactec FX system. The species identification was done using the matrix‐assisted laser desorption/ionization time-of‐flight mass spectrometry (MAL-DI-TOF MS. Bruker. Germany). Antimicrobial susceptibility tests were performed on a Phoenix system (Becton-Dickinson Diagnostic Systems. Sparks. MD) and interpreted in accordance with the European Committee on Antimicrobial Susceptibility Testing criteria (https://www.eucast.org/clinical_breakpoints). All positive blood cultures with Gram negative bacilli, S. aureus or Enterococcus species were considered as true bacteremia. Blood cultures with coagulase negative staphylococci, diphtheroid, or viridans streptococci were considered as true bacteremia if two blood cultures that were obtained in 24-hours yielded the same bacteria.

[Results]

- Discuss any side effects related to antibiotics used between two phases of the study (P1 and P2); if data are unavailable, acknowledge this as a limitation.

Response: We did not have sufficient data on antibiotic side effects across the different periods, which is one of the study's limitations. It was included in the limitations section (page 16, lines 520-521).

[Discussion]

- Justify why the CDSS recommends performing two sets of blood cultures.

Response: To achieve an optimal volume and to mitigate false-positives triggered by contamination., 2 cultures are drawn for each collection from two separate insertion sites. The CDSS recommends performing two sets of blood cultures in accordance with local guidelines, which were developed based on international guideline recommendations.

 

Minor Comments

---------------------

[English Proficiency]

- The manuscript requires thorough proofreading to correct grammatical errors and enhance the readability. An example to consider revising for clarity and grammatical accuracy is: "Only 2 patients in P1 (2/91, 2.2%) had two sets of blood cultures, whereas this rate was significantly higher in P2 (34/136, 25.0%) (p<0.001)."

Response: The sentence has been revised (page 12, lines 389-392).

These comments aim to assist in refining your manuscript for potential publication. Please consider these suggestions carefully to ensure a comprehensive and clear presentation of your study's findings.

 

 

Comments on the Quality of English Language

Please refer previous quality of English language and my comments for authors.

Response: English editing services were obtained through MDPI, and the necessary revisions were made.

Reviewer 3 Report

Comments and Suggestions for Authors

Taş and colleagues examined the impact of implementing a local guideline on the management of febrile neutropenia and the effectiveness of a computerized decision support system (CDSS) using real-life data from 227 patients treated between April 2016 and January 2019. The primary goal was to achieve a 20% increase in the rate of appropriate empirical treatment for bacteremic FN patients. Researchers compared outcomes before (P1: 91 patients) and after (P2: 136 patients) the guideline's release, finding that appropriate empirical treatment increased significantly from 58.3% in P1 to 88.1% in P2. Additionally, the need for escalation of antibacterial treatment decreased significantly in P2. The CDSS showed high compliance with the local guidelines, performing similarly to consulting physicians in delivering appropriate empirical antibacterial treatment.

The use of CDSS also proved to be a valuable tool, with its performance closely aligning with that of consulting physicians, suggesting that it can reliably support clinical decision-making. 

However, the study has many limitations (mostly addressed by the authors), and they could have included a longer follow-up period to assess the sustainability of these improvements and evaluate the potential impact on patient survival rates.

Line 63: Please cite the document without including the link, as the citation itself suffices.

Tables 1, 2, and 4: Remove unnecessary bullet points to enhance clarity and streamline the presentation of data.

Line 210: Remove the extra space to ensure consistent formatting throughout the document.

Since FN has already been defined earlier in the text, there is no need to mention "febrile neutropenia" in full repeatedly. Use "FN" consistently for conciseness.

Regarding the four-year gap in publishing this work, it's worth addressing the factors that contributed to this delay. Understanding these reasons can help in identifying potential areas for process improvement in future research endeavors. Possible explanations might include challenges in data collection, extended periods of peer review, or revisions that required additional analysis. Providing a brief rationale for the delay could offer valuable context to the readers and reviewers.

Author Response

Dear Reviewer, thank you for your review and suggestions.

Below, you will find your comments followed by our responses in order.

Taş and colleagues examined the impact of implementing a local guideline on the management of febrile neutropenia and the effectiveness of a computerized decision support system (CDSS) using real-life data from 227 patients treated between April 2016 and January 2019. The primary goal was to achieve a 20% increase in the rate of appropriate empirical treatment for bacteremic FN patients. Researchers compared outcomes before (P1: 91 patients) and after (P2: 136 patients) the guideline's release, finding that appropriate empirical treatment increased significantly from 58.3% in P1 to 88.1% in P2. Additionally, the need for escalation of antibacterial treatment decreased significantly in P2. The CDSS showed high compliance with the local guidelines, performing similarly to consulting physicians in delivering appropriate empirical antibacterial treatment.

The use of CDSS also proved to be a valuable tool, with its performance closely aligning with that of consulting physicians, suggesting that it can reliably support clinical decision-making. 

However, the study has many limitations (mostly addressed by the authors), and they could have included a longer follow-up period to assess the sustainability of these improvements and evaluate the potential impact on patient survival rates.

Line 63: Please cite the document without including the link, as the citation itself suffices.

Response: Corrected.

Tables 1, 2, and 4: Remove unnecessary bullet points to enhance clarity and streamline the presentation of data.

Response: Corrected.

Line 210: Remove the extra space to ensure consistent formatting throughout the document.

Response: Corrected.

Since FN has already been defined earlier in the text, there is no need to mention "febrile neutropenia" in full repeatedly. Use "FN" consistently for conciseness.

Regarding the four-year gap in publishing this work, it's worth addressing the factors that contributed to this delay. Understanding these reasons can help in identifying potential areas for process improvement in future research endeavors. Possible explanations might include challenges in data collection, extended periods of peer review, or revisions that required additional analysis. Providing a brief rationale for the delay could offer valuable context to the readers and reviewers.

Response: This study was published as Dr. Taş's (first author) specialization thesis in December 2021 in the National Thesis Center. At the time of the thesis publication, the COVID-19 pandemic was intensifying both in our country and at our hospital. In February 2022, Dr. Taş was assigned to another city for obligatory service, and due to the workload from the COVID-19 pandemic and a subsequent loss of motivation from the obligatory service, the study could not be converted into a manuscript. After the manuscript was prepared, we submitted it to several journals. However, as you may also know, the manuscript review process is not always as quick and efficient as it is here (some journals can even delay a decision for 2-3 months just to reject the manuscript without review). These circumstances have caused the delay. Thank you for your valuable feedback and comments. 

 

 

Reviewer 4 Report

Comments and Suggestions for Authors

The study examined the impact of a local guideline on febrile neutropenia management and the effectiveness of a computerized decision support system (CDSS) in bacteremic patients. Results showed that the introduction of a local guideline improved treatment quality and reduced escalation rates in FN patients. The high compliance rate with CDSS-based decisions in P2 demonstrates the usefulness of CDSS for these patients. Certain issues must be rectified before making a final decision on publication.

1. The authors should explicitly clarify how the CDSS performs and include a diagram depicting the decision-making process. 

2.  They should provide a clear and concise description of how the principle of antibiotic escalation is applied in the therapy of febrile neutropenia, following the guidelines of CDSS

3. Table 1 shows that the length of neutropenia in the second period is longer (7 days) compared to the first period (6 days), and the P value is less than 0.05. Was there evidence of a selection bias?

4. Please clarify the sentence in Lines 155-156. “Only 2 patients 155 (2/91, 2.2%) 1 in P1 had two sets of blood cultures where this rate was much higher in P2 156 (34/136, 25.0%) (p<0.001).”

Comments on the Quality of English Language

The quality of English can be enhanced. 

Author Response

Dear Reviewer, thank you for your review and suggestions.

Below, you will find your comments followed by our responses in order.

Comments and Suggestions for Authors

The study examined the impact of a local guideline on febrile neutropenia management and the effectiveness of a computerized decision support system (CDSS) in bacteremic patients. Results showed that the introduction of a local guideline improved treatment quality and reduced escalation rates in FN patients. The high compliance rate with CDSS-based decisions in P2 demonstrates the usefulness of CDSS for these patients. Certain issues must be rectified before making a final decision on publication.

1. The authors should explicitly clarify how the CDSS performs and include a diagram depicting the decision-making process. 

Response: Therefore, Figure 2 was created and added to the revised text (page 9 ). Also, screenshots from the CDSS during the implementation of different clinical scenarios are presented in Supplementary File 1. 

2. They should provide a clear and concise description of how the principle of antibiotic escalation is applied in the therapy of febrile neutropenia, following the guidelines of CDSS

Response: The principle of antibiotic escalation in the therapy of febrile neutropenia was included in the Methods section. The CDSS was not used in the daily practice. The number of access to the CDSS for the trial was 50 for each participant (pages 9-10, lines 325-337).

3. Table 1 shows that the length of neutropenia in the second period is longer (7 days) compared to the first period (6 days), and the P value is less than 0.05. Was there evidence of a selection bias?

Response: We mentioned this in the discussion section (page 15, lines 467-472). Patients included in the analysis were randomly selected by the infection control committee during the specified periods to assess adherence to febrile neutropenia protocols. Despite the lack of randomization, demographic characteristics were similar between P1 and P2  except for the neutropenia duration. This indicates that relatively comparable groups were evaluated for guideline adherence before and after its implementation. 

4. Please clarify the sentence in Lines 155-156. “Only 2 patients 155 (2/91, 2.2%) 1 in P1 had two sets of blood cultures where this rate was much higher in P2 156 (34/136, 25.0%) (p<0.001).”

Response: The sentence has been revised (page 12, lines 389-392).

 

Comments on the Quality of English Language

The quality of English can be enhanced. 

Response: English editing services were obtained through MDPI, and the necessary revisions were made.

 

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

General Comments

==============

Thank you for submitting your manuscript entitled "Developing an institutional febrile neutropenia protocol: A pathway to a computerised clinical decision support system." This work is timely and could significantly impact infection control practices within the field of infectious diseases. Almost all responses were reasonable.

Reviewer 4 Report

Comments and Suggestions for Authors

The study investigated the effects of a local guideline on the management of febrile neutropenia and the efficacy of a computerized decision support system (CDSS) in patients with bacteremia. The study found that the implementation of a local guideline had a positive impact on the quality of treatment and led to a decrease in escalation rates among patients with FN. The high level of compliance with CDSS-based decisions in P2 highlights the effectiveness of CDSS for these patients. All of the reviewer's questions have been resolved. The English editing services were acquired from MDPI, and the required revisions were expertly executed. 

1.     A new figure was included in the revised text on page 9. In addition, Supplementary File 1 includes screenshots from the CDSS showcasing the implementation of various clinical scenarios.  

2.     The principle of antibiotic escalation in the therapy of febrile neutropenia was mentioned in the Methods section. The CDSS was not utilized in the daily routine. Each participant had 50 access to the CDSS for the trial, as indicated on pages 9-10, lines 325-337.

3.     The authors discussed this in the discussion section on page 15, lines 467-472. Patients included in the analysis were randomly selected by the infection control committee during the specified periods to evaluate compliance with febrile neutropenia protocols. In terms of randomization, it should be noted that the demographic characteristics of P1 and P2 were quite similar, except for the duration of neutropenia. This suggests that similar groups were assessed for adherence to the guidelines both before and after its implementation.

4.     The sentence in Lines 155-156. “Only 2 patients 155 (2/91, 2.2%) 1 in P1 had two sets of blood cultures where this rate was much higher in P2 156 (34/136, 25.0%) (p<0.001).” The sentence has been revised (page 12, lines 389-392).

Comments on the Quality of English Language

The English editing services were obtained from MDPI, and the necessary revisions were skillfully carried out.