Assessing Diagnostic Performance of Molecular Culture for Neonatal Sepsis: Protocol of the CHAMPIONS Study

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This protocol is well described and organized , however 

 neonatal sepsis is restricted to neonatal age group < 28 days and not 90 days .(line 137)

Researchers  mentioned [ international multicenter study ] , but participants from 11 centers in Netherlands 

Authors can add the expected benefits in terms of duration of test and cost .

Good luck with the study 

 

Author Response

Comment 1: neonatal sepsis is restricted to neonatal age group < 28 days and not 90 days .(line 137)

Response 1: We thank the reviewer for this comment. We agree that the upper threshold of 28 days is commonly used for late onset sepsis. However, there are various studies that use 90 days as a threshold (e.g. PMID: 38146858). Particularly for (extremely) preterm infants, the occurrence of culture confirmed sepsis remains common beyond 28 days of age and is often considered the result of ongoing risk factors often associated with sepsis, such as hospital admission, central catheters in situ , episodes of necrotizing enterocolitis and immune system immaturity. We chose this wider definition of late onset sepsis based on the notion that test performance of Molecular Culture will not be different under the condition of >28 and <90 days day old infant sepsis evaluations, as well as the fact that sepsis evaluations at e.g. day 60 of infants born at GA 24-28 weeks may still yield positive cultures and therefore lend support to answering our research questions. For the chosen outcomes in our study, we see no reason why this definition would harm or skew our analysis. More so, the distribution of causative pathogens will shift somewhat with older infants, further strengthening results by indicating diagnostic performance for a more diverse landscape of bacterial species. Moreover, we will differentiate between any (suspected) community or nosocomial infections in the case of infants who had not been admitted prior to their evaluation.  

Comment 2: Researchers  mentioned [ international multicenter study ] , but participants from 11 centers in Netherlands 

Response 2: We thank the reviewer for this comment. This error has been corrected. We have previously been in contact with centers outside of the Netherlands for study participation but have found this to be too logistically and financially challenging at this point.

Comment 3: Authors can add the expected benefits in terms of duration of test and cost.

Response 3: We thank the reviewer for this comment. We have written an additional section to comment on this in lines 291-300. We do however feel it is too premature to make more firm estimations on expected benefits of potentially decreased turnaround times and cost. This holds especially true, since there is no information yet regarding cost price of Molecular Culture for this indication, neither have there been prospective studies that assess actual reductions of turnaround time in clinical practice of Molecular Culture. There is, to our knowledge a deficit of studies on the effects of implementing any molecular assay (e.g. multiplex PCR, 16S sequencing) for the high speed detection and identification of bacteremia for neonatal sepsis, with regards to cost efficiency and the extent of preventing unnecessary treatment or de-escalating earlier. We found 1 study (PMID: 27472282) that showed that for 22% of infants evaluated for late onset sepsis by using both conventional culture and  multiplex PCR, length of treatment was impacted, with PCR result availability after 24-48 hours. However, we feel this is insufficient to make a proper estimation of expected benefit of Molecular Culture.

Reviewer 2 Report

Comments and Suggestions for Authors

I have read this protocol with great interest. My main concern relates to the a priori belief of performance, as only sensitivity, but not specificity will be explored.

Have the authors considered aspects related to specificity in this trial. 

Secondly, the 'performance' of the PCR will also depend on how the samples are collected, including but not limited to blood volume for the blood culture (cf Selonka papers). Has this been standardized and are HC volume collected, registered (like eg weighted). This is perhaps even more relevant because of the molecular culture applied, and might be a major issue if this study has the intention to be used in a med dev registration. 

Where will data be collected, 'only' in NICU, level 2/3, or also in pediatric wards and or ER departments (as you threshold is 90 days)

I'm somewhat surprise that there are (at least based on the current affilliations, no infectious specialists or neonatologists involved, while i assume that there has been interaction with these HCP to develop the protocol. Can you also be explicit about any patient/parent/public involvement ? 

Have you considered the core outcome datasets as suggested for neonatal and pediatric infections (could be useful at least as background data collection) (like PMID 38051733)

(i'm not involved in any of the references suggested)

 

minor

please check 'international', as this seems to be a study in the Netherlands ? 

how will you define agreement ? and who will define agreement. LIke, e.g. the PCR shows 2 potential pathogens, the BC one ? or none ? 

What are next steps ? the figure 1 suggests that PCR analysis can be done in 'some hours', but this study is based on collected samples, so retrospective. How does the product development plan looks like ? 

Line 218: what do you mean with 'may': will this be based on expert assessment ? 

Author Response

Comment 1: Have the authors considered aspects related to specificity in this trial. 

Response 1: We thank the reviewer for this comment. We have definitely considered specificity as this is crucial for the clinical value of the test and will calculate test specificity for both conventional culture results as well as clinical sepsis. We realize this was not sufficiently clear in the manuscript and have written an additional statement on this in lines 135-138. It may have seemed as though we would only focus on test sensitivity, as may be deducted from the power analysis, but this is not the case. We have however used the hypothesized sensitivity as the benchmark for power analysis because specificity will not be the limiting factor in calculating power, since test negatives will most certainly be more prevalent than test positives for both Molecular Culture as conventional culture, purely based on incidence. 

Comment 2: Secondly, the 'performance' of the PCR will also depend on how the samples are collected, including but not limited to blood volume for the blood culture (cf Selonka papers). Has this been standardized and are HC volume collected, registered (like eg weighted). This is perhaps even more relevant because of the molecular culture applied, and might be a major issue if this study has the intention to be used in a med dev registration. 

Response 2: We thank the reviewer for this comment. We do not seek med device registration, as the technique already obtained a CE-IVD label. Blood volumes will be registered and included as a variable , which was mentioned in line 165. Volume instructions are standardized for this study, although it is inherent to pediatric practice that more often than not, the desired blood volume cannot be collected, especially from infants. The procedure of blood sample collection will follow the exact same steps as blood sampling for conventional culture, implying aseptic conditions and use of either a sterile syringe to vacuum blood from a newly placed peripheral or central catheter, followed by needle replacement and injection into EDTA tubes, or alternatively a vacutainer system that allows direct collection into an EDTA bottle. We were not able to find the Selonka papers, could the author, if additional questions exist, provide us with a PMID or doi?   

Comment 3: Where will data be collected, 'only' in NICU, level 2/3, or also in pediatric wards and or ER departments (as you threshold is 90 days)

Response 3: we thank the reviewer for this comment and have added a remark to clarify this in line 121-122

Comment 4: I'm somewhat surprise that there are (at least based on the current affilliations, no infectious specialists or neonatologists involved, while i assume that there has been interaction with these HCP to develop the protocol. Can you also be explicit about any patient/parent/public involvement ? 

Response 4: It may have been insufficiently clear from the author raster to the reviewer, but dr Niemarkt is a certified neonatologist and Dr van der Kuip is a certified pediatric infectious disease specialist. Dr Bos and Budding are medical microbiologists. We therefore believe our body of authors represents a diverse team with expertise on all necessary levels. We have, at this stage, not included any patient / parent or public stakeholders in deciding on methodology but do plan on this in future studies when actual implementation is the study aim. A statement on patient/parent/public involvement has been added in lines 126-128.

Comment 5: Have you considered the core outcome datasets as suggested for neonatal and pediatric infections (could be useful at least as background data collection) (like PMID 38051733)

Response 5: We thank the author for this comment. We have considered these most definitely. However the COS the reviewer cites here is still under development. There is a modest abundance of clinical sepsis criteria available in published studies. However, lack of agreement on which one is superior is currently lacking, which becomes painfully clear from PMID: 34743180. For this exact reason, we chose to employ multiple definitions as well as prospectively indicate that any future definitions may be employed pending publications during study time frame, such as the COS mentioned. We cannot however predict whether we would have all the necessary datapoints available to meet any future definition of which the individual variables are as of yet unknown.

Comment 6: please check 'international', as this seems to be a study in the Netherlands ?

Answer 6: We thank the reviewer for this comment and have adapted accordingly. We were previously in talks to include centers outside of the Netherlands but have for logistical and financial reasons not to do so at this point.

Comment 7: how will you define agreement ? and who will define agreement. LIke, e.g. the PCR shows 2 potential pathogens, the BC one ? or none ?

Response 7: We thank the reviewer for this comment and agree that this is a complicated issue. As outlined in lines 270-272, we intend to use a step up method for assessing agreement of MC results with conventional culture results. The first step will imply assessing agreement of test positivity and negativity, regardless of species determination. In other words, to what extent are conventional culture tests positive and negative and to what extent are Molecular Culture results positive and negative. The second step will assess species agreement. If on any level (positivity/negativity; species differentiation; monomicrobial/polymicrobial) there are discrepancies between the conventional culture result and the Molecular Culture results, we will report on this. Depending on the amount of discrepancies, this will be done either as embedded case series (in case of small amount of discrepancies) or crosstabulated (in case of larger amount of discrepancies with the full data published as supplementary content). With regards to the definition of agreement, both test positivity and negativity, as well as species determination are easy to assess when comparing conventional culture and Molecular Culture. This becomes more complex in the case of mono- and polymicrobial cultures. We have therefore added 'Test agreement definition' in lines 225-233. As for estimating which one of either tests shows a higher agreement with actual disease in case of discrepancies, we aim to do that by using the secondary outcome, which is highly important in the case of test result discrepancies.

Lastly, Molecular Culture produces relative fluorescence units for any detected species. This measure could be used as a proxy for bacterial load but since we have no reference standards for this, we will not include it in any formal analysis, but may apply post hoc criteria to assess its validity against colony counts in conventional culture.

Comment 8: What are next steps ? the figure 1 suggests that PCR analysis can be done in 'some hours', but this study is based on collected samples, so retrospective. How does the product development plan looks like ? 

Response 8: We thank the reviewer for this comment. For this study, we only aim to assess diagnostic performance with regards to sensitivity / spec / PPV / NPV. This includes temporary storage of samples and offsite (retrospective) laboratory analysis, as the reviewer already mentioned. If the results of this (retrospective) design are sufficiently promising, we would proceed to a study in which we implement the test in onsite microbiology laboratories to assess both the actual turnaround time in clinical practice, as well as consequences for decision making. The estimation of several hours is purely based on the pipeline of the developer, future studies will have to assess performance onsite, actual turnaround time, as opposed to ideal turnaround time and consequences for decision making.   

Comment 9: Line 218: what do you mean with 'may': will this be based on expert assessment ? 

Response 9: We thank the reviewer for this comment. We think the line order is different in the version the reviewer received, since there is no 'may' in the line 218 of the document we possess. But we suppose the reviewer is hinting towards the following phrase:

For LOS, CoNS may always be considered pathogenic

We understand the confusion regarding this topic and state the following.

1. Whenever there is agreement between the conventional culture results and the Molecular Culture as per the outlined ‘agreement’ definition (line 225-233), we will not attempt to dissect any further whether these results are clinically appropriate. This is the primary outcome. That is to say, in the case where conventional culture shows a species that could be considered contaminant, e.g. CoNS in term infants without central catheters, which also shows in the Molecular Culture results, it is substantiated the Molecular Culture either shows the same pathogen or contaminant species as the conventional culture and this implies adequate test performance. Contamination as a result of insufficient aseptic conditions or low load non pathogenic bacteremia that shows in both conventional and Molecular Culture, can not be considered the result of inferior performance.
2. When there is disagreement between Conventional culture and Molecular Culture, such as one being positive and the other negative or showing different species, both test results will be cross tabulated against the various clinical sepsis definitions. We can then deduct conclusions based on the best agreement of either one test with these clinical definitions. This exonerates us from choosing 1 hard definition of clinical sepsis, because this is potentially more flawed.

Reviewer 3 Report

Comments and Suggestions for Authors

thankyou for the opportunity to review your manuscript

introduction. An interesting hypothesis that requires a robust justification in your introduction. You mention the risk profile in general for antibiotic v no antibiotic. Your study is looking at de-escalating empiric antibiotics - is there any evidence in the literature around benefits of de-escalating early?

your last paragraph reads like a conclusion and doesnt actually 'introduce' the paper ie we propose a protocol for...

materials and methods. thorough and well written. will you be looking at clinical outcomes as well eg antibiotics prescribed, empiric vs direct, length of time on antibiotics, how unwell the patient was of the period of hospitalisation (eg admission to ICU, artificial ventilation etc). 

discussion. the primary outcome is quite simple and elegant, i would like to see the discussion explore the implications of the suspected outcome, what do you hope to achieve? what are the risks involved with implementing? eg critically unwell are likely to stay on empirics much longer despite the MC results suggesting a narrowing focus is ok...

 

 

 

 

Author Response

Comment 1: introduction. An interesting hypothesis that requires a robust justification in your introduction. You mention the risk profile in general for antibiotic v no antibiotic. Your study is looking at de-escalating empiric antibiotics - is there any evidence in the literature around benefits of de-escalating early?

Response 1: We thank the reviewer for this comment. Obvious benefits exist such as that shorter therapy frequently implies shorter hospital stay, which naturally only goes for term infants with no other health issues. This reduces health care cost and increases parent child bonding. For preterm infants, reducing unnecessary stress is beneficial. In the case that therapy can be discontinued, this implies a decrease in catheter placements to provide antibiotic therapy and care provider contacts. The more important question is whether there is a dose dependent (i.e. treatment duration dependent) relation between antibiotic exposure in early life and long term health outcomes. More importantly, would reductions in treatment courses from 2 to 3 days (as usual following convential culture results) to 1 day or less (after results of Molecular Culture) have clinically meaningful impact, other than the consequences mentioned above. Some studies have assessed the risk of development of Necrotizing enterocolitis (e.g. PMID: 21489560) and find increased odds of NEC after prolonged antibiotic therapy. But this does not really answer the question the reviewer has, since this regards de-escalating therapy quicker than after 36-72 or after conventional culture results. Most studies on different health outcomes do not differentiate between short and longer courses but more often look at the amount of treatment courses. For example, 3 studies looked at prenatal and perinatal antibiotic exposure and risk of health outcomes of the offspring later in life (PMID: 29678946; PMID: 28088397; PMID: 31994697). For reference, also see PMID: 37214166. Doses were defined as full treatments however instead of duration of treatment. It is therefore complex to extrapolate from this what the difference is between 72 hours and 8 hours of antibiotics for example, given implementation of a higher speed molecular diagnostic assay for pathogen identification.

Comment 2: your last paragraph reads like a conclusion and doesnt actually 'introduce' the paper ie we propose a protocol for...

Response 2: We thank the reviewer for this comment and have adapted this paragraph.

Comment 3: materials and methods. thorough and well written. will you be looking at clinical outcomes as well eg antibiotics prescribed, empiric vs direct, length of time on antibiotics, how unwell the patient was of the period of hospitalisation (eg admission to ICU, artificial ventilation etc).

Response 3: We thank the reviewer for this comment. We will most definitely. As can be deducted from the clinical sepsis definitions, Line 254-255. we will include such parameters to assess severity of disease and to enable testing against potential future definitions of clinical neonatal sepsis that may evolve in the meantime (e.g. such as PMID: 38051733)

Comment 4: discussion. the primary outcome is quite simple and elegant, i would like to see the discussion explore the implications of the suspected outcome, what do you hope to achieve? what are the risks involved with implementing? eg critically unwell are likely to stay on empirics much longer despite the MC results suggesting a narrowing focus is ok...

Response 4: We thank the reviewer for this comment. We have written several additional statements in the discussion. Line 291-300 and 313-325.

 

 

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

nothing to add