Review Reports - A Vulnerability Index to Assess the Risk of SARS-CoV-2-Related Hospitalization/Death: Urgent Need for an Update after Diffusion of Anti-COVID Vaccines

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The authors assessed the risk of hospitalization/death for SARS-CoV-2 in vaccinated and not vaccinated patients in GP settings.

The text is not easy to understand and seems more like a statistical manuscript than an article useful to clinicians.

The efforts to research in GP areas are great, but the authors could be clearer and could explain in a simpler way.

Authors should explain:

inclusion and exclusion criteria, in “Methods” section
Did you considered any co-variables ? Did you performed a multivariate analysis?
ethics committee approval (number/code and year, please)?
Patients were enrolled by EMR from a database in which hundred of General Practictioner and their patients were included: where are they? A study group with all this clinicians should be considered.
Figure 1: which of the two images refers to vaccinated people and which to non-vaccinated people?
Did you find any adverse event related to vaccination (i.e. thrombosis, death, autoimmune diseases, trigger for other diseases, etc etc)? Discuss about it in “introduction”, references are very scarce.
Can we have a calibration plot for death-only risk?
did you find any population at higher risk for hospitalization/death?
keep it simple.

Author Response

Reviewer#1

The authors assessed the risk of hospitalization/death for SARS-CoV-2 in vaccinated and not vaccinated patients in GP settings.

The text is not easy to understand and seems more like a statistical manuscript than an article useful to clinicians.

The efforts to research in GP areas are great, but the authors could be clearer and could explain in a simpler way.

We would like to thank the reviewer for these comments. Point-by-point responses are reported below. In specific, we agree on the fact that the manuscript should contain more clinical aspects, such as the usefulness in daily practice especially once implemented in physician’s informatic tools. We therefore extended the manuscript according to the journal guidelines for Brief Reports.

Authors should explain:

inclusion and exclusion criteria, in “Methods” section

Response#1: As requested by the Reviewer, we specified both inclusion and exclusion criteria (page 3, lines 24-30).

Did you considered any co-variables ? Did you performed a multivariate analysis?

Response#2: We conducted multivariate analysis using Cox regression. We adopted the same covariates with which the primary score, nominally HS-CoVId, was developed (Lapi et al., Exp Rev Vaccines 2022) (page 4, lines 5-8; lines 12-19).

ethics committee approval (number/code and year, please)?

Response#3: Given the retrospective nature of the study, no Ethics committee authorization was required. The Internal Review Borad of the Italian College of General Practitioners and Primary Care approved the study. In this respect, the manuscript was amended accordingly (page 4, lines 37-38).

Patients were enrolled by EMR from a database in which hundred of General Practictioner and their patients were included: where are they? A study group with all this clinicians should be considered.

Response#4: The Health Search Database (HSD) is formed by a sample of almost 800 GPs. They entered this research network, on voluntary bases, by attending specific courses on how to perform data entry. There are selection criteria for GPs who are interested to be part of HSD, such as years of practice (at least three), number of patients in charge (at least 600; an Italian GP can register up to 1500 patients). Currently, participating GPs are homogenously distributed across the Italian regions. Some details on database representativeness and validity were added (page 6, lines 38-45).

Figure 1: which of the two images refers to vaccinated people and which to non-vaccinated people?

Response#5: We apologize for this missing information. We amended the Figure accordingly (see the amended caption).

Did you find any adverse event related to vaccination (i.e. thrombosis, death, autoimmune diseases, trigger for other diseases, etc etc)? Discuss about it in “introduction”, references are very scarce.

Response#6: We would like to thank the Reviewer for this comment. The study objective was based on development and validation of a score predicting the risk of COVID-19-related complications (i.e. hospitalization/death) . We did not evaluate adverse events likely due anti-COVID vaccines. In this respect, we are currently involved in multi-country studies requested by European Medication Agency (EMA) to assess the risk profile of anti-COVID vaccines. Currently, the available literature, as reported by the last EMA official report (https://www.ema.europa.eu/en/human-regulatory-overview/public-health-threats/coronavirus-disease-covid-19/covid-19-medicines/safety-covid-19-vaccines), confirmed the favorable safety profile of this vaccination. We added further information on these aspects (page 2, lines 30-34).

Can we have a calibration plot for death-only risk?

Response#7: We added some information on the number of deaths and hospitalizations forming the outcome variable. The low number of deaths (n=743; 35% of the outcome) did not allow us to create a reliable model on this outcome given the reduced analysis power. In addition, the chice of a composite outcome was in line with the scope of prediction algorithm which consists of supporting GPs in case of prescription of hospital admission for COVID-19 patients being at risk of infection-related complications.

did you find any population at higher risk for hospitalization/death?

Response#8: We provided data on the number of patients being classified at ‘low/moderate’ (n=90743) or ‘high’ (n=33577, 27%) risk by the HS-CoVId (page 5, lines 39-44). These cut-offs were established on the bases of highest sensitivity and specificity of the algorithm (please see responses to the Reviewer#5).

keep it simple.

Response#9: As requested by the reviewer, we attempted to simplify the presentation and interpretation of the results. In this respect, we extended the discussion on how these findings are useful for clinical practice (page 6, lines 19-34, lines 38-45 and lines 51-53; page 7, lines 15-18).

Reviewer 2 Report

Comments and Suggestions for Authors

Lapi et al., have described an index to asses SAR-COV-2 associated hospitalization or death before and after use of vaccination. The study can be a useful tool for the practitioners to make risk assessments.

Few questions:

1) Is there a particular reason to not to include pediatric population in the study?

2) Does this model include or exclude other viral co-morbidities caused by seasonal influenza?

Comments on the Quality of English Language

No comments.

Author Response

Response#1: We would like to thank the reviewer for these comments.

Few questions:

Is there a particular reason to not to include pediatric population in the study?

Response#2: In Italy, pediatric patients are not mainly cared by GPs. Even though there is no limitation to have pediatric patients, they generally switch from pediatricians to GPs at 14-15 years of age. In essence, the cohort of pediatric patients in HSD is very small so sensibly reducing the analysis power. Furthermore, older adults and co-frail patients are those who can mainly benefit from the HS-CoVId, so excluding children and adolescents.

Does this model include or exclude other viral co-morbidities caused by seasonal influenza?

Response#3: The reviewer raised an important point. This model was focused on patients being infected by COVID-19 to estimate their potential complications due to SARS-CoV-2, so excluding other viral morbidities. It is respect, we are currently developing a model to predict complications due to lower respiratory tract infections irrespective of their etiology. Instead, this is an update of HS-CoVId, in which GPs should generate lists of COVID-19 sufferers whether they were vaccinated or not according to regulatory and clinical indications. These lists might support the clinical decision making to prioritize some patients’ subgroups, according to their risk profile, to implement preventive and/or treatment interventions (page 6, lines 19-34).

Reviewer 3 Report

Comments and Suggestions for Authors

Very interesting study. Congratulations to the authors.

There are some minor mistakes including: antiCOVID-19 -> anti-COVID-19,

But the scientific value is still very high.

Author Response

Very interesting study. Congratulations to the authors.

Response#1: We would like to thank the reviewer for these comments.

There are some minor mistakes including: antiCOVID-19 -> anti-COVID-19,

But the scientific value is still very high.

Response#2: We apologize for these mistakes. They were amended accordingly.

Reviewer 4 Report

Comments and Suggestions for Authors

this brief is interesting. the abstract is well presented. the methods are clear. the results are described. the discussion is not sufficient, because it is important to evalue the use of this scores in the clinical practise

Author Response

Response#1: We would like to thank the reviewer for these comments. We attempted to extend the discussion section as requested especially in the valued of the score in clinical practice. In essence, using the HS-CoVId, GPs should generate lists of COVID-19 sufferers whether they were vaccinated or not. These lists might support the clinical decision making to prioritize some patients’ subgroups according to their risk profile, which will be stratified by the presence of vaccination (page 6, lines 19-34 and lines 38-45).

Reviewer 5 Report

Comments and Suggestions for Authors

Thank you for asking me to review this manuscript. An important point is made about the changing situation with SARS-CoV-2 infections and the need to update risk assessments. The data presented demonstrates changing risks, although the 57 hospitalisations/deaths in the follow up cohort do not amount to a large number for analysis. For the reader it would be useful to have some simpler figures to read, such as the sensitivity and specificity of the scoring system for hospitalisation/death and for instance in a sample size of 1000 patients with SARS-CoV-2 infections how many the score would correctly predict for progression.

Author Response

Response#1: We would like to thank the reviewer for these comments. To tackle the reduced power due to the 57 cases of hospitalization/death, we indeed conducted a secondary analysis by including the entire cohort of vaccinees (N=124320) irrespective of COVID-19 diagnosis. By doing so, we identified a higher number of cases of hospitalizations/deaths, so ensuring the analysis power among immunized patients (page 4, lines 33-36). In addition, through the application of the same selection criteria to the first semester 2023 (the most recent update for HSD), we performed a temporal validation (as proxy of external validity (Ramspek et al., Clin Kidney J 2021)) of the HS-CoVId. In line with prior analyses, we provided results which were consistent with those obtained for the primary model (page 7, lines 15-18).

For the reader it would be useful to have some simpler figures to read, such as the sensitivity and specificity of the scoring system for hospitalisation/death and for instance in a sample size of 1000 patients with SARS-CoV-2 infections how many the score would correctly predict for progression.

Response#2: The Reviewer raised a good point with which we agree. Using the Cox methodology to identify the best cut-points, we were able to stratified patients in those at low/intermediate and high risk. This classification was based on the highest value of sensitivity and specificity which were equal to 92 (95% CI: 88-95%) and 70% (95% CI: 69-70%), respectively (page 4, lines 32-33; page 5, lines 39-44). Given the journal guidelines we were allowed to select one Figure only. We would like to keep the present graphs because they support the study aim. Nevertheless, we provided data on temporal (as proxy of external) validation, which were reassuringly consistent with those obtained for the primary analysis as well as those stemming from our prior work (Lapi et al., Exp Rev Vacc 2022) (page 7, lines 13-18).

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Dear authors,

However, I believe that authors's really interested in science and they tried to do the best they could.

I would reject the work but I would give authors another chance.

Please at least:

write in a simple way who are the patient "at risk" and who might be vaccinated accordingly to your analysis
Keep it simple and delete all useless information (or drop it in an appendix) , such as "DM" "DPCM" or all the statistical
Create a study group with all the GPs involved (without their data, you weren't here)
Approval by EC is also necessary for retrospective studies

Author Response

We would like to thank the reviewer and the Acedemic Editor for these comments. Point-by-point responses are reported below and highlighted (in yellow) in the manuscript (we left the prior amendmends to keep track of the prior revisions).

Reviwer’s comments

Dear authors,

The manuscript should be rejected. It's unreadable for a clinician (this is a journal for clinicians), it remains a statistical paper, unuseful for anybody except for a statistician and/or some engineer interested in the creation of an EMR. However, I believe that authors's really interested in science and they tried to do the best they could.

Response#1: We would like to thank the reviewer for these comments. In the previous responses we extensively revised the manuscript (R1) by discussing the clinical and public health implications of the updated version of the HS-CoVId (page 6, lines 27-40; page 6, lines 44-51). The main application of the score is to prioritize vaccine administration among those patients who have different risk of incurring in COVID-related hospitalization/death. To do so, decision tools embedding this score should be adopted in primary care. Please note that the first version (vaccination was not yet available) HS-CoVId has been published in peer-reviewed journal (Lapi et al., Exp Rev Vacc 2021) and the methodology was pivotal to get the paper accepted. Officially, the development, validation and update of the HS-CoVId need to be compliant with the TRIPOD statements (Moons et al., Ann Int Med 2015). Furthermore, most of the authors (5 out of 8) are clinicians besides being experienced researcher in the field of clinical epidemiology, public health, vaccinology and primary care.

I would reject the work but I would give authors another chance. Please at least:

We would like to thank the reviewer for this opportunity.

write in a simple way who are the patient "at risk" and who might be vaccinated accordingly to your analysis

Response#2: We attempted to provide further definition of the patients “at risk” (page 6, line 43). As mentioned over the paper, we underlined the GP’s need to be equipped with decision tools able to prioritize vaccine administration to those at major risk.

Keep it simple and delete all useless information (or drop it in an appendix), such as "DM" "DPCM" or all the statistical

Response#3: As stated above, the first version (vaccination was not yet available) HS-CoVId has been published in peer-reviewed journal (Lapi et al., Exp Rev Vacc 2021) and the methodology was pivotal to get the paper accepted. Officially, the development, validation and update of the HS-CoVId need to be compliant with the TRIPOD statements (Moons et al., Ann Int Med 2015). As such, the adopted statistics need to be part of the ‘Method’ section. DM and DPCM are crucial part of the Italian regulation concerning the mandatory informatic equipment of GPs. In our view, it is important to mention this aspect in the paper as well. Furthermore, it was requested in the prior revision (R1) as well.

Create a study group with all the GPs involved (without their data, you weren't here)

Response#4: We would like to thank the reviewer for this comment. It is true that with no GPs’ data, every retrospective study cannot be conducted. Several papers have been conducted using HSD (again, a search in MEDLINE would ascertain the scientific validity of the data source). Two of the coauthors are among the founders of HSD and in the national meeting of the Italian College of General Practitioners and Primary Care (SIMG, the Italian acronym), the HS network is always recognized for its scientific relevance and production.

Approval by EC is also necessary for retrospective studies

Response#5: According to the Italian regulation on observational studies the secondary use of this kind of data do not need an ethical approval (or a notification to an Ethic Committee (EC)) because of this main reason: all observational studies require the list of participating GPs who are collecting data. In the use of this information, we are unable to identify both GPs and the aggregated EHRs. We are therefore unable to identify the ECs in charge of evaluating the study. Furthermore, the most recent regulation only mentions pharmacological studies for which there is the opportunity to refer to a unique EC for a multi-centric investigation. Please refer to the Italian regulations for further information: 2023: https://www.gazzettaufficiale.it/atto/serie_generale/caricaDettaglioAtto/originario?atto.dataPubblicazioneGazzetta=2023-02-07&atto.codiceRedazionale=23A00853&elenco30giorni=true; 2007: https://www.aifa.gov.it/en/-/linee-guida-studi-osservazionali.

Besides ours (e.g. Lapi et al., Exp Rev of Vacc 2021), prior investigations from other authors using similar data sources (e.g. Ingrasciotta et al., PlosONE 2019; https://pubmed.ncbi.nlm.nih.gov/31536588/) adopted the same procedures.

Academic Editor’s comments

I believe that ethical approval is mandatory for any study, although retrospective. Please ask authors to provide it. For retrospective observational studies italian law could ask only an informative report of the project. It is a decsion in the hand of any local ethical committee, but an attempt could be done

We would like to thank the Editor for this comment and for the opportunity to provide responses with this second round (R2).

Response#1: As replied to the Reviewer, according to the Italian regulation on observational studies the secondary use of this kind of data do not need an ethical approval (or a notification to an Ethic Committee (EC)) because of this reason: all observational studies require the list of participating GPs who are collecting data. In the use of this information, we are unable to identify both GPs and the aggregated EHRs. We are therefore unable to identify the ECs in charge of evaluating the study. Furthermore, the most recent regulation only mentions pharmacological studies for which there is the opportunity to refer to a unique EC for a multi-centric investigation. Please refer to the Italian regulations for further information: 2023: https://www.gazzettaufficiale.it/atto/serie_generale/caricaDettaglioAtto/originario?atto.dataPubblicazioneGazzetta=2023-02-07&atto.codiceRedazionale=23A00853&elenco30giorni=true; 2007: https://www.aifa.gov.it/en/-/linee-guida-studi-osservazionali.