The Association between the Police, Ambulance, Clinician Early Response (PACER) Model and Involuntary Detentions of People Living with Mental Illness: A Protocol for a Retrospective Observational Study

Round 1

Reviewer 1 Report

The study of Heffernan, et al, investigated an interesting and relevant topic as that of the PACER model to provide emergency healthcare to patients with severe mental disorders. The topic is relevant, and the objectives are interesting and valuable. The expected results of the study protocol could be really useful for policymakers, stakeholders and the healthcare management not only in the Canberra Region, ACT territory. Despite this, this study protocol has various flaws that need to be amended.

Following, you can find my concerns and suggestions along the manuscript, in a very schematic structure.

Introduction

• The Introduction section is quite long. Please, shorten this section; specially making more concise the sentences and the paragraphs not related to the issue of Emergency Departments (ED). For example, the paragraph from line 35 to line 56 could be synthetized in a few sentences. The description of ED functioning in Australia, starting from line 57, should instead be kept in the Introduction, but shortened. These lines are interesting and important to provide a useful insight to the ED issue, but in a shorter way.
• Line 26: the Authors stated that the “45% of Australians will experience mental illness in their lifetime”. I suggest the Authors to be more cautious, and to highlight that the percentage reported it comes from an estimate; maybe rephrasing the sentence with something like “recent estimates that nearly 45% of Australian will experience …”.
• Lines 128-129: the sentence is not clear. Maybe the “and” must be deleted? Without this conjunction, the sentence will have a different meaning. Please rephrase.
• The “Mental health legislation” sub-section should be shortened and made briefer and more concise.
• The “PACER” sub-section is clear and well-written. Greetings!

Materials and Methods

• In the Materials and Methods section please better specify what is the outcome and what is the exposure. Authors stated what will be the exposure classes that will be considered and the referent one, but I will provide more information on this detail.
• Please provide a more extensive and precise description of Inclusion and, specially, Exclusion criteria. For example, it is not clear whether PACER interventions for patients who experienced a more critical and urgent condition (needing a more urgent intervention and to be prioritized by the Eds) will be deleted. Please make it clearer.
• Authors stated, at lines 456-457, that “each patient presentation is considered a single episode of care and is closed at the point of outcome”. Thus, a patient with more than one ED presentation will be contribute more than one time to the general estimates of the model. The Authors should take into account of this during the Statistical analysis, planning a specific statistical analysis taking into account for the within-subject correlation.

Each ED presentation is a single event, but since a patient could experience more than one ED presentation, it should be considered that two ED presentations experienced by the same patients will be correlated somehow. Please, plan and add a specific analysis for this issue.

• Maybe Death could be assessed as a secondary outcome. It will be extremely interesting.
• In the “Data sources” sub-section the Authors did not provide enough information on the Data sources and which kind of data will be used. Will they use administrative data, maybe on electronic records? Will they use regional registry data or data from different regional healthcare facilities?

The whole sub-section needs to be rephrased providing more detailed information on the data that will be used and how they will be retrieved.

• In the Statistical Analysis plan, please consider other sensitivity analyses for taking into account for unmeasured confounding.
• Other analyses, like stratification for gender, age classes, years of education, could be performed to see whether these variables should be considered as confounders of effect modifiers.
• In the “Sample size” sub-section, a consideration (i.e., numeric value) regarding the power of the study, given the estimated sample size and the alpha level of 5%.
• Authors stated that a disadvantage of larger sample size is the issue of p-values approaching zero. They are right. The p-value usually is influenced by the sample size. But this issue should not be presented as a problem by the Authors. All the differences among the distributions of covariates in the different exposure levels should be tested using the standardized mean difference (SMD), which was proposed by Austin[1]^,[2] and is an alternative to the p-value that is not influenced by the sample size. Usually, standardized mean differences < 0.10 were considered as negligible and not statistically significant.
• The relationship between the outcome and exposure levels will be assessed using a logistic regression, stated the Authors. But in the “Statistical Methods” sub-section no description of the Odds Ratios (ORs) was provided. Please, consider that the relative risk (also known as risk ratio [RR]) is the ratio of risk of an event in one group (e.g., exposed group) versus the risk of the event in the other group (e.g., nonexposed group); whereas the odds ratio (OR) is the ratio of odds of an event in one group versus the odds of the event in the other group.[3] The Authors should consider that though OR also indicates the nature of association between exposure and outcome, it is not identical to RR.

The relationship of OR and RR is complex. Table 3 shows RR and OR for different event rates. When there is no association between exposure and outcome, both OR and RR are identical and equal to 1.0. When there is an association between an exposure and an outcome, OR exaggerates the estimate of their relationship (is farther from 1.0 than RR)³. Only when the outcome is rare (typically <10%), the value of OR is not too different from that of RR, and the two can be used interchangeably.

• The statistics describing the distribution of the covariates measured on the enrolled population, how will they be assessed and reported? I guess that continuous variables will be described as mean ± standard deviation (SD), whereas absolute frequencies and percentages will be reported for categorical variables. Please add a description in the relative sub-section.

[1] Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med. 2009 Nov 10;28(25):3083–107.

[2] Austin PC. Using the Standardized Difference to Compare the Prevalence of a Binary Variable Between Two Groups in Observational Research. Commun Stat - Simul Comput. 2009 May 14;38(6):1228–34.

[3] Ranganathan P, Aggarwal R, Pramesh CS. Common pitfalls in statistical analysis: Odds versus risk. Perspect Clin Res. 2015 Oct-Dec;6(4):222-4. doi: 10.4103/2229-3485.167092. PMID: 26623395; PMCID: PMC4640017.

Author Response

The study of Heffernan, et al, investigated an interesting and relevant topic as that of the PACER model to provide emergency healthcare to patients with severe mental disorders. The topic is relevant, and the objectives are interesting and valuable. The expected results of the study protocol could be really useful for policymakers, stakeholders and the healthcare management not only in the Canberra Region, ACT territory. Despite this, this study protocol has various flaws that need to be amended.
Following, you can find my concerns and suggestions along the manuscript, in a very schematic structure.
Introduction
· The Introduction section is quite long. Please, shorten this section; specially making more concise the sentences and the paragraphs not related to the issue of Emergency Departments (ED). For example, the paragraph from line 35 to line 56 could be synthetized in a few sentences. The description of ED functioning in Australia, starting from line 57, should instead be kept in the Introduction, but shortened. These lines are interesting and important to provide a useful insight to the ED issue, but in a shorter way.
RESPONSE: Thank you for your suggestion. We have shortened our introduction significantly in line with your comment.
· Line 26: the Authors stated that the “45% of Australians will experience mental illness in their lifetime”. I suggest the Authors to be more cautious, and to highlight that the percentage reported it comes from an estimate; maybe rephrasing the sentence with something like “recent estimates that nearly 45% of Australian will experience …”.
RESPONSE: We have amended this sentence as suggested.
· Lines 128-129: the sentence is not clear. Maybe the “and” must be deleted? Without this conjunction, the sentence will have a different meaning. Please rephrase.
RESPONSE: We have amended this sentence as suggested.
· The “Mental health legislation” sub-section should be shortened and made briefer and more concise.
2
RESPONSE: We have removed the lengthy criteria and made it more concise as requested.
· The “PACER” sub-section is clear and well-written. Greetings!
Materials and Methods
· In the Materials and Methods section please better specify what is the outcome and what is the exposure. Authors stated what will be the exposure classes that will be considered and the referent one, but I will provide more information on this detail.
RESPONSE: We have made amendments to this section to provide a clearer description of our study aims.
· Please provide a more extensive and precise description of Inclusion and, specially, Exclusion criteria. For example, it is not clear whether PACER interventions for patients who experienced a more critical and urgent condition (needing a more urgent intervention and to be prioritized by the Eds) will be deleted. Please make it clearer.
RESPONSE: We have made amendments to this section of the manuscript for greater clarity, as suggested.
· Authors stated, at lines 456-457, that “each patient presentation is considered a single episode of care and is closed at the point of outcome”. Thus, a patient with more than one ED presentation will be contribute more than one time to the general estimates of the model. The Authors should take into account of this during the Statistical analysis, planning a specific statistical analysis taking into account for the within-subject correlation.
RESPONSE: Thank you for this suggestion. It is possible that repeat consumers are identified in the data set. This will only be known after the data has been extracted. We will check for this in the data extraction phase of our study and seek to omit consumers who may skew the data due to repeat presentations. We have made amendments to the manuscript as suggested.
Each ED presentation is a single event, but since a patient could experience more than one ED presentation, it should be considered that two ED presentations experienced by the same patients will be correlated somehow. Please, plan and add a specific analysis for this issue. RESPONSE: As above with thanks
· Maybe Death could be assessed as a secondary outcome. It will be extremely interesting. RESPONSE: Thank you for your suggestion. Whilst this is a very interesting outcome, it is not within the scope of the study. Death is generally not an outcome examined in psychiatric research as it is a relatively rare event. Furthermore, it is unlikely that we could maintain the de-identification of any consumers who have died due to the rarity of the event and the smaller geographical area.
3
· In the “Data sources” sub-section the Authors did not provide enough information on the Data sources and which kind of data will be used. Will they use administrative data, maybe on electronic records? Will they use regional registry data or data from different regional healthcare facilities? RESPONSE: We have made amendments to this section of the manuscript for greater clarity as suggested.
The whole sub-section needs to be rephrased providing more detailed information on the data that will be used and how they will be retrieved. RESPONSE: We have made amendments to this section of the manuscript for greater clarity as suggested.
In the Statistical Analysis plan, please consider other sensitivity analyses for taking into account for unmeasured confounding. RESPONSE: We have made amendments to the statistical analysis section of the manuscript. We have not planned to undertake sensitivity analysis and we have provided further details regarding our logistic regression analysis.
· Other analyses, like stratification for gender, age classes, years of education, could be performed to see whether these variables should be considered as confounders of effect modifiers. RESPONSE: Years of education is not a variable that is collected during a crisis mental health assessment; however, we will control for other demographic variables in our multiple analysis. We have updated the manuscript for greater clarity.
· In the “Sample size” sub-section, a consideration (i.e., numeric value) regarding the power of the study, given the estimated sample size and the alpha level of 5%. RESPONSE:
· Authors stated that a disadvantage of larger sample size is the issue of p-values approaching zero. They are right. The p-value usually is influenced by the sample size. But this issue should not be presented as a problem by the Authors. All the differences among the distributions of covariates in the different exposure levels should be tested using the standardized mean difference (SMD), which was proposed by Austin[1],[2] and is an alternative to the p-value that is not influenced by the sample size. Usually, standardized mean differences < 0.10 were considered as negligible and not statistically significant. RESPONSE: We have deleted this section of the manuscript following advice from the reviewer.
· The relationship between the outcome and exposure levels will be assessed using a logistic regression, stated the Authors. But in the “Statistical Methods” sub-section no description of the Odds Ratios (ORs) was provided. Please, consider that the relative risk (also known as risk ratio [RR]) is the ratio of risk of an event in one group (e.g., exposed group) versus the risk of the event in the other group (e.g., nonexposed group); whereas the odds ratio (OR) is the ratio of odds of an event in one group versus the odds of the event in the other group.[3] The Authors should consider that though OR also indicates the nature of association between exposure and outcome, it is not identical to RR.
The relationship of OR and RR is complex. Table 3 shows RR and OR for different event rates. When there is no association between exposure and outcome, both OR and RR are identical and equal to 1.0. When there is an association between an exposure and an outcome, OR exaggerates the estimate
4
of their relationship (is farther from 1.0 than RR)3. Only when the outcome is rare (typically <10%), the value of OR is not too different from that of RR, and the two can be used interchangeably.
· The statistics describing the distribution of the covariates measured on the enrolled population, how will they be assessed and reported? I guess that continuous variables will be described as mean ± standard deviation (SD), whereas absolute frequencies and percentages will be reported for categorical variables. Please add a description in the relative sub-section. RESPONSE: We have made a number of amendments to the statistical analysis section and hope this is satisfactory to the reviewers’ suggestions.
[1] Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med. 2009 Nov 10;28(25):3083–107.
[2] Austin PC. Using the Standardized Difference to Compare the Prevalence of a Binary Variable Between Two Groups in Observational Research. Commun Stat - Simul Comput. 2009 May 14;38(6):1228–34.
[3] Ranganathan P, Aggarwal R, Pramesh CS. Common pitfalls in statistical analysis: Odds versus risk. Perspect Clin Res. 2015 Oct-Dec;6(4):222-4. doi: 10.4103/2229-3485.167092. PMID: 26623395; PMCID: PMC4640017.

Reviewer 2 Report

The Abstract in its sub-sections needs re-organization and it does not adequately summarise the gist of the study.

Also, the writing style of the manuscript is not overall academic and formal. The article is proposed to be supplemented with a flowchart illustrating the research technique. A review of the literature is insufficient. It is critical to include some recent work (2018–2020) in the literature review. A literature review should be added in order to illustrate the central topic in a more detailed way. Some further explanations and interpretations are required for the results.

It is recommended to include a well-organized discussion of the findings, strengths, and limitations of the present project with additional explanation/details and a conclusion with future work.

I think the submission holds promise, but comprehensive editing is required. 

Author Response

The Abstract in its sub-sections needs re-organization and it does not adequately summarise the gist of the study.
RESPONSE: We have revised the abstract for further clarity as suggested.
Also, the writing style of the manuscript is not overall academic and formal. The article is proposed to be supplemented with a flowchart illustrating the research technique. A review of the literature is insufficient. It is critical to include some recent work (2018–2020) in the literature review. A literature review should be added in order to illustrate the central topic in a more detailed way. Some further explanations and interpretations are required for the results.
RESPONSE: We have performed a thorough literature review and included all relevant studies into our protocol document. Reviewer 1 has requested that the introduction be condensed, so we have avoided adding additional information; however, we are confident that all contemporary relevant literature has been included.
It is recommended to include a well-organized discussion of the findings, strengths, and limitations of the present project with additional explanation/details and a conclusion with future work.
RESPONSE: This is a protocol document, so we do not have findings yet. We have added reference to STROBE guidelines and included a limitations section. Our conclusion notes the usefulness of the proposed project in directing future service delivery, however, we do not know yet what the findings will be.
I think the submission holds promise, but comprehensive editing is required.
The authors should ask for the help of a native English-speaking proofreader because there are some linguistic mistakes that should be fixed.
RESPONSE: All authors are English speaking as first language, have reviewed the revised version for editing and grammar.

Round 2

Reviewer 1 Report

Authors have answered to the suggestions and comments of the previous submission.

Author Response

Reviewer one:

1. English revision suggested.

RESPONSE: We have undertaken extensive review of the grammar and language in this document. All authors are native English speaking with a combined total of 476 published academic papers. We are wondering if the fact we sent through a tracked change document which might have been sent to reviewers as a clean document might account for some of the concerns around English language.

Reviewer 2 Report

The study describes an innovative therapy protocol that could be established with the aid of smart technologies via algorythms applied in Health System facilitating early correspondence of mental health providers. Although, some further revisions should be made by authors for its amelioration:

The title needs further thought - shortened and more accurate. 

The Abstract in its sub-sections needs re-organization and it does not adequately summarise the gist of the study.

The article is proposed to be supplemented with a flowchart illustrating the research technique.

Also the sections after introduction should be reformed under literature review. The section of results needs revisions too. Also the section of materials and methods should more well-organized. Some further explanations and interpretations are required for the results.

It is recommended to include a well-organized discussion of the findings, strengths, and limitations of the present project with additional explanation/details and a conclusion with future work.

I think the submission holds promise, but comprehensive editing is required.

The authors should ask for the help of a native English-speaking proofreader because there are some linguistic mistakes that should be fixed.

Also, the writing style of the manuscript is not overall academic and formal.

Author Response

The study describes an innovative therapy protocol that could be established with the aid of smart technologies via algorithms applied in Health System facilitating early correspondence of mental health providers. Although, some further revisions should be made by authors for its amelioration:

1. The title needs further thought - shortened and more accurate. 

RESPONSE: The title of the study was developed using the PICO framework after significant review by the research team, and as part of a student dissertation. The title has been accepted by the sponsoring University and two Human Research Ethics Committee’s and is unable to be changed.  

2. The Abstract in its sub-sections needs re-organization and it does not adequately summarise the gist of the study.

RESPONSE: We understand that Nursing Reports does not require a structured abstract. We have provided additional reworking of the abstract to further detail the study protocol.

3. The article is proposed to be supplemented with a flowchart illustrating the research technique.

RESPONSE: We have provided two flow charts in Figure 1 and 2 of the manuscript which describe the primary and secondary outcomes of the study. Please see pages 7 and 9.

4. Also the sections after introduction should be reformed under literature review.

RESPONSE: We believe the reviewer is asking for a subheading titled ‘Literature Review’. We prefer to use for informative subheadings to guide the reader through the relevant background and literature. However, we have made further amendments to the subheadings this section.

5. The section of results needs revisions too.

RESPONSE: Without specifying the changes that need to be made, we are unable to ascertain what the reviewer is referring to here, however we have reworded this section for further clarity. We have made further amendments to this section that we hope will please the reviewer.

6. Also the section of materials and methods should more well-organized.

RESPONSE: As per above, we are unsure of what changes the reviewer is suggesting, however we have made some further amendments to improve readability.

7. Some further explanations and interpretations are required for the results.

RESPONSE: We are unable to provide interpretation as this is a protocol document, however we will report our results using the STROBE guidelines for reporting observational studies, after the research has concluded.

8. It is recommended to include a well-organized discussion of the findings, strengths, and limitations of the present project with additional explanation/details and a conclusion with future work.

RESPONSE: We have made amendments to this section for greater clarity. As this is a protocol document, we are limited in what information can be provided in this section.

9. I think the submission holds promise, but comprehensive editing is required.

RESPONSE: We have conducted extensive editing of the manuscript.

Comments on the Quality of English Language

10. The authors should ask for the help of a native English-speaking proofreader because there are some linguistic mistakes that should be fixed.

RESPONSE: We have undertaken extensive review of the grammar and language in this document. If the reviewer or editor can be specific about where these linguistic mistakes are, we will be happy to edit as necessary.

11. Also, the writing style of the manuscript is not overall academic and formal.

RESPONSE: As above, the manuscript has been comprehensively reviewed and changes made to improve the overall language and formality of the manuscript.

Round 3

Reviewer 2 Report

The Abstract in its sub-sections needs re-organization and it does not adequately summarise the gist of the study. For instance, the sections (into, methods, results, discussion, conclusion) should be described briefly and clarified. The article is proposed to be supplemented with a flowchart illustrating the research technique. A review of the literature is insufficient. It is critical to include some recent work (2018–2020) in the literature review. A literature review should be added in order to illustrate the central topic in a more detailed way. Some further explanations and interpretations are required for the results. Furthermore, it is recommended to include a well-organized discussion of the findings, strengths, and limitations of the present project with additional explanation/details and a conclusion with future work.

I think the submission holds promise, but comprehensive editing is required.

 The authors should ask for the help of a native English-speaking proofreader because there are some linguistic mistakes that should be fixed. The title needs further thought - shortened and more accurate. The language of the manuscript is not academic. For instance "we" referring to the authors should be replaced.

Author Response

Please see attachment 

Author Response File: Author Response.pdf