**Multiagency Approach to Disaster Management, Focusing on Triage, Treatment and Transport**

Editors

**Amir Khorram-Manesh Krzysztof Goniewicz**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Amir Khorram-Manesh University of Gothenburg Sweden

Krzysztof Goniewicz Polish Air Force University Poland

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Topic published online in the open access journal *International Journal of Environmental Research and Public Health* (ISSN 1660-4601), *Safety* (ISSN 2313-576X), *Sustainability* (ISSN 2071-1050), and *Healthcare* (ISSN 2227-9032) (available at: https:// www.mdpi.com/topics/Multiagency\_Approach\_Disaster\_Management).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-7706-7 (Hbk) ISBN 978-3-0365-7707-4 (PDF)**

Cover image courtesy of Camilo Jimenez

© 2023 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**


Evidence from 33 General Hospitals and Their Anticipated Impact on Healthcare during Disasters and Public Health Emergencies

Reprinted from: *Healthcare* **2021**, *9*, 1462, doi:10.3390/healthcare9111462 .............. **127**

#### **Yu-Long Chen, Wen-Chii Tzeng, En Chao and Hui-Hsun Chiang**


## **Preface to "Multiagency Approach to Disaster Management, Focusing on Triage, Treatment and Transport"**

The Multiagency Approach to Disaster Management topic collection, published by MDPI, brings together a diverse array of research articles that delve into the intricacies of multiagency collaboration in the context of disaster management. This compilation of studies provides valuable insights into interagency cooperation's benefits, challenges, and opportunities and offers innovative solutions to enhance disaster resilience and preparedness.

Disaster management has evolved beyond traditional response and recovery measures in recent years. The focus has shifted towards proactive planning, prevention, and preparedness. Recognizing the complexities associated with modern-day disasters, this collection emphasizes the importance of a multiagency approach to address these challenges effectively. The collaboration between government agencies, non-governmental organizations, private sector entities, and local communities is crucial to achieving a comprehensive and coordinated response to disasters.

This topic collection comprises research articles that cover a wide range of subjects related to multiagency disaster management, including:


The articles in this collection contribute to the body of knowledge on multiagency approaches in disaster management by addressing the gaps and challenges that often hinder effective cooperation. Moreover, the collection highlights the importance of adaptive, flexible, and inclusive strategies that consider the unique needs and capacities of various stakeholders involved in disaster management.

The Multiagency Approach to Disaster Management topic collection is a valuable resource for researchers, practitioners, policymakers, and other stakeholders in the disaster management field. The shared knowledge and experiences found in these articles not only promote a deeper understanding of the benefits of collaboration but also provide practical guidance for implementing multiagency approaches to improve disaster resilience and preparedness.

> **Amir Khorram-Manesh and Krzysztof Goniewicz** *Editors*

## *Article* **Gender and Public Perception of Disasters: A Multiple Hazards Exploratory Study of EU Citizens**

**Arturo Cuesta 1,\*, Daniel Alvear 1, Antonio Carnevale <sup>2</sup> and Francine Amon <sup>3</sup>**


**\*** Correspondence: arturo.cuesta@unican.es

**Abstract: Aim:** To explore gender influence on individual risk perception of multiple hazards and personal attitudes towards disaster preparedness across EU citizens. **Method:** An online survey was distributed to 2485 participants from Spain, France, Poland, Sweden and Italy. The survey was divided into two parts. The first part examined perceived likelihood (L), perceived personal impact (I) and perceived self-efficacy (E) towards disasters due to extreme weather conditions (flood, landslide and storm), fire, earthquake, hazardous materials accidents, and terrorist attacks. The overall risk rating for each specific hazard was measured through the following equation R = (L × I)/E and the resulting scores were brought into the range between 0 and 1. The second part explored people's reactions to the Pros and Cons of preparedness to compute the overall attitudes of respondents towards preparation (expressed as a ratio between −1 and 1). **Results:** Although we found gender variations on concerns expressed as the likelihood of the occurrence, personal consequences and self-efficacy, the overall risks were judged significantly higher by females in all hazards (*p* < 0.01). We also found that, in general, most respondents (both males and females) were in favour of preparedness. More importantly, despite the gender differences in risk perception, there were no significant differences in the attitudes towards preparedness. We found weak correlations between risks perceived and attitudes towards preparedness (rho < 0.20). The intersectional analysis showed that young and adult females perceived higher risks than their gender counterparts at the same age. There were also gender differences in preparedness, i.e., females in higher age ranges are more motivated for preparedness than men in lower age ranges. We also found that risk perception for all hazards in females was significantly higher than in males at the same education level. We found no significant differences between sub-groups in the pros and cons of getting ready for disasters. However, females at a higher level of education have more positive attitudes towards preparedness. **Conclusions:** This study suggests that gender along with other intersecting factors (e.g., age and education) still shape differences in risk perception and attitudes towards disasters across the EU population. Overall, the presented results policy actions focus on promoting specific DRR policies and practices (bottom-up participatory and learning processes) through interventions oriented to specific target groups from a gender perspective.

**Keywords:** gender; public perception; multiple hazards; risk perception; preparedness

### **1. Introduction**

Between 2000 and 2021 in total 14.189 disasters have occurred worldwide causing around 1.5 million casualties. Of these, 1.633 disasters have occurred in Europe with 169.402 reported casualties [1]. The role people play before, during and after a disaster is of crucial importance. In fact, the active participation of individuals and communities is a principle of the Sendai Framework for Disaster Risk Reduction 2015-2030 (SFDRR) [2]. Bottom-up participatory and learning processes in which citizens can act by themselves

**Citation:** Cuesta, A.; Alvear, D.; Carnevale, A.; Amon, F. Gender and Public Perception of Disasters: A Multiple Hazards Exploratory Study of EU Citizens. *Safety* **2022**, *8*, 59. https://doi.org/10.3390/ safety8030059

Academic Editor: Raphael Grzebieta

Received: 17 May 2022 Accepted: 2 August 2022 Published: 5 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and/or together with emergency services are the suggested mechanisms to improve Disaster Resilience and Response (DRR) [3,4]. The Sendai Framework also recognizes the importance of integrating a gender perspective into all DRR policies and practices. Hence, for effective bottom-up implementation of DRR policies and practices, we need first to understand the differences/similarities of the risks perceived by both women and men and their subsequent attitudes toward preparedness (as a behavioural precursor).

Although disasters affect whole communities, they are not gender neutral as they impact women and men differently. Gender issues (economic, social, and political inequalities) can create specific vulnerabilities for women in disasters [5,6]. Moreover, gender structures shape the roles, experiences, and responsibilities of individuals in disasters [6,7]. The typical gender roles in disasters are described by Enarson [8] and Fothergill [9].

Gender can also be related to risk judgments and attitudes towards safety [10–13]. In this sense, risk perception and preparedness have been the central investigated issues. Some studies directly address gender influence on these subjects and others include gender among other predictors/variables by simply reporting gender "differences". Regardless of the method used, the literature indicates that women in general perceive hazards as being more serious and riskier than men [8,14–17] and that men express more confidence to face disasters [13,18,19]. Researchers have also focused on preparedness by exploring gender among other factors (e.g., race/ethnicity, age, education, etc.). Some studies showed that gender acts as a predictor of preparedness with women being less likely to be prepared than men for specific hazards [20–23] but other studies were not conclusive (e.g., [24,25]).

Risk perception is usually conceptualised as a logical predictor of preparedness. However, the link between these constructs is still not clear [26,27]. Whereas some studies found that risk perception is associated with or predicts preparedness [23,28,29], others did not [30–32]. Furthermore, most previous studies concentrated on unique disasters (past and/or potential) in specific geographical regions with distinct degrees of gender relations/inequalities, or they were conducted for specific communities or groups of people living in the affected areas. Hence most research findings, although useful for regional and local authorities, are context dependent and difficult to generalize to other hazards and areas. For instance, for an EU policy implementation. Another aspect not fully addressed in the literature is the analysis of gender along with intersectional factors (intersectional approach) (Gendered Innovations: http://genderedinnovations.stanford.edu/ terms/intersectionality.html, accessed on 10 May 2022) in the context of disaster response as gender identities, norms, relations and attitudes both shape and are shaped by other social attributes.

The aim of this study is to investigate gender influence on risk perception for multiple hazards and attitudes toward preparedness from a regional perspective, i.e., across the EU population. The first question to investigate is, since women are likely to perceive higher risks than men, is it reasonable to think that they are also more motivated for preparedness? Otherwise (i.e., if there is no positive association between risk and preparedness in gender groups) is it reasonable to infer that gender, among other intersectional factors, contributes to shaping people's attitudes towards disasters? Therefore, the present study aims at contributing to current knowledge by analysing datasets from a multinational survey. The collected responses provided the opportunity to explore gender differences/similarities of EU citizens (from Spain, Poland, Sweden, France and Italy).

The main objectives of the current study are listed below:


Datasets produced here not only have scientific value but also have the potential to inform policymakers and first responders for developing risk management policies and training and communication campaigns, thus improving disaster response and resilience of society as viewed using a gender perspective in Europe.

#### **2. Method**

The Checklist for Reporting Results of Internet E-Surveys (CHERRIES) was used as a reference to provide exhaustive information on the survey and to facilitate reproducibility [33].

**Design**.—The survey was designed to cover people's risk perceptions and attitudes towards preparedness for disasters. The questions used to investigate these factors are listed in Table 1. To analyze risk perception we focused on three main factors inspired by the Protection Motivation Theory developed by Rogers [34,35] and also applied to disaster research [36,37]. These three factors are: (1) the likelihood of disasters to occur (L), (2) the personally relevant impact if disasters occur nearby (I) and (3) the perceived self-efficacy to face the disasters (E). Each question was asked in relation to extreme weather conditions, fires, earthquakes, hazardous material accidents and terrorist attacks. The rationale for the selection of these hazards was their global relevance in Europe (Table 1): meteorological (storms, extreme temperatures, floods), climatological (wildfires), geophysical (earthquakes), technological (industrial accidents) [1] and terrorism (terror attacks) [38,39].

**Table 1.** Human consequences of selected disasters for the last 20 years in Europe. Sources: EM-DAT [1] and GTD [40].


\* Bomb and shooting attacks in Western Europe.

In addition, 9-item questions were included to explore the attitudes of males and females towards disaster preparedness: 4 statements for the Pros and 5 statements for the Cons. For simplicity, the statements are expressed as Resilience, Information, Confidence, Assistance for the Pros and Uselessness, Buck-passing, Avoidance, Denial and Cost for the Cons (Table 2).

**Table 2.** Survey questions and the related available answers. \* Extreme weather conditions, Fire, Earthquake, Hazardous materials accidents, and Terrorist attacks. ˆ words in parentheses were not included in the questionnaire but are included here to remind the reader of the survey design.



**Table 2.** *Cont.*

**Ethics**.—The questionnaire was anonymous, and the privacy policy of the individual's posted information was noted (e.g., the purpose of the study, length of time to complete the survey, personal data and data protection, withdrawal rights, etc.). Due to the nature of this study written informed consent was not required. However, respondents were informed about the purpose of the study, and their rights and gave consent to participate by filling in the agreement part of the survey form. This study was approved by the Ethical Committee of the University of Cantabria.

**Development**.—A pilot was conducted involving 56 participants, allowing us the possibility to know whether a designed questionnaire fulfilled the purpose of the study (i.e., the respondents were asked whether the questions were clear and if they interpreted them as expected). The English version of the questionnaire was reviewed by two external experts and then translated into the target languages by native speakers. During the translation process, we paid special attention to achieving semantic, idiomatic, experiential, and conceptual equivalence to the original version. The initial translation into each target language was made by two independent translators per language to detect and resolve subtle differences/discrepancies. Also, the resulting versions were back-translated to ensure the accuracy of the translation. Then, the online prefinal versions were sent again to the translators for checking and final approval. Check-box answers were provided in the questionnaire to reduce the time to answer each item. Different scales were used. We considered a standard 5-point Likert scale (with a neutral option) for the Pros and Cons of preparedness as we wanted to collect enough granularity in opinions and attitudes. For self-efficacy, we reduced the response options using a 3-point Likert scale forcing the respondents to provide two pieces of information (two polar points along with a neutral option) based on the assumption that collapsing data from a longer scale into three-point scales does not diminish the reliability or validity of the resulting scores while enabling to collect clear responses about perceptions of self-efficacy to face disasters. For likelihood and impact, we used a 4-point Likert with no neutral option thus participants were required to form a judgment while reporting the intensity of the direction. Place of residence (village/town/city), education (no studies/primary/secondary/university), age, occupation (self-employed/employee/unemployed/retired/student) and gender (male/female/binary/other) were gathered at the starting section of the questionnaire.

**Survey administration**.—The usability and functionality of the electronic questionnaires were tested before fielding the final versions. A hired survey company sent an email invitation to individuals 2.485 living in the targeted countries. In total, we received 1.047 responses (response rate of 41.13%). Respondents belonged to validated databases and were given a monetary incentive for their participation. The company ensured a level of quality control, before and during the data collection.

The questionnaire had in total of 26 items in addition to the sociodemographic information on the first screen. Items were randomized to prevent biases in responses. Overall, the questionnaire took approximately 10–15 min to complete. The responses (only one per participant) were automatically captured and checked through the online survey system. The timeframe for the data collection was from 1 to 14 November 2020.

**Participants**.—Out of the 1.047 responses 1.2% identified as "non-binary" or "other" rather than "man" or "woman". This "non-binary" group comprised a very small sample size for statistical testing. Therefore, the population sample for the study involved 1.014 respondents (510 who identified as men and 504 who identified as women) from five countries representative of northern (Sweden), southern (Italy and Spain), eastern (Poland) and western (France) regions of Europe. Table 3 displays the characteristics of the surveyed participants. We compared our sample and the sociodemographic characteristics of those surveyed with the Eurostat census data [41]. The Eurostat for adults (aged 20 years and over) shows that 52% of females gave a 2.27% point (pp) difference between our data and the EU population. The age of respondents (20–69 years) was quite representative with an average difference of 4.69% (pp). Yet, there was an over-representation from respondents <29 years (absolute difference of 9.93%) and an under-representation from respondents >60 years (absolute difference of 7.68%). The dwelling type of our sample had absolute differences of 8.8% for cities, 0.3% for towns and 9.2% for rural areas when compared with Eurostat data. Education level (Secondary and University: sample = 91.4% vs. EU population = 79.50%) and occupation (people in the labour force; sample = 69% vs. EU population = 77.10%) had differences but reasonably represented in our study.


**Table 3.** Baseline characteristics of study participants. Significant *p*-values in bold.

**Analysis.**—Descriptive statistics are presented as absolute counts and/or percentages for ordinal variables while interval variables are expressed by means (with SD). To measure an individual's risk rating (R) for each of the five hazards we computed the likelihood (L), the personal impact (I) and the perceived self-efficacy (E) through the following equation R = (L × I)/E based on [16]. We assumed that the perceived self-efficacy affects the risk perceived rather than simply considering the perceived likelihood and impact to

measure risk ratings [17]. Hence, the perceived risk is minimized/reduced (or not) by the perceived self-efficacy here assumed as a value between 1 and 3 where 1 is "I don't know what to do", 2 is "I might know what to do" and 3 is "I know what to do". In the first case, self-efficacy does not change the perceived likelihood and impact. In the second case likelihood and impact are reduced by half. In the third case likelihood and impact are reduced by three times. The resulting scores were brought into the range between 0 and 1 for better understanding and further comparison with other datasets. To measure the attitudes toward preparedness, the responses to each item were summed to create composite scores (of Pros and Cons) for each respondent. The resulting scores were also normalized, and the overall attitudes were expressed as a ratio between −1 and 1 that resulted from subtracting the Pros score from the Cons score. Non-parametric methods were used to assess differences between groups: cross-tabulation and Pearson's chi-square for relative frequencies, Wilcoxon rank sum test and Kruskal-Wallis (Dunn's test) for ordinal and interval scales. The JASP statistical program v0.15 was used for statistical tests throughout the entire study (JASP Team, 2021). For all analyses performed in our study, *p*-values < 0.05 were considered statistically significant.

#### **3. Results**

**Risk perception**.—The variables related to likelihood (L), impact (I) and self-efficacy (E) for multiple hazards are listed in Table 4. There were gender differences when anticipating the occurrence of extreme weather (W =137,559, *p* = 0.03) and fire (W = 138,582, *p* = 0.01) considered less likely by males than females. We also found that gender is associated with the perceived impact of extreme weather (W = 139,124, *p* = 0.01), fire (W = 137,607, *p* = 0.03) and earthquake (W = 141,289, *p* < 0.01) if it occurs nearby. Nevertheless, the item score distributions of the perceived impacts for hazardous materials accidents (W=133,452, *p* = 0.27) and terrorist attacks (W = 131,533, *p* = 0.50) did not differ significantly between males and females. Our results also suggest that males expressed higher perception of their coping abilities than females to face potential hazards: extreme weather conditions (χ<sup>2</sup> = 20.4, *p* < 0.01), fire (χ<sup>2</sup> = 22.45, *p* < 0.01), earthquake (χ<sup>2</sup> = 12.18, *p* < 0.01), hazardous materials accident (χ<sup>2</sup> = 36.60, *p* < 0.01) and terrorist attack (χ<sup>2</sup> = 47.93, *p* < 0.01). Importantly, gender differences were found to be statistically significant (*p* < 0.01) in the overall risk perception with higher scores in females than in males (Table 5).

**Table 4.** Absolute counts of respondents in the perceived likelihood (from 1 = highly unlikely to 4 = highly likely), impact (from 1 = very low to 4 = very high) and self-efficacy (1 = I do not know what to do; 2 = I fairly know what to do; 3 = I know what to do) for extreme weather conditions, fire, earthquake, hazardous material accidents and terrorist attack. *p*-values of the two-sided Wilcoxon rank sum test for likelihood and impact and Chi-Square test for self-efficacy. The significant *p*-value is in bold.



**Table 4.** *Cont.*

**Table 5.** Differences in overall risk perception according to gender. Normalized Mean scores, SD standard deviation [0, 1]. *p*-values of the two-sided Wilcoxon rank sum test. The significant *p*-value is in bold.


**Attitudes towards preparedness**.—Most respondents were in favour of getting prepared for disasters (Table 6). There were no statistically significant gender differences for Resilience "it is easier to get back to normal", Information "people have information about what to do" and Confidence "taking action makes me worry less" as Pros of preparedness. Interestingly, the importance of preparedness for helping others (i.e., Assistance) was significantly higher for females than males (W = 138,204, *p* = 0.02). Around one-fourth of respondents did not form an opinion on the Cons of preparedness and chose the neutral option "undecided" for Avoidance (28% females; 25% males), Denial (23% females; 25% males) and Cost (22% females; 24% males). No significant gender differences were found for Uselessness "getting ready won't make a difference", Buck-passing "It is not my responsibility", and Cost "It takes too much time, effort, or money". Yet, differences were statistically significant for Avoidance "I would rather not think about bad things happening" (W = 138,848.5, *p* = 0.02) and Denial "It doesn't matter; disasters don't happen where I live" (W = 119,186, *p* = 0.03). However, one of the interesting results that emerged from the data was that gender differences in the composite scores for Pros and Cons of getting ready and the overall attitudes toward preparedness were not statistically significant (Table 7).


**Table 6.** Respondents' reactions to the Pros and Cons of disaster preparedness (from 1 = strongly disagree to 5 = strongly agree). *p*-values of the two-sided Wilcoxon rank sum test. The significant *p*-value is in bold.

**Table 7.** Two-sided Wilcoxon rank sum test results for the attitudes of males and females towards preparedness. Pros and Cons [0, 1]. Overall attitude [−1, 1].


**Risk perception and preparedness**.—A question directly addressed in this study was whether the perceived risk can motivate preparedness. We computed Spearman's rank correlation to assess the relationship between our risk perception results (likelihood, impact, self-efficacy and overall risk perception) for each of the reported hazards and the overall attitudes towards preparedness. We found weak correlations for the gender groups in all cases (rho < 0.20) suggesting that in our study the considered risk factors have a very low association with motivations for preparedness.

**Gender and intersectional factors**.—While gender is important it is shaped by other factors likely to reveal subgroup differences among males and females. We conducted an additional intersectional analysis considering gender related to age and educational background. This analysis revealed interesting findings that emerged during the process of the investigation.

*Gender and age:* We defined six categories for the comparison: YF (young female < 30 years), AF (adult female 30–50 years), OF (Older female > 50 years), YM (young male < 30 years), AM (adult male 30–50 years), OM (older male > 50 years). The mean and standard deviation of risk scores produced by each subgroup are displayed in Table 8. Kruskal-Wallis tests showed statistically significant differences in risk perception between subgroups (Table 8). Pairwise comparisons using Dunn's test indicated that several subgroups were observed to be significantly different (Table 9). Interestingly, risk perception for all hazards in females was significantly higher than in males in the same range of age (AF vs. AM and YF vs. YM). We only found significant differences between the same gender in males > 50 years (OM) who perceived higher risks in all hazards than males 30–50 years (AM).

**Table 8.** Mean and standard deviation of risk scores [0, 1] by gender and age and *p*-values from the Kruskal-Wallis test (α = 0.05).


**Table 9.** Results of pairwise comparison using Dunn's test (z-statistic and *p*-values) for risk perception according to gender and age (α = 0.05). Grey cells indicate significant differences between subgroups for all hazards.


\* *p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

Table 10 displays the mean and standard deviation of preparedness scores and the results of the Kruskal-Wallis test showing significant differences between subgroups (Table 10). Results of the pairwise comparisons by Dunn's test (Table 11) revealed significant intergender differences in the overall measures between adult females (AF) and young males (YM), older females (OF) and adult males (AM), older females (OF) and young males and (YM). We also found differences in the overall measures between the same genders: adult vs. older in females (AF vs. OF), adult vs. older males (AM vs. OM) and older vs. young males (OM vs. YM).


**Table 10.** Mean and standard deviation of preparedness scores by gender and age and *p*-values from the Kruskal-Wallis test (α = 0.05). Pros [0, 1] = in favour of preparedness; Cons [0, 1] = against preparedness; Overall [−1, 1] = Overall attitude toward pre-preparedness.

**Table 11.** Results of pairwise comparison using Dunn's test (z-statistic and *p*-values) for attitudes towards preparedness (α = 0.05). Grey cells indicate significant differences between subgroups for all the measures.


\* *p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

*Gender and Education***:** We categorized the sample into six subgroups according to gender and education background: PF (primary female), SF (secondary female), UF (university female), PM (primary male), SM (secondary male), UM (university male). There were significant differences between the subgroups (Tables 12 and 13). We found that risk perception for all hazards in females was significantly higher than in males at the same education level, i.e., secondary (SF vs. SM) and university (UF vs. UM). We also found that females with secondary education (SF) perceived risk to be significantly higher than their gender counterparts with university education (UM).

The Kruskal-Wallis test showed significant differences only between subgroups by gender and education for the overall attitudes toward preparedness (Table 14). Pairwise comparisons showed significant intergender differences for UM vs. PF (*p* = 0.012), UF vs. PM (*p* = 0.018) and UF vs. SM (*p* < 0.01). We also found differences between subgroups of the same gender: SF vs. PF (*p* = 0.012), and UF vs. PF (*p* < 0.01).


**Table 12.** Mean and standard deviation of risk scores [0, 1] by gender and education and *p*-values from the Kruskal-Wallis test (α = 0.05).

**Table 13.** Results of pairwise comparison using Dunn's test (z-statistic and *p*-values) for risk perception according to gender and education background (α = 0.05).


*\* p* < 0.05; \*\* *p* < 0.01; \*\*\* *p* < 0.001.

**Table 14.** Mean and standard deviation of preparedness scores by gender and education and *p*-values from the Kruskal-Wallis test (α = 0.05). Pros [0, 1] = in favor of preparedness; Cons [0, 1] = against preparedness; Overall [−1, 1] = Overall attitude toward preparedness. The significant *p*-value is in bold.


#### **4. Discussion**

This exploratory study looks at gender differences in two central aspects (1) risk perception of multiple hazards and (2) attitudes towards disaster preparedness. People (*n* = 1.014) from different countries (Italy, Sweden, Poland, France and Spain) were included in this study.

**Risk perception**.—Our results indicate gender differences in concerns about hazards expressed as the likelihood of the occurrence, personal consequences and self-efficacy. Females were more aware of the occurrence of disasters resulting from extreme weather and fire. Females also exhibited a higher perception of the potential consequences of extreme weather, fire and earthquake than males. These results are in line with previous findings confirming that women are more worried about natural hazards than men, especially if family members are threatened [10,42]. It is important to note that items did not necessarily have the same meaning for females and males as they may give priority to different hazards and/or show different concerns about the same hazards [43]. The questions of this section included "what in your view is the impact for you and your family?" Female respondents perhaps felt more oriented towards home and family when they thought about the presented hazards. Following this type of interpretation would yield an explanation as to why female respondents showed higher concerns. Our results reinforce a recent finding that reports no gender effects in the perceived vulnerability regarding terrorist attacks [44].

Self-efficacy has been identified as an important variable that should be considered within the context of hazards research, since it may be linked to the perceived risk and the adoption of hazard adjustments [45]. Our study confirms that gender is an important factor in the perceived self-coping abilities to deal with disasters. Males reported higher self-efficacies for all the presented hazards (i.e., extreme weather, fire, earthquake, hazardous accident and terrorist attack). A possible explanation suggests that women may be less confident than men, but this conceivably denotes a more realistic view of their own self-capacities [13].

To measure the overall risk perception (R) for the five hazards we computed the three subjective factors, namely: the likelihood of a disaster to occur (L), the impact if a disaster occurs nearby (I) and the perceived self-efficacy to cope with the disaster (E) through the following equation R = (L × I)/E. The resulting scores of males and females were compared. Our results showed that the risks were judged significantly higher by females in all hazards. The main reason that explains these results may be associated with the incorporation of self-efficacy (E) by assuming that this factor affects the risk perceived rather than simply considering the perceived likelihood (L) and impact (I) to measure risk ratings [17]. Additionally, we computed this approach (i.e., R = L × I) and the differences were also statistically significant (females scored higher risk than males) when applying two-sided Wilcoxon rank sum test for extreme weather conditions (W = 140,553.5, *p* < 0.001), fire (W = 141,096, *p* < 0.001) and earthquake (W = 138,242, *p* = 0.03). But the differences were not significant for hazardous materials accidents (W = 131,362, *p* = 0.53) and terrorist attacks (W = 129,487, *p* = 0.83). Overall, this finding emphasises gender differences in risk perception for natural hazards. But this might not be the case for man-made hazards, thus warranting further research to explore the factors that may account for it.

**Attitudes towards preparedness.**—Participants were also asked about the Pros and Cons of preparedness to measure their individual interest in getting prepared. Responses to some items differed between males and females. The importance of being prepared to help others (Assistance) was significantly higher in female respondents. This result was in line with previous studies attesting that women tend to be more altruistic than men (see for some references [46–49]. The statement that disasters "don't happen where I live" (Denial) had significantly higher scores in male respondents denoting differences in judgments based upon such events [43] and optimistic bias [50]. The 'It will not happen to me' belief is a very important aspect of preparedness that has already been reported [51] since overconfidence can keep individuals from realizing how little they know and how much information they may need to be ready. By contrast, females were significantly more likely to "not think about bad things happening" (Avoidance) than males. This result supports previous studies attesting that gender is a significant predictor of coping through avoidance [52,53]. Avoidance here can be associated with information avoidance leading to misinformation which has been recently analysed in the context of the COVID 19 pandemic [54,55]. More research is needed to explore gender influence on this aspect of behaviour in the context of disasters.

**Risk perception and preparedness.**—A question addressed in this study was whether the perceived risk can motivate preparedness. Risk perception has been considered a predictor or correlates of preparedness behaviour. While some studies found that risk perception predicted or was associated with preparedness others found no effects [27]. Overall, results showed that most respondents (both males and females) were in favour of preparedness. More importantly, despite the gender differences in risk perception, there were no significant differences in attitudes towards preparedness. We found weak correlations for the gender groups in all cases (rho < 0.20) suggesting that in our study the considered risk factors have a very low association with motivations to seek preparedness.

**Gender and intersectional factors.**—Previous results open a new question addressed in this study. There are differences/similarities between women and men but which women? and which men? An intersectional approach was used to analyze the multiplicative impact of gender when combined with age and educational background which also shapes the identity, perceptions, and attitudes of individuals towards disasters. This analysis produced more detailed information showing the differences between different but interdependent categories and factors. We compared six subgroups categorized by gender (male; female) and age (<30 years; 30–50 years; >50 years). The subgroups were observed to be significantly different, especially between subgroups of different genders. Young and adult females perceived higher risks than their gender counterparts of the same age. There were also gender differences in preparedness between subgroups of different ages denoting that age is also a contributing factor to people's attitudes (e.g., females in a higher age range are more motivated for preparedness than men in a lower age range). We defined six subgroups according to gender (male; female) and education (primary; secondary; university). We found that risk perception for all hazards in females was significantly higher than in males at the same education level (e.g., secondary and university education background). Furthermore, females with secondary education (SF) perceived risks to be significantly higher than their gender counterparts with university education (UM). When it comes to the attitudes towards preparedness, we found no significant differences between subgroups in the reported scores for the pros and cons of getting ready for disasters. However, females at a higher level of education have more positive attitudes towards preparedness.

The current study has several strengths. First, it contributes to the literature by providing a general approach to exploring gender on public perception of multiple hazards/disasters, which predominantly has been concerned with specific disasters and affected communities. Second, the datasets generated in this study are available allowing third parties to conduct further research. Third, this study proposes a new approach to computing the subjective factors (likelihood, impact and self-efficacy) of risk perception and personal attitudes toward preparedness (Pros and Cons). Fourth, the resulting scores were presented in more analytical forms, i.e., [0, 1] and [−1, 1] for better understanding and further comparison with other datasets. Fifth, the intersectional analysis presented here (combining gender age and education) will allow policy-makers and first responders to better identify subgroups and factors among the population to implement DRR policies and practices. Finally, as mentioned the results produced here not only have scientific value but also have the potential to inform EU policymakers and first responders. The current study also has its limitations. First, compared with other studies a "small" sample size was employed (*n* = 1.014). However, we believe that the subset of the population used was representative thus providing a sufficient amount of information to conclude the EU population. Second, the study is limited in scope, i.e., whether or not gender is "significant" along with other two factors (i.e., age and education level). Third, the association between

risk perception and preparedness was not found, perhaps due to the design of the items in the questionnaire as this was not the initial purpose of the study. Fourth, non-parametric methods were conducted for the statistical analysis potentially leading to less powered results. Finally, the results presented here are rather indicative than definitive.

#### **5. Conclusions**

Results presented in this study constitute the first process of gender analysis (e.g., data collection, data processing, and analysis) and advocate to conduct of the second process which is interpretative in nature [56]. The gender discrepancies may reveal the underlying mechanisms apart from biological and physiological differences such as everyday life behaviours and beliefs as well as stereotypes derived from gender norms [57]. Conceivably, socioeconomic and cultural differences between men and women are more evident in lowerincome countries leading to a higher exposure of women to risks in case of a disaster [5]. The present results suggest that gender differences in relation to risk perception of multiple hazards might still be present in European societies. The different social roles and activities of men and women within the household and community are examples of how gender norms and ideals manifest. The role of nurturer and caregiver primarily played by women has been associated with greater concern about the risk of potential disasters and the well-being of others [58]. Also, different gender roles can be reinforced in disasters because expectations for men and women are usually based on stereotypes. For instance, a recent study focused on actions during a large Swedish forest fire, indicated that women were praised when they followed the traditional norms but denigrated when they performed what was perceived as male-coded tasks [59].

Our results suggest the same predisposition of females and males to seek preparedness. Women are slightly present in emergency planning and disaster management programs but more involved in household and community care in practice [57,60,61] and often ignored in official evaluations after disasters [5]. It is argued here that gender skills may benefit the prevention and mitigation of hazard situations.

Although limited to risk perception and preparedness, the outcomes of this study can provide insights into the integration of gender-sensitive practices in disaster preparedness and response. First, conducting more qualitative and quantitative research to better understand gender-based roles and responsibilities is highly desirable. For studying a complex area such as gender constructs and roles, multi-disciplinary research could be beneficial. Second, improving women's capacities and knowledge (training and education) can increase individual and community resilience. Third, promoting policies and actions to involve women in official emergency management programs and decision-making is essential to minimize gender gaps in disaster planning and response. Much work remains to be done to systematically integrate gender analysis into relevant domains of safety science and technology—from strategic considerations for establishing research priorities to guidelines for establishing best practices in formulating research questions, designing methodologies and interpreting data.

The practical implications of this study can be summarized as follows:


**Author Contributions:** Conceptualization, A.C. (Arturo Cuesta), D.A. and A.C. (Antonio Carnevale); data curation, A.C. (Arturo Cuesta); Formal analysis, A.C. (Arturo Cuesta); Funding acquisition, all authors; investigation, A.C. (Arturo Cuesta); methodology, A.C. (Arturo Cuesta), D.A. and A.C. (Antonio Carnevale); Project administration, D.A.; supervision, A.C. (Arturo Cuesta) and D.A.; writing—original draft, A.C. (Arturo Cuesta); writing—review & editing, F.A. All authors have read and agreed to the published version of the manuscript.

**Funding:** This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 832576.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Ethics Committee of the University of Cantabria (CE Proyecto 06/2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The datasets in this study are available from the corresponding author upon request.

**Acknowledgments:** The authors would like to thank the European Commission and the members of the ASSISTANCE consortium for their collaboration and support to this study. We would like to express our very great appreciation to the team members involved in the translation of the questionnaire into Italian (CYBERETHICS LAB SRLS), Swedish (RISE RESEARCH INSTITUTES OF SWEDEN AB), Polish (PRZEMYSLOWY INSTYTUT AUTOMATYKI I POMIAROW PIAP), French (THALES SA) and Spanish (UNIVERSIDAD DE CANTABRIA).

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **The Need for a Preparedness Training Model on Disaster Risk Reduction Based on Culturally Sensitive Public Health Nursing (PHN)**

**Haris Sofyana 1,\*, Kusman Ibrahim 2,\*, Irvan Afriandi 1, Erna Herawati <sup>3</sup> and Heru Santoso Wahito Nugroho <sup>4</sup>**


**Abstract:** The Indonesian Disaster Risk Index (IRBI) in 2018 found that 52.33% of districts or cities in Indonesia were at high risk of natural disasters and the others were at moderate risk. The World Risk Index places Indonesia at number 33 in the very high-risk category. The policy direction of the Implementation of Disaster Management in Indonesia in 2020–2024 is to increase disaster resilience toward sustainable prosperity for sustainable development. Purpose: This study aims to identify the various needs for a culturally sensitive PHN-based disaster risk-reduction preparedness training model. Methods: This study used a descriptive qualitative research design. Data collection was done through in-depth interviews, Focus Group Discussions (FGDs), and expert panel stages in the Indonesian language. Samples involved in the research included 4 experts and 11 informants. Results: There were 10 themes generated from the results. The analysis results revealed that the level of knowledge, attitudes, and skills of the community is still low. Almost all of the people of Mekar Mukti Village stated that they had never received community-based disaster management training. Conclusions: The study findings highlighted the importance of the Disaster Risk-Reduction Preparedness Model Based on Culturally Sensitive Public Health Nursing for the community.

**Keywords:** preparedness training; disaster risk reduction; public health nursing; culturally sensitive

#### **1. Introduction**

Indonesia is an archipelago prone to disasters. It is located in the path of major earthquake sources such as the megathrust–plate subduction zone and active faults on the mainland. Based on the BNBP report, 295 faults have been identified to be active fault segments that have the potential to produce earthquakes above 6.5 magnitude [1]. Thus, Indonesia's natural disaster risk is high. The Indonesian Disaster Risk Index (IRBI) in 2018 found that 52.33% of districts or cities in Indonesia were at high risk of natural disasters while the others were at moderate risk. These conditions place Indonesia as one of the countries with the highest rates of natural disasters in the world. Indonesia's Natural Disaster Risk Index (IRB) statistical data in 2021 for tsunamis, floods, landslides, droughts, and forest fires are also relatively high compared to other countries [2].

In 2021, there were 5400 natural disasters in Indonesia, which was an increase from 2500 natural disasters in 2018. This illustrates the very high rate of natural disasters in Indonesia. Floods (1310 incidents), tornadoes (814 incidents), and landslides (633 incidents) are the dominating natural disasters. Natural disasters in 2021 caused more than 8.6 million people to suffer and be displaced, along with the deaths of 676 people. Furthermore, more than 142,000 houses and 3700 education, health, office, road, and bridge facilities were impacted by the natural disasters [3].

**Citation:** Sofyana, H.; Ibrahim, K.; Afriandi, I.; Herawati, E.; Wahito Nugroho, H.S. The Need for a Preparedness Training Model on Disaster Risk Reduction Based on Culturally Sensitive Public Health Nursing (PHN). *IJERPH* **2022**, *19*, 16467. https://doi.org/10.3390/ ijerph192416467

Academic Editors: Amir Khorram-Manesh and Krzysztof Goniewicz

Received: 23 September 2022 Accepted: 10 November 2022 Published: 8 December 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

In addition to natural disasters, Indonesia is also still trying to control the spread of COVID-19 that caused the loss of 100,000 people and has been labeled as a National Non-Natural Disaster [4]. National data on COVID-19 cases, as of 17 March 2022, recorded 153,411 deaths with new confirmed cases of 9528 people, increasing to 5,958,610 people. The COVID-19 pandemic in Indonesia has had an impact on almost all development sectors. One of these development sectors is the Reform (Strengthening) of the Disaster Resilience System, developed to be able to overcome non-natural disasters on a national scale without reducing resilience to handle natural disasters occurring at the same time as non-natural disasters [3]. Data from the Centre For Research On The Epidemiology Of Disasters (CRED) for 2008–2018, revealed that every year Indonesia occupies the top 10 in the world as the country most frequently affected by natural disasters and the country with the highest number of deaths from natural disasters [5]

West Java is one of the areas in Indonesia with a high frequency of natural disasters. 4465 of 5957 villages were included in the category of high-level disaster-prone villages [6]. Additionally, there are numerous active volcanoes in West Java, including Mount Salak, Mount Gede, Mount Tangkuban Parahu, Mount Papandayan, Mount Guntur, Mount Galunggung, and Mount Ciremai. The complex geographical conditions of the West Java region paired with the largest population in Indonesia make this province's natural disaster risk high. Based on the 2020 Indonesian Disaster Risk Index (IRBI), West Java Province has a risk index of 145.81 (high) [4].

This further pushes the direction of the policy for the Implementation of Disaster Management in Indonesia for 2020–2024, which aims to improve disaster resilience toward sustainable prosperity for sustainable development [7].

Strengthening the paradigm shift in disaster management from conventional emergency response to disaster risk reduction is one strategy to accomplish this. According to the Disaster Risk Reduction Paradigm (DRR), the community is an active participant in disaster management, thus it must have the necessary knowledge, attitudes, and abilities [8]. This means that in order to effectively manage disasters, the paradigm of disaster risk reduction also calls for community empowerment, which may be accomplished through a variety of community socialization, education, and training programs.

The study by Setiawan et al. (2017) recommend the importance of training to empower rural communities living in disaster-prone areas to normalize the physical and psychological problems of natural disaster victims [9]. This is in line with the study conducted by Salasa et al. (2017) that identified community preparedness as an important key to minimizing health problems resulting from natural disasters. These results also show that the empowerment process through a contingency planning approach is able to increase youth preparedness against life threatening situations due to disasters. It is advised for all disaster activists to empower youth with contingency planning to increase their readiness to face life threatening situations [10]. In addition, it is necessary to adjust communication strategies to the sociocultural aspects of the local community to effectively emphasize the importance of community empowerment as it would be more easily accepted and relevant to the needs of the people. Culture has become one of the factors for a community's ability to survive during disasters [11].

Other studies have linked this cultural aspect approach with approaches to nursing care for communities affected by disasters. The concept of nursing actions based on cultural considerations provides strong evidence that in a disaster, community nurses must take advantage of cultural forms such as bonds and relationships by providing information and supplements, respecting culture such as local rules and characters as well as healing and comforting the affected residents [12]. In line with this, nurses need to consider the socio-cultural scope of the community and long-term care in order to respond effectively to disasters. This can start from planning, noting positive and negative consequences of assistance, and individual planning in the community [13]. This is in accordance with Senday's disaster risk reduction framework, which is establishing a national platform for disaster risk reduction. This becomes a general term used as a national policy for coordination and as a policy guide on multisectoral and interdisciplinary disaster risk reduction that involves all relevant entities in a country, whether public or private. Disaster risk reduction requires knowledge, capacity and input from various sectors and organizations, including UN agencies present at the national level. National platforms provide the means to enhance national action to reduce disaster risk, and they represent national mechanisms for the International Disaster Reduction Strategy [14]. Facilitating the community's ability to anticipate, prepare for and recover from disasters is an important component of the UNISDR strategy for disaster risk reduction [15].

One of the theories in nursing that is closely related to community empowerment and cultural care is the theory of the Transcultural Nursing model, published by Madeleine Leininger in 1978. The theory requires an awareness and appreciation of cultural differences that influence providing nursing care with approaches to respecting individual cultural values [16]. Therefore, nurses are required to have knowledge of and engage in a practice that is based conceptually on culture [17]. Transcultural nursing is an area of community and cultural science in the field of nursing that focuses on the differences and similarities between cultures by respecting care, health and illness based on cultural values, beliefs and actions. Thus, several aspects of culture can be studied in an effort to provide nursing care [18]. Based on the background, the research problem is how to create a culturally sensitive disaster risk-reduction model in disaster-prone areas. Therefore, this study aimed to identify the various needs for a culturally sensitive Public Health Nursing (PHN)-based disaster risk-reduction preparedness training model. This study is imperative to explore the need for community preparedness in disaster management based on local culture.

#### **2. Materials and Methods**

#### *2.1. Study Design*

This study used descriptive qualitative research design to identify the various needs for a culturally sensitive Public Health Nursing (PHN)-based disaster risk-reduction preparedness training model.

The initial quantitative survey had already been conducted in order to identify the model's demands for training and testing, and to evaluate the model's effectiveness using a small sample, track its development, and alter the procedure in response to input that would be seen throughout model training. Sixty respondents from the Pasir Jambu subdistrict, Bandung district, and Sugih Mukti village community participated in this survey, with 22 women (38.3%) and 37 males (61.7%) being the majority. The composition of the initial respondents consisted of the following elements: government/village officials, community leaders, health cadres, and youth organizations. According to the survey results, earthquakes (80%), landslides (11.7%), and high winds (8.3%) are the top three disaster hazards that the residents of Sugih Mukti Village feel most at risk from. The examination of knowledge and attitudes about disasters and disaster management produced very unsatisfactory results, with the average knowledge score being 58.75 (SD 12.54) and the average attitude score being 59.31 (SD 10.57). None of the responders could show that they had the necessary abilities or fundamental rescue techniques for catastrophe victims, which were self-performed by the community.

#### *2.2. Settings and Participants*

The study setting was Marga Mukti Village, Pasir Jambu District, Bandung Regency, West Java, Indonesia. The study was conducted from April to August 2022. The participants were divided into key informants and general informants, with the following inclusion criteria: academics, experts chosen by researchers as conceptual reference materials, indigenous people of the local community who serve as community leaders and have experienced disasters, and traditional stakeholders who are used as reference figures and experts. A total of 4 experts and 11 informants from the defense forces, government officials, community leaders, BNPB, business owners, and professional nurses were selected through a purposive sampling technique.

#### *2.3. Data Collection*

Data collection was done through in-depth interviews, Focus Group Discussions (FGDs), and expert panel stages in the Indonesian language. In-depth interviews were conducted for 30–60 min, and FGDs were conducted for 60–90 min. Each respondent's responses were calcified until data saturation was reached. The protocol for preventing the spread of COVID-19 is carried out by the standards established by the Indonesian government, including the use of masks and face shields, establishing a minimum distance of one meter between participants, washing hands with disinfectant before entering and leaving the room, and measuring body temperatures before participants enter the room. The expert panel stages were held virtually by using Zoom for 60 min.

Field notes are made by writing down everything significant that happened while conducting the research, including taking photos of significant areas. The study team conducted the interviews with the aid of two administrative officers who supported the interview procedure. Data saturation was accomplished after the researcher identified repeated answers from research informants with meanings (themes) that lead to the same topic.

#### *2.4. Data Analysis*

Data analysis is carried out in four stages of analysis, as described by Leininger (2002). The first stage is collecting, describing, and documenting raw data. At this stage, researchers collect data, followed by explaining and documenting the raw data obtained from interviews, FGDs, surveys, and documentation studies. The second stage is the identification and categorization of descriptors and components, specifically, selecting and classifying descriptors and data elements that are the primary study topic. At this point, the emphasis is on identifying the contributing elements and impediments that hinder the creation of a community-based disaster risk-reduction model with a culturally sensitive strategy to enhance community preparedness. The third stage is the pattern and contextual analysis, which identifies patterns of interactions, values, beliefs, and practices when they are related to informants and data gathered through field observations. Identifying the key themes, research findings, and patterns of community engagement in disaster riskreduction strategies in the community are covered in the fourth stage, which is titled "major themes, research findings, theoretical formulations, and recommendations". The steps of interpretation and synthesis of results form the foundation at this level.

#### *2.5. Ethical Considerations*

Ethical approval for this study was granted by the Health Research Ethics Committee of the Ministry of Health, Republic of Indonesia, Bandung Polytechnic of Health (ref. No. 01/KEKP/EC/II/2022). Permission to conduct the study was also obtained from the community leader of Marga Mukti Village, Pasir Jambu District, Bandung Regency, West Java, Indonesia. The voluntary and confidential nature of the study was explained to participants before each interview, Focus Group Discussions (FGDs), and expert panel stages process. To enhance confidentiality, pseudonyms were used in the study.

#### *2.6. Trustworthiness*

Trustworthiness was completed by identifying research findings evaluated through a process including credibility, confirmability, meaning in context, recurrent patterning, saturation, and transferability. Credibility was established by participating in the daily lives of informants, ongoing observation, triangulation, peer debriefing, and community support groups, community leaders, and social organizations that promote DRR. Confirmability was clarified by stating ideas or findings that were heard, seen, or experienced with key informants and several general informants. In order to interpret and grasp the significance of meaning in context, researchers incorporated the information from interviews, observations, and documents. All key informants and some general informants then corroborated regarding the interpretation. All information was based on contextual reality and the environment. Recurrent patterning was carried out with researchers who used informants' repeated experiences, expressions, events, or activities in relation to community-based disaster risk reduction. Transferability and saturation were met when the data collected reveal duplication of content related to ideas, meanings, experiences, descriptions, and other similar expressions from the informants or repeated observations. Further research findings were reported in a rich language style, including quotes, comments, and stories that added to the richness of the report and provided for understanding the context of the experience in which it all took place. Researchers attempted to facilitate transferability by providing detailed documentation across all phases of research.

#### **3. Results**

#### *3.1. Document Analysis*

Based on the document analysis, Sugihmukti Village is located in Pasir Jambu District, Bandung Regency, West Java Province. This village located in a mountainous highland, with an area of 1767.96 km2, at 107.407795 east longitude and −7.19077 south latitude. The distance from the village to the subdistrict capital is as far as 7000 km. Sugih Mukti Village is a disaster-prone village due to its unstable soil structure and is located in an area adjacent to an active volcano and geothermal mining activities in the Patuha Mountains. Previous research was conducted on community representatives involving 60 respondents from two disaster-prone areas in the village of Sugih Mukti. The results reveal that the priority disaster risks that are most experienced by the people of Sugih Mukti Village are earthquakes (80%), landslides (11.7%), and strong winds (8.3%), while the others are fire disasters and conflicts between communities. The results of measuring the level of knowledge, attitude and skill of the community about disaster, are obtained: knowledge, 58.75% (SD 12.54); attitude, 59.31% (SD 10.57); and skills in providing basic health assistance, 0%, indicating the inability to take action. Based on these results, this research was developed.

#### *3.2. Expert Discussions and National Seminar*

The initial stage of the research began with expert discussions and national seminars with four key informants from BNPB, PPKK-Kemenkes, PPNI, and academics of disaster nursing experts. The results of expert discussions and seminars were identified as central themes in the development of a culturally sensitive Public Health Nursing (PHN)-based disaster risk-reduction preparedness training model. The resulting research themes are as follows.


#### *3.3. Focus Group Discussions (FGDs)*

The next stage was FGDs and in-depth interviews with 11 informants from the defense forces, government officials, community leaders, BNPB, business owners, and professional nurses. The results of FGDs and in-depth interview are us follow (Table 1):










Based on Table 1, five themes are generated which become the basis for developing a preparedness training model for Public Health Nursing. The five themes are:


#### **4. Discussion**

#### *4.1. Description of Model Requirements*

Figure 1 describes the results found through investigating the need for a Public Health Nursing (PHN) disaster risk-reduction preparedness training model as a guide for community nurses. The model is an integrated implementation of community care and empowerment stages in disaster-prone areas. The thematic exploration results from all community members show that a disaster risk-reduction preparedness training model that is integrated with community empowerment is very much needed.

**Figure 1.** Thematic description of Disaster Risk Reduction (DRR) needs in community nursing.

This need is in line with BNPB's values, which are preparedness and level of community independence as the most important aspects for disaster management, followed by the help of family members, friends, SAR team, health team, and surrounding components [19]. To strengthen this opinion, the community becomes the focus of studies in disaster risk reduction, which starts from groups of school children and adolescents, health cadres, and community leaders as targets for empowerment in disaster risk reduction. Community empowerment in disaster risk reduction requires a community-centered, interprofessional socialization approach that is systematic, culturally informed, and focuses on modeling the role of trainers, caring for officers as well as attention to the needs of participants [20].

Involving the community in disaster risk-reduction studies will increase the awareness of the community. Research on public health recommended the need for a stronger emphasis on public health in managing disasters. For example, the Reserve Corps provides civil medical training to improve public health infrastructure and provides greater opportunities to collaborate with communities in disaster management [21] Lebowitz (2015). In order to make these various components effective, a model is needed that can accommodate the increasing capacity of the community. According to Paton (2019), facilitating the community's ability to anticipate, prepare, and recover from disasters is an important component of the UNISDR strategy [15]. Pourvakhshoori et al.'s (2017) research provides clarity that nurses are part of the health profession which has a very important role [22].

Understanding the experience of nurses in disasters can help identify problems in the disaster nursing services field, which can be resolved by better planning and preparation [22]. One of the major problems in the disaster management system is the lack of planning to employ and organize volunteers in the health sector during a disaster [23]. The nursing process approach, using the concept of the Transcultural Nursing model, is used as an effort to accelerate the Public Health Nursing (PHN) program, which is in accordance with the government program.

The integrated implementation program of Public Health Nursing in Disaster Risk Reduction (DRR) and the transcultural nursing approach is very relevant to community groups that are socioculturally heterogeneous. In the context of providing holistic service, services with a culturally sensitive approach are needed; nurses must adapt to the system, norms, and culture that apply in a community.

This is in line with the anthropological view, which believes that cultural factors influence human behavior. When faced with danger, people not only consider the dangers they can face, but also prioritize factors such as social values, religious beliefs, traditions, and attachments to a particular place or location. Culture has the power to increase or decrease a society's vulnerability to disasters. Lack of consideration of the cultural aspects of the affected communities can hinder effective DRR strategies, thereby increasing the vulnerabilities of the affected communities rather than reducing them. Therefore, culture has the power to increase or reduce community vulnerability to disasters [11] Kulatunga (2010). Research by Kertamuda and Chris (2012) explains that the culture of resignation and patience in the three largest ethnic groups on the island of Java (Betawi, Sundanese, and Javanese) is still very dominant in responding to disasters [24].

#### *4.2. Constructed Model Requirements*

Figure 2 shows the structure of the model that was built based on the thematic analysis of the need for disaster risk-reduction preparedness training. The training model is constructed from the concept of community nursing theory (PHN) as well as the strengthening of the transcultural nursing model. The steps of community empowerment in disaster management are in line with the steps of the community nursing process, which are assessment, planning, intervention, implementation, and evaluation.

**Figure 2.** Constructed requirements for the Public Health Care-Based Disaster Preparedness Training integration model.

The three types of priority disasters that the people of Sugih Mukti frequently experience are earthquakes, landslides, and hurricanes. This result is similar to the Bandung Regency BPBD report, which identifies the types of priority disasters in Bandung Regency: floods, landslides, cyclones, droughts, and earthquakes. Sugih Mukti Village, Pasir Jambu District, Bandung Regency, is included in the list of areas prone to landslides and earthquakes [25]. This is due to the geographical location of Sugih Mukti Village. It is located within a geothermal exploration area, also known as the Geothermal Patuha Mountain in Bandung district. The activity of the Geodipa Energi project is a leading factor that increases Sugih Mukti Village's vulnerability to landslides and volcanic earthquakes [26]. Research conducted on the residents reveals that the level of knowledge, attitudes, and skills of the community are still low. Almost all of the people of Mekar Mukti Village stated that they had never received community-based disaster management training. These results are in line with research conducted by Setiawan et al. (2017), which emphasizes the importance of training to empower rural communities living in disaster-prone areas that will normalize the physical and psychological problems of natural disaster victims [9].

The role of the community nurse becomes very important in this situation. Research by Walsh et al. (2012) explains that effective disaster preparedness, response, and recovery requires a well-planned, concerted effort by experienced professionals who can apply specific knowledge and skills in critical situations. The results can provide a useful starting point to identify the level of competence expected of health professionals in disaster medicine and public health [27]. A study by Gulzar et al. (2012) provides results that support this. The intervention, collaboratively chosen by the CHN, was shaped by a planning framework that fits the nursing process. It is a method used to take a holistic approach in the earthquake-affected area. Such a framework includes four phases; assessment, planning, implementation, and evaluation. The framework provides systematic and specific directions for healthcare providers while promoting public health [28].

Communities in disaster-prone areas must have the ability, strength, and independence in managing disasters at every stage. Results from a study link cultural factors with Disaster Reduction Risk (DRR) activities and highlight how culture has influenced DRR activities. In some ways, culture has become one of the factors of community survival from disasters. Therefore, it can be said that culture has the power to increase or decrease people's vulnerability to disasters [11]. An expert opinion, Rush et al. (2019), explain that disasters can occur in any community at any time. Such an opinion becomes true in any disaster. Nurses have a central role in managing and preparing for medical care during these catastrophic episodes [29]. Study results from Salmani et al. (2019) show the importance of having working management. It is necessary to have complete legislation, NGOs, and sociocultural factors, preparedness, response, retention, relocation, termination, and follow-up in each nurse's role in disaster-prone areas [23].

However, not all nursing roles can be easily implemented in disaster situations. A study by Ahmadi et al. (2018) found that the elders of a community that went through a disaster faced various challenges in their everyday life, which means that more efforts were needed to help them reach the stage of recovery. Taking into account these conditions, the need for community empowerment during the disaster period must not be postponed. Training the community to be independent must be done constantly by taking into account local cultural conditions [30]. The results of research in China show several conditions that push the need for community empowerment in disaster locations, which follow [20]. The behavior of the trainers influences the participants through their interactions during the training process. A mental health training program is needed to identify the needs of disaster workers and victims. Building a systematic interprofessional education strategy is required. Systematic interprofessional education can assist in responding to complex local problems due to a collaborative approach because it bridges the gap between theory and practice, and solves local needs and international guidelines.

Training regulations are needed to maintain and monitor the quality of the content, standards, ethics, and codes of conduct at all levels. A community-centered interprofessional education approach is required. It focuses on modeling the role of trainers, caring for staff, attending to the needs of trainees, and building a systematic, culturally informed, and informed interprofessional education strategy.

The results of the study by Lebowitz (2015) supported this opinion, regarding the need for a stronger emphasis on public health workers in managing disasters. For example, civil medical training for reserve corps can be done to improve public health infrastructure and provide greater opportunities to collaborate with the community in handling disasters [21].

Raising awareness through disaster education and socialization is very necessary. By doing so, everyone can understand risks, be able to manage threats, and, in turn, contribute to encouraging community resilience from the threat of disaster. Additionally, social cohesion, collaboration, and mutual trust are the adhesive values of the society that have been nurtured, both individually and collectively by the community, to prepare for, respond to, and rise from adversity caused by disasters.

#### **5. Limitation**

This research is limited since the model has not been tested or used in this study, and its effectiveness has not been determined. During the following stage of the research, the model will be tested and used. Additionally, obtaining data relating to culturally sensitive features and local strengths has not been thoroughly investigated, and requires a more thorough investigation in future research.

#### **6. Conclusions**

The study findings highlight the importance to the community of the Disaster Risk-Reduction Preparedness Model Based on Culturally Sensitive Public Health Nursing. The community is approached using a combination of the Public Health Nursing (PHN) and Transcultural Nursing principles. An integrated model for disaster preparedness training based on public healthcare as a solution to empower communities in managing disaster risk is validated, constructed, and designed. This model consists of five main components: nursing care assessment instruments and introductory surveys for disasterprone communities; public health nursing-based disaster preparedness training integration curriculum; public health nursing-based disaster preparedness training module integration training process; public health nursing-based disaster preparedness training process; and maintenance of training results.

**Author Contributions:** Conceptualization, H.S., K.I., E.H. and I.A.; methodology, H.S., K.I., E.H. and I.A.; software, H.S. and H.S.W.N.; validation, H.S. and K.I.; formal analysis, H.S., E.H. and H.S.W.N.; investigation, H.S. and K.I.; resources, H.S., I.A. and H.S.W.N.; data curation, H.S. and E.H.; writing—original draft preparation, H.S., K.I., E.H., I.A. and H.S.W.N.; writing—review and editing, H.S., K.I., E.H., I.A. and H.S.W.N.; visualization, H.S., K.I. and I.A.; supervision, K.I., E.H. and I.A.; project administration, H.S. and H.S.W.N.; funding acquisition, H.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Ethical clearance: this research paper has passed the ethical clearance test and received approval from the Institute for Ethics Studies, Polytechnic of Health Ministry of Health, Bandung (No. 01/KEPK/EC/II/2022).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the subject(s) to publish this paper.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The acknowledgment of support was obtained from Dean of the Faculty of Medicine Padjajaran University, Dean of the Faculty of Nursing Padjajaran University and Dean of the Faculty of Social and Political Sciences, Padjadjaran University.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **A Multi-Attribute Decision Support System for Allocation of Humanitarian Cluster Resources Based on Decision Makers' Perspective**

**Sara Rye 1,\* and Emel Aktas <sup>2</sup>**


**Abstract:** The rush of the humanitarian suppliers into the disaster area proved to be counterproductive. To reduce this proliferation problem, the present research is designed to provide a technique for supplier ranking/selection in disaster response using the principles of utility theory. A resource allocation problem is solved using optimisation based on decision maker's preferences. Due to the lack of real-time data in the first 72 h after the disaster strike, a Decision Support System (DSS) framework called EDIS is introduced to employ secondary historical data from disaster response in four humanitarian clusters (WASH: Water, Sanitation and Hygiene, Nutrition, Health, and Shelter) to estimate the demand of the affected population. A methodology based on multi-attribute decisionmaking (MADM), Analytical Hierarchy processing (AHP) and Multi-attribute utility theory (MAUT) provides the following results. First a need estimation technique is put forward to estimate minimum standard requirements for disaster response. Second, a method for optimization of the humanitarian partners selection is provided based on the resources they have available during the response phase. Third, an estimate of resource allocation is provided based on the preferences of the decision makers. This method does not require real-time data from the aftermath of the disasters and provides the need estimation, partner selection and resource allocation based on historical data before the MIRA report is released.

**Keywords:** disaster response; need estimation; resource based; MADM; AHP; MAUT; utility theory; humanitarian clusters; humanitarian supply

#### **1. Introduction**

The overall aim of the disaster relief operation is to ensure the survival and health of the maximum possible number of victims [1,2]. This operation is required to benefit the affected community's development and reduce the vulnerability of the population to future hazards. In the days and weeks immediately following a disaster, the basic relief supplies and services are provided free of charge to save and preserve human lives. This enables families to meet their basic needs for medical and health care, shelter, clothing, water, and food. However, the problem is in the early hours after the disaster strike, there is no official estimate of these needs. The first official UN report of preliminary Multi-Cluster/Sector Initial Rapid Assessment (MIRA) is released three days after the disaster strikes. This present problems as some quick decisions and actions need to be taken in the absence of detailed assessments and lack of appropriate Decision Support Systems (DSS) which may lead to loss of lives amongst others. For example, during the UK flood in 2014, even though good warning systems were in place [3], the lack of decision-making tools, led to the death of seven people and the destruction of 1700 homes. It is critical to understand that these negative effects happened in the presence of the exact knowledge of where and when the storm/flood would strike, in a developed country with a sufficient budget for prevention. Therefore, the lack of DSS in developing countries would be far more devastating. Like

**Citation:** Rye, S.; Aktas, E. A Multi-Attribute Decision Support System for Allocation of Humanitarian Cluster Resources Based on Decision Makers' Perspective. *Sustainability* **2022**, *14*, 13423. https://doi.org/10.3390/ su142013423

Academic Editors: Amir Khorram-Manesh and Krzysztof Goniewicz

Received: 14 September 2022 Accepted: 11 October 2022 Published: 18 October 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

other decision-making problems with human elements, the preferences of decision makers also play a role in this resource allocation under uncertainty. The objective of this research is to provide a DSS for partner selection/ranking using only the data available before the release of MIRA report to reduce the partners' proliferation problem. So, this research addresses two questions: First to what extent is possible to estimate the resources required for humanitarian response operation in the absence of MIRA report within the first three days. Second to what extent is possible to optimise the resource allocation decisions within humanitarian response operation considering the preferences of the decision makers.

The preferences become important when you note that decision makers in disaster affected area, operate based on their background, beliefs, experience, and political views. For example, in some areas where the conflicts are an issue, some decision makers are reluctant towards the use of military relief supplies. Some decision makers due to experience might prefer government or NGO, International or local suppliers, UN or voluntary resources, and so on. To answer the above questions, this research aims to develop a DSS which could optimise the allocation of resources prior to the release of MIRA report and based on the decision makers' preferences. The DSS includes a framework for need assessment of the affected population to enhance the need-based resource allocation through decision makers' preferences. In this process, various humanitarian guidelines and official reports were used to argue that it is possible to outline minimum standard requirements for each disaster type and based on the affected countries' socio-economic characteristics. This estimates a list of requirements in disaster situation for affected population with priorities. This list can be the basis for need-based resource estimates. The estimates then are used to optimize the allocation of the resources available by humanitarian suppliers based on the principles of utility theory and resource-dependency theory. The agent-based optimisation technique which is where all DSS methods above overlap, are in general based on the principles of decision theory Neumann-Morgenstern in the 40s. This allows agents to select decision criteria, evaluate, and compare the options and act upon them. This can be viewed as combination of Utility theory and Probability theory [4]. Decision-making in a disaster situation, in particular fits well within both utility and probability as it is a decision scenario under uncertainty. The literature in this ilk are mainly divided in two branches; Rational choice and Expected utility (EU) and Behavioural and Prospect theory [5]. The former has arisen from mathematical literature, and provides clear formulations, whereas the latter is more practice-based and tries to show the controversies in the Expected utility theory [6]. This research is not an attempt to focus on the challenges facing Utility theory or study how and why decision makers decide the way, they do. The research focused on Utility theory to maximise the preferences of the decision makers who decide based on the reasons out of scope of this research. In fact, the investigation into the reasoning behind their preferences can be the subject of further studies on Prospect theory by other scholars.

This research provides a technique for supplier ranking/selection in disaster response by analysing the archival data, and decision support tools. Using Linear programming optimisation, Analytical Hierarchy processing (AHP) and Multi-attribute utility theory (MAUT) a DSS is developed based on secondary data. The DSS includes Phase1-ESTIMATION of the need in four humanitarian clusters (WASH, Nutrition, Health and Shelter). This will be the basis for estimating the demand of the affected population. Phase 2-OPTIMISATION selects the set of suppliers (and their resources) based on the decision maker' preferences. Using the AHP technique, a matrix of hypothetical decision makers' preferences is built and used to find the value of each supplier in the eye of the decision maker. The Significance of EDIS is that despite using numerical data, it does not require data gathering at the time of the disaster and uses historic data. EDIS can be complementary to existing methods for task allocation and scheduling techniques in disaster management, as a quicker data feed. This research also provides an insight into decision-making to reduce the uncertainty based on the principles of resource- dependency theory and through collaboration, as the most suitable group of suppliers are selected to share their resources based on the optimisation technique using the principles of the utility theory. Methodological contribution is a design

to simulate the decision-making under uncertainty by taking into account the opinion of human agents (decision maker). It also uses mathematical optimisation in addition to the opinion of human agents, which integrates the heuristic and mathematical approaches of decision-making. It also bridge the gap in needs prioritisation by providing numerical priorities. Practical contribution is that by providing a range of it enables the decision maker to decide based on their budget limitations and personal preferences. It also gives different humanitarian organisations the chance to customise the model using their own database if required.

The structure of the paper is organised as follows. We present a literature review followed by an elaboration of data sources. The method is then outlined where input and output are provided. The results section expands on input/output and provides details of the optimization through AHP. The process of collecting the preferences of the decision makers through a questionnaire is outlined and then the ranking of the suppliers through MAUT is provided. The discussion outlines the answers to the research questions, elaborates the contribution and then presents the limitations and the future research directions.

#### **2. Literature Review**

The present research addresses the partner proliferation problem in disaster response networks as one of the most recurring problems in humanitarian operations. The existing experiences of failure in disaster management operations in large-scale disasters, signals the necessity to investigate an effective disaster relief management, which is successful in minimising the negative effects of the disasters [7] specifically with the focus on reducing the problem of partner proliferation. Due to counterproductive effects of this phenomenon on the whole disaster relief operation, the quality of response is damaged [8]. The proliferation of actors is induced due to the extreme requirements of the disaster which forces to mobilise and recover all the available sources [9] and therefore all available partners are encouraged to participate.

The negative effect of this reactionary response [10] is twofold. First, the mandatory growth in the relief budget in the public sector (UN, Red Cross and governments) as well as the fund raising by the private sector (such as NGOs) exceeds the absorption capacity of an overstretched humanitarian industry. This pushes the inexperienced actors including the public image seeking companies into activities outside their area of expertise [11,12]. This situation leads to the oversupply of uncoordinated and inexperienced partners [13]. This rush of all available partners creates a range of partners from competent and incompetent, reputable and disreputable, opportunistic and committed, well-established and just-formed in addition to individuals, tourists and also companies which aim to generate a favourable public image, to increase their long-term profit. They enter the disaster-effected area in a chaotic pattern and cause the proliferation problem Figure 1. This as mentioned before, results in the budget stretch leading to the oversupply of a range of heterogeneous uncoordinated and inexperienced partners [13].

Figure 1 shows the chaotic pattern of partners' rushing into the affected area of Hurricane Katrina. This increases the load on the affected populations, local authorities, and coordination structures for information or services. It also increases the costs due to replicated offices and overheads, produces a counterproductive duplication and confusion of efforts, and leads to competition between agencies for donations, facilities, and publicity.

The second negative effect of proliferation is the increase in the risks of inappropriate aid, due to the time pressure of competition and the rush for publicity. This increases the risk to the quality of the response and reputation of the humanitarian community through the actions of inexperienced or irresponsible agencies and damages the quality of the responses [8]. The damage is enhanced by the fact that this wasted effort could be used instead to take advantage of the capabilities of the partners within the network and creates competition between the agencies over funding [11,14]. The study suggests that one of the reasons for failure in disaster relief network lies in the incompatibility of the disaster relief situation with the existing collaborative structures used for managing the response operation.

**Figure 1.** Partner's entry pattern into the affected area of Hurricane Katrina. Source: Comfort (2007).

The uncertainty and the lack of information [15] together with damaged infrastructure [16] unequal and ineffective distribution of demand and supply and their respective fluctuations [9,17], unsteady flow of the financial resources obtained by fund-raising from occasional donors [18] all make the planning and long-term outlook almost impossible. Additionally, long-term approaches in practice are usually profit-based whilst in disaster situations the non-financial factors such as the time value of commodities are much greater than the costs associated [18,19], which make the conventional profit-based values less accurate. Therefore, due to the lack of control and information in disaster situations, the existing structures such as supply chains or project-based collaborations might fall short in practice because these structures require a certain amount of knowledge about the supply, demand, timing, costs, etc. which are generally unknown in disaster situations.

This research proposes restructuring the relief network to accommodate the characteristics of the disaster situation to work with the minimum data available and without much pre-planning. The negative impacts of proliferation can be reduced if the partners are carefully selected according to the requirements of each particular disaster to make sure the interaction between heterogeneous partners does not have a counterproductive effect. An efficient operation needs to be supported with a suitable selection of partners who work together efficiently and guarantee the success of collaboration. The current study builds upon empirical research carried out in the field of decision making in disaster response operations as a response to calls [20] stating that an optimal network structure to assist in resolution of disasters is yet to be developed.

Dealing with the proliferation problem in a disaster situation falls under the heading of Decision Support Systems (DSS) in disaster situation [10,21]. The current literature mostly utilises DSS borrowed from logistics or production management studies into the disaster decision. The resource allocations generally include criteria-based optimisations. This criteria could include tangible characteristics of the resources including their location [22,23], Facilities [24], price [25], time [26–29], due date [30–33]. These are based on a fully informed decision environment and are time consuming to calculate or use complicated software and database which might not be available at the time of the disaster.

For example, task and resource allocation based on request from the aid centres assuming the data are available and reliable with no mention of the gap between the disaster strike and the data release [34]. They also are mostly based on the established distribution centres and fail to consider the ad hoc centres. The similar research considers community-based DSSs which tests all variations of the aid team to find the best [35] this trial and error is time consuming and there is no guarantee that the teams keep their members, performance and dynamic.

Others investigate distribution of resources [36,37], scheduling of supply chain for the delivery of resources [38], using genetic algorithms [39] integer linear programming in [40] to minimize the transportation cost, reinforcement learning [41] or MCDM to enhance the operational effectiveness of humanitarian activities [42]. The relief urgency index [43] using time-varying demand, population density, vulnerable population, damage, and last delivery to improve the relief distribution, fails to include the weight and the scalability of above factors. Other tools include stochastic optimization techniques knowing the exact number of national resources [44], using Nash equilibrium [45,46].

Other DSS rely on characteristics of the suppliers instead of resources. This could include measurable characteristics of the suppliers such as their attributes [26,47–49], partners' goal achievement probabilities [50] and performance indicators [51]. We argue that these criteria, although useful for planning and mitigation phases of disaster, are unsuitable for a disaster response due to the scarcity of data and time pressure associated with the disaster situation.

Additionally, regardless of the characteristics of the resources and/or suppliers, the decisions made by humans during disaster response, are highly affected by their preference. This has been addressed in few papers including risk preferences of decision makers [52], deep learning in resource prioritisation [53], mathematical models for resource optimisation based on community values in mitigation phase of disaster in few African countries [54], policy based resource optimisation for response [55]. The above criteria are often combined with mathematical optimisation techniques including AHP [56–59], Multi-attribute decision making under uncertain conditions [60–65], linear programming [10,66–68] and rule-based techniques [22], case-based reasoning [69] and spatial modelling of the resources [70]. It is noteworthy to mention this is a review of static models and dynamic models such as relief delivery models and route optimisations [71] or workflow modeling are not the focus of this research.

Based on the argument above, the research focuses on the partner selection in disaster situations as a solution to the partner proliferation problem. However, although a huge body of literature exist on the "how to restructure the selected partners", these approaches face a serious problem of duplication of efforts and the counterproductive effect of the operations during the disaster response operation. The existing research on this area mainly focuses on preparation, mitigation, and recovery phases by suggesting various long-term collaborative structures such as supply chains [72–75]. The problem arises from the high state of uncertainty in the response phase due to the temporary and urgent nature of the aid required, and the chaotic nature of disaster strike. This uncertainty affects the available data required for planning [15], the stream of financial resources [18] and unknown and fluctuating, supply and demand [9,15,17]. Due to the scarcity of the date before the release of MIRA report, this article develop a decision making framework (EDIS) for selecting suitable partners by reviewing the records of natural onset disasters, which have happened worldwide since 1980, and their data are available in various humanitarian databases (Emdat.be, 2014; Munichre.com, 2014;ReliefWeb, 2014; Gdacs.org, 2014). EDIS ultimately deals with the proliferation problem by ranking and selecting the most suitable partners based on the principals of the Decision theory and Resource-based theory.

The significance of this research is that in addition to dealing with the primary problem of the research (proliferation problem), it provides a framework for estimation of the needs, and resource optimisation through the allocation of the resources to the needs during the disaster response operation. This framework is noteworthy because currently the first official report of the disaster effects is released 72 h after the disaster strikes leading to a three-day gap between the decisions about the distribution of aid, and obtaining information about the actual needs amongst the affected population. The EDIS framework in this sense is an attempt to cover this gap by using the data available at the time of the disaster striking. This characteristic is also helpful because when a disaster strikes in many

areas the people who decide about the allocation of the resources, are not trained in the field of decision making or logistics. Instead, they happen to be in the disaster-affected area before experts arrive, and this framework could help them to make decisions using historic data and without the use of any complicated software, only excel sheets.

#### **3. Data**

Various scholars and humanitarian organisations categorise the criteria or requirements in the response operations. The preliminary review identified myriads of criteria [2,20,76–79]. The list of these criteria of requirements is presented in Figure 2.

**Figure 2.** The requirements of the disaster affected area.

Figure 2 shows where these requirements overlap. The majority of the organisations emphasise on the importance of the key life-saving activities including food security and nutrition, shelter and settlement (including non-food items), water and sanitation and health actions. Additionally, except for one organisation [76] which focuses solely on saving the lives of the survivors, the rest of the sources agree that rescue, evacuation and fatality management, education and logistics are also important. Some criteria emphasise on the importance of secondary hazard control [20,79] whilst some criteria are only mentioned by one source only such as psychological support [78,80], warning and security [2], livelihood [81], emergency infrastructure and recovery of lifeline services and activating emergency operation [20], mass prophylaxis, emergency triage, critical resource logistics, emergency information and warning, incident management, emergency operation management, volunteer and donation management, responder safety and health, emergency public safety and security, isolation and quarantine, secondary hazard control, medical surge and medical supply [79]. To summarise, the key life-saving activities or mass care activities are shared by all above organisations and therefore are the focus of this study.

The minimum standards of needs for key life-saving activities is drawn from the previous practice in similar disasters, published by humanitarian organisations [2,20,76–79]. The significance of this method is that by knowing the number of affected population, and based on the minimum standard, the required units of aid for each disaster scenario can be calculated. This process in this article is called "need estimation".

The data required for the estimation phase is collected from standard minimum requirements published in the following. This includes the internal reports and working papers from variety of government archives including Census Bureau, Department of Laboure, military, European Central Bank [82], Federal Emergency Management Agency [79], various bodies of UN [1,2,83,84], World Health Organisation [85,86], Global Health Council [80,87,88], Office for the Coordination of Humanitarian Affairs [81,89–95] and various foundations and associations including OXFAM [96], The Association for Healthcare Resource & Materials Management, Health Industry Group Purchasing Association, Health Industry Distribution Association [97], Sphere project [76], National Voluntary Organisations Active in Disaster [98] in addition to other reports [99–101]. Table 1 shows the literature used in developing the need assessment technique.



The resources in Table 1 were used to consolidate a need-based list of life-saving activities. This list was then categorized based on the humanitarian cluster system offered by Inter-Agency Standing Committee [77] to address the right of the affected population to receive the assistance required to live with dignity. This minimum standard requirement will be used in the first step of the methodology as described below.

#### *The Effect of Type of Disaster on Need Estimation*

The demand also may vary based on the type of the disaster because the type of disaster influences the extent of the effects. For example, earthquake causes the highest rate of death within different type of disasters. Additionally, some linguistic measures [76] shows that in an earthquake or high wind, food scarcity is not an issue, whilst it is quite probable in tsunami. For example, based on these data, it is unlikely that the affected population suffers from the food scarcity in the aftermath of earthquakes or winds, whilst it Is quite probable in after a tsunami. By adding to the effects of the different types of disasters, Table 2 is created. The following ranks are applied to the situation If Small = 1, Rare = 2, Few = 3, Moderate = 4, Many = 5, Common = 6, High = 7. It is noteworthy that the ranks need to be considered as priorities and not the actual numbers. Therefore, we started the priorities from 1 for simplification. It is possible to start it from any other number such as 0.57, 0.58, 0.59, or even start from 1000; 1100; 1200 as long as it makes it possible to show higher priorities. The result of this accumulation is summarised in Table 2 as ranked from 1–7, (1) being the lowest weight to (7) to the highest weight effect. The numbers are only representatives of weights and is not to be treated as actual numbers.


**Table 2.** Weights of the effects in various types of disasters.

\* The word "varies" is transferred from its original [76] and implies that the different records and scholars never agreed on a number on the specific disasters.

Table 2 shows that when earthquakes strike, fatality management, and medical mass care require the highest level of resources followed by food and shelter. Another conclusion is that after floods, the most required resources are shelter whilst after a flash flood and tsunami the highest priority is food cluster. Because the data set was void of information about the eruptions, the definition from [102] was used for this disaster type. It suggests that in eruptions the population displacement is often a consequence. Therefore, in general the eruption response prioritises are temporary shelter materials; safe water and basic sanitation; food supplies; and the short-term provision of basic health services and supplies. Using this data, decision makers could know roughly that when an earthquake strikes fatality management needs more participants than food supplying Suppliers. However, this rule does not indicate prioritising the population, and in applying this rule, it should always be taken into consideration that the live population has a higher priority. As a result, the mass care needs of the live population should be dealt with first before fatality management is put into place. This data is further used in combination to minimum standard requirement to estimate the needs as described as follows in the method.

#### **4. Method**

Decision-making methods suitable for a disaster network allocation, can be viewed as a multi-criteria decision-making problem [103]. Some scholars emphasize on the importance of DSS techniques in addressing specific disaster response problems [22,70]. The DSS designed for this research for allocation of the resources to the affected population is called EDIS (Estimating for DISaster response) Framework. EDIS follows two consecutive phases combining the existing decision techniques and determinants, suitable for the characteristics of the disaster response. The first step is "estimation" of the minimum resources required for the affected population and the second step is "optimisation" of these requirements by the decision makers as illustrated in Figure 3.

**Figure 3.** Inputs and outputs of the paper.

The principles of the resource-based view outlines that if the collaboration is to be successful it needs to focus on the resources, also based on the principles of the resourcedependency theory the companies collaborate in order to acquire critical resources and reduce uncertainty [104] which is the case in the disaster situation. The idea is to use the historic data about how many units of resources are required in similar situations in order to estimate the approximate needs. The rest would be an optimisation problem using the mathematical programming based on the principles of the utility theory. This insight outlines the design of the research to investigate two propositions.

This research addresses two questions: First to what extent is possible to estimate the resources required for humanitarian response operation in the absence of MIRA report within the first three days. Second to what extent is possible to optimise the resource allocation decisions within humanitarian response operation considering the preferences of the decision makers. This question is answered by a scenario-based decision making process to optimise the balance of available resources in possession of suppliers (supply) to the needs of the affected area (demand) as illustrated in Figure 3. The focus on this research are natural onset disasters or disasters with no-notice [22] such as eruption or earthquake.

This system categorises the minimum standard needs of affected population as Shelter, Nutrition, WASH (water and sanitation), and Health in 43 main needs. For simplicity and illustration purpose an example of the result of this accumulation is articulated in Table 3.

Table 3 accumulates data from different resources leads to an average number. For example, IFRC (Cited by WHO, pp. 48–49) states: "200 people/day 10–20 beds for overnight observation, Supplies to treat 30,000 people. For a month, per 12–14 h shift: 1 Doctor, 1 Pharmacist/Nurse, 1Curative/Community Health Nurse, 1 Midwife/Nurse, 2 General Technicians" from this statement, we can conclude that for 200 people/day we require a maximum of 20 beds, 1 doctor, 3 nurses and 2 other medical personnel and 1000 units of treatment supplies (30,000 for 30 days).

There are two points to keep in mind when looking at the above numbers. Minimum requirements for each cluster are expressed based on the person or household needs. A household is defined as a group of people who eat from a common pot, and share a common stake, interpreting and improving their socio-economic status from one generation to the next [105]. There are many options available for food as long as it provides the 2100 kcal required for each person [76] and complies with the cultural norms of the affected society. Additionally, the demand also may vary based on the type of the disaster.


**Table 3.** The examples of minimum requirements for life saving activities.

#### *Optimisation*

For the optimisation of partner selection, a DSS is required that embeds the partner selection criteria for partner configuration. For the particular case of this research the decision methods used in literature were compared to identify the most suitable technique to be used in the research. A review shows a variety of hard methods (with quantitative and numerical values) and heuristic methods (with linguistic and quantitative values) in the decision-making field. As mentioned before the process of optimization in this research includes balancing the available suppliers' resources (supply) to the needs of the affected area (demand). Therefore, a multi-criteria [53,106–109] resource based [55] DSS which accommodates the characteristics of the disaster response is required [103]. These characteristics may include the time pressure [24], big database [110] and multiple perspective of decision makers [111].Variety of hard methods (with quantitative and numerical values) and heuristic methods (with linguistic and quantitative values) in the decision-making field can be used. The suitability of them is assessed in Table 4.

**Table 4.** Criteria suitable for disaster response DSS.


Table 4 indicates the strength and weaknesses of each method. There is one specific group of hard or mathematical methods capable of accommodating numbers and quantitative values (as opposed to fuzzy or qualitative values) such as goal programming and integer programming [112]. These methods for the purpose of this research seem to be unsuitable because they formulate the problem in objective terms and fail to accommodate subjective attributes, here subjective preferences of decision makers. In addition, due to the high load of computation, these methods are not suitable for this research, a big dataset.

Another set of methods, which are vastly used in this area are evolutionary algorithms [107], however they become very slow when the number of selections arises and they might offer only a local optimal solution [67]. Additionally, the main drawback of all above methods is that they require a high level of specialised knowledge that is likely to be well beyond what possessed by disaster response decision makers. Alternatively, neural network analysis is suitable in disaster response networks for large data sets for training [53,113], however the quality of estimation in disaster situation under certainty is not trustworthy for training. Expert systems such as fuzzy logic are suitable for linguistically expressed expert's experience for multi-criteria optimisation [114,115]. Because this method is based on drawing fuzzy based rules out of the series of data, and in the absence of data, the rules cannot be confidently drawn. Both fuzzy methods and neural networks are only as strong as their database, so in the absence of such a strong database the rule-based system may fail [116]. This is noteworthy to mention that there is no record of decision makers' choices of suppliers in the disaster response in the literature despite a good record of disaster impacts in the literature.

Another group of methods, such as Multi-Attributive Decision-Making (MADM) as part of Multi-Attributive Utility Theory (MAUT) used for disaster response [65,109] also seems more suitable for optimization in this research. The reason is their capability of accommodating the non- certain preferences of decision makers, and linguistic expert's opinion which are required for supplier selection. These may include Analytical Hierarchy Process (AHP), Analytical Network Process (ANP) and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS). MADM is a branch in the decision-making for choosing between a finite number of alternatives.

In the EDIS, we assume that the number of suppliers in disaster response is finite so it seems appropriate to use MADM. One of the weaknesses of MADM methods is the rank reversal problem [117], which means that result of the ranking (direction of maximising or minimising and the ranking method itself) differs with the quality of the information available and the set of criteria representing the reality. However, in the uncertain environment of the disaster response, the decision maker always has to settle for available or obtainable data. This is because of the time pressure [24] and the often destroyed infrastructure which makes it impossible to improve the quality of the data. Therefore, the low quality of the data is going to affect the result of their decision, no matter what decision-making method they choose. Thus, these methods still seem like good candidates. Within popular MADM methods ANP which is used for prioritization [118] is incapable of accommodating the subjective perspectives of decision makers [103], which is one of the elements of the optimisation model in EDIS.

Another option, Technique for Order Performance by Similarity to Ideal Solution (TOPSIS), is used for group decision-making under uncertainty of information in order to select suitable suppliers [58,119]. This method can rank alternatives regarding defined criteria by minimising their distance from a positive ideal solution and maximising their distance from the negative ideal solution. However, this method also is based on objective values and therefore it ignores the subjective decision maker preference required in our research.

The most suitable option within MADM is AHP, which is used extensively in supplier selection [59,120–122]. This method is a good method for our research because unlike the other MADM methods mentioned above, it accommodates the subjective values, including the decision maker's preferences. To summarise Table 4, due to time pressure inherent in disaster situation, a DSS methods with a high execution time such as evolutionary algorithms, which slows down towards the end, need to be avoided. They also require a high degree of technical mathematic understanding, which the average decision maker in disaster response network might not have.

Another characteristic of any DSS run by people is that their preferences may hugely influence the result. Thus, the methods in which the subjective preferences of decision maker are not accommodated should be excluded such as ANP or TOPSIS. For those reasons the final candidate for optimization here are AHP to rank the utility of disaster response suppliers based on the decision maker's preferences to allocate the resources in demand by affected population to the resources offered by supplier. This selection is based on the resources the supplier has in accordance to the estimated needs for the respected disaster impact.

#### **5. Results**

The present research suggests two steps; the first step estimates the needs required for a particular disaster, based on minimum standard requirement for disaster. The second step is to optimize the resource allocation using the principles of utility theory by ranking the decision maker's preferences and the disaster needs priorities into an AHP model. In summary the article provides an optimization technique between demand and supply of the disaster affected area. The results are classified based on the input and output demonstrated in Figure 2.

#### *5.1. Input*

The input is the demand estimated based on the affected population, minimum standard requirement, and the weight of the disaster type. The minimum requirements of life saving activities (Table 3) coupled with the weight of disaster type (Table 2) can provide an estimation for the required resources to address the humanitarian needs for that disaster type in that cluster. However, the socio-economic characteristics of the affected country can also influence the need estimation as discussed below.

#### 5.1.1. Inputting the Effect of Socio-Economic Characteristics on Need Estimation

In addition, the economic characteristics of the affected regions could influence the priority of needs. Typically, the events that result in the highest numbers of fatalities are located in regions with increased risk and vulnerable populations. This is often compounded by limited infrastructure and poor integration of the health system into disaster preparedness, response, and recovery [86]. For example, more foreign medical care is required for a disaster, which strikes in Sub-Saharan Africa, than a disaster in the Middle East, due to the capabilities of medical infrastructure. Therefore, different levels of attention are required for various clusters in different types of disasters.

For example, after an earthquake, the food cluster in Japan and Philippines require different levels of attention, due to their different level of infrastructures. To address this issue the indicators of socioeconomic development have been included in the model. These indicators including lack of coping capability and susceptibility were drawn from the medical capabilities, and sanitation/nourishments of each country are annually calculated by the United Nation and published in the world risk report [123,124].

These indicators include the 'coping capability' indicators, which were calculated, based on (amongst other criteria) the number of physicians and hospital beds/per 10,000 inhabitants by UN. This indicator has been added to the model as weights, to signal the health cluster capability of the country. Furthermore, a 'susceptibility' indicator based on (amongst other criteria) access to the water sanitation and nourishment calculated by UN is also added to the model as a weight to signal the food and WASH cluster. These weights signal the criticality of the situation on a specific cluster in a particular country. It also provides an opportunity for comparison between different disasters as in Table 5.


**Table 5.** Comparing two different disasters with their weights.

Table 5 shows that by comparing the 2005 disaster in Pakistan with the 2011 disaster in New Zealand, without knowing any other information, including the type of disaster, we can tentatively claim that the health cluster (in terms of hospital beds and physicians) in Pakistan is almost two times less likely to cope with the disaster effects than New Zealand. The reason is that Pakistan's lack of coping capability is 87.39% compared to New Zealand's, which is 39.79%. The same principle can be used to interpret the susceptibility based on access to food and nourishment. It shows that Pakistan (38.84% susceptibility) is three times more likely to suffer from mal-nourishment, lack of water, and sanitation than New Zealand (with 16.19% susceptibility).

These numbers should also be considered as probabilities or risk factors and not actual numbers. They are only to be used for signalling what areas of needs should be prioritised. Combining the criteria affecting the needs in a disaster situation (including evidence from previous experiences, the type of disaster, and economic aspects of the affected region), the priority for each task can be calculated. Assume we must choose between disaster response clusters in both Pakistan and New Zealand at the same time. Based on the data in Table 3 the priorities would be shelter and fatality management in both counties because their priorities are higher than other clusters and equal to 200. The next priority is food and WASH for Pakistan (both 194 points for priority), followed by the Health cluster for Pakistan (87.39 points for priority), then food and WASH for New Zealand (80.95 points for priority), followed by the Health cluster for New Zealand (30.79). This data is obtainable and calculated without knowing any other information about the disaster including its type.

#### 5.1.2. Estimating the Required Resources: An Example

The affected number for Pakistan earthquake 2005 is used for an example. The total of 75,000 injured and 2,800,000 displaced population are the basis for this calculation. There are few assumptions associated with this example. First, assuming there is an overlap between the injured and displaced population, and for that reason we then assumed that the injured only use the health cluster and water for patient needs and the rest are being used by the displaced. There are four prominent categories of needs, one for each humanitarian cluster including health, nutrition, WASH and shelter. Multiplying the needs for one person in Table 1 and estimated number of people in need of that particular help, would provide the total number of needs required for that cluster. So the need for each cluster is calculated as: [Total unit required for a cluster = Minimum standard requirement \* estimated impact]. A sample of 21 needs for the illustrative purposes are distributed between four humanitarian clusters is presented in Table 6 combining the result of Tables 2 and 5.


**Table 6.** Needs estimation for Pakistan earthquake 2005.

Table 6 is calculated based on the minimum standard requirement in Table 2. For example, in the health cluster the need for a doctor in Pakistan earthquake 2005, is 75,000 doctors or 16,605 L water/day. Additionally, the cluster priority column shows that the community is less likely to cope with shelter shortage than the other needs, so in allocating the resources, the shelter (200 cluster priority) needs to be prioritised slightly over nutrition and water (194 cluster priority) and then health needs (87 cluster priority). This is also confirmed by the number of displaced who would require shelter, water and food (2,800,000 people) as opposed to the number injured (75,000 people).

#### *5.2. Output: The Optimized Set of Resources Available from Different Suppliers*

By entering the preferences of the decision makers, their subjective views which can affect the decisions are taken into account. The supply is the optimised in terms of the ranks of suppliers who have resources available for the required response phase.

#### 5.2.1. Building AHP Model Based on Decision Makers' Preferences

Due to the subjective nature of decisions, different decision makers, provided with the same options and data, make different decisions, based on their preferences. In disaster situation when we have different suppliers, choosing between different suppliers and their resources is important to optimise the allocation of resources. A set of questionnaires are conducted from experts in disaster management field. The data collection process is described below.

#### Collecting Decision Maker's Preferences

This questionnaire was provided to the participants which overall took three weeks to complete for 42 participants. The information about the research and invitation for participation was distributed amongst various organisations (Environment agency, Crisis departments of five different embassies, Business continuity departments of Munich RE, Barclays Bank and Lloyds bank, and individuals who had connections with humanitarian organisations including UN, UNISDR, UNICEF, World Vision, Caritas International, British Red Cross, American Red Cross, Save the children and various specialised forums and groups related to disaster management on LinkedIn (including Business Continuity and Disaster Recovery Professionals, Business Continuity Management & Risk, Business Continuity/Disaster Recovery Network, Disaster & Emergency Management, Disaster, Disaster, Disaster Management—Multi Hazard Risk Assessment, Disaster Researchers and Disaster Management Professionals, Disaster Risk Management Practitioners, Emergency Preparedness Consultants/Trainers Group, GWU Institute for Crisis, Disaster and Risk Management, Humanitarian & Disaster Response Technology Network, Innovations in Disaster Management and Emergency Response !,Natural disasters and natural hazards, Natural Hazards and Disaster Risk Management, Performance Management, Professionals in Emergency Management, World Conference on Disaster Management). 68 people initially expressed interest and were sent the questionnaire but at the end 42 filled questionnaires were returned.

The respondents are asked to identify in respect to each one, the criteria for partner selection which criterion is more important and how much more important on a scale of 1 to 9. This is the basis for questionnaire 1 (decision preference). These criteria include the type of partners (Government, NGO, Military, International organisations such as Red Cross and UN and volunteers), size, experience of the partners, their surge capacity (the ability to rapidly expand beyond normal capacity to meet the increased demand) and their cluster (WASH, nutrition, health, shelter). The first questionnaire is given to both groups of participants in order to identify their preferences. The goal, criteria, and sub-criteria considered in this questionnaire are articulated in Figure 4.

**Figure 4.** Components of the questionnaire about the selection decision.

The first row in Figure 4 shows that the goal of this questionnaire is to define the characteristics of the desirable partners in the view of each decision maker. The second row gathers the data about the characteristics of the desired partner in terms of the following criteria: *Type of the partner* in respect of being governmental, NGO, International, Military or Volunteer organisation as sub criteria. *Size of the partners* based on ANLAP's (2012) categories for humanitarian organisations, being Small (under 10 million USD expenditure), Medium (between 10–49 million USD expenditure), Big (between 50–99 million USD expenditure) and Very big (more than 100 million USD expenditure). *Experience of*

*the partners* being Low (Under 5 disasters), Medium (Under 10 disasters), High (under 50 disasters) and Expert (more than 50 disasters). *Partner's surge capacity* (the ability to rapidly expand beyond normal capacity to meet the increased demand) being None (0% of the total capacity), Low (under 10% of the total capacity), Medium (under 30% of the total capacity) and High (over 30% of the total capacity). *Partner's international expansion* being Yes (expanded internationally such as UN), No (expanded only locally such as local charities). *Partner's ability* to address the needs for humanitarian cluster being WASH, Nutrition, Health, and Shelter. So the numerical preferences for the above decision criteria being type, size, experience, surge capacity, expansion and cluster is collected through the questionnaire in Table 7.


**Table 7.** Pairwise Comparison Questionnaire to elicit decision-maker's preferences.


**Table 7.** *Cont.*

The data gathered in this questionnaire was then used to calculate the preference weights using AHP. The preferences of the decision maker can be quantified using AHP. This is calculated by a set of pairwise comparison matrices where the verbal preference (e.g., extremely less/more important) is translated into numerical values (e.g., 1/9 to 9). The AHP weight calculated for these values can get values from zero to 1.0 or from 0% to 100% as shown in Figure 5.


**Figure 5.** A snapshot of the process of calculating AHP preference.

Imagine we have a decision maker who prefers to be involved government organisation the most; in other words, if s/he wanted to decide based on the type of the organisation s/he would definitely choose the government over International organisations. A decision maker with the AHP values is calculated as the government had the highest value for this particular participant (60.8%) whilst the International organisations and volunteers had the lowest value (6.5%) as shown in Table 8.



For example, in Figure 5 the government had the highest value for this participant (60.8%) whilst the international organisations and volunteers had the lowest value (6.5%). In other words, if s/he wanted to decide based on the type of the organisation s/he would definitely choose the government over international organisations. This process gives a full set of preference for each unit of resource per partner, presented in Table 9.

**Table 9.** Example of AHP for participant/unit of resource per partner.


Table 9 shows that the preference for doctors (a resource in the health cluster) for this participant is AHP = 0.98 or 9.8%; whilst s/he considered water (a resource in the WASH cluster) much more important (AHP = 0.58 or 58%). In addition, the AHP weight of each resource for each partner was calculated. For example, the water provided by Partner 4 had a higher preference (74%) than the water provided by partner 2 (49%) due to the preference this participant had towards the characteristics of these partners (including type, size, expansion and so on).

#### 5.2.2. Calculating MAUT for Each Supplier

Based on the above priorities calculated by AHP, the MAUT produced for each supplier can be calculated as follows. *Ui*(*x*) is a single utility function or preference function associated with candidate i, which represents the utility values the decision maker attaches to each candidate and is obtained by using the AHP process. To aggregate the scores of each attribute in the MAUT process, the linear additive utility form is the frequently simplified assessment procedure as given by Equation (1)—Utility function of the candidates based on the available resources:

$$\begin{array}{l} m \\ V(y\dot{q}) = \sum \dot{r}\ddot{q}.\mu\dot{q}(x). \\ \dot{q} = 1 \end{array}$$

where *rij* represents the resource j available to candidate *i*. The *V*(*yi*) will be the value of the candidate *i* because of the resource *j* they have available. The AHP weights calculated before then were used to calculate the utility of each resource as well as the utility of that resource for that partner Table 10.

**Table 10.** An example of the utility for a participant.


For example, in Table 9 the utility of the water provided by Partner 1 is 15.30, whilst the utility of water for all the partners is 25.11. The total utility of all the resources that each partner holds can be calculated as the accumulated values of that partner's utilities. For example, for these particular participants, the utility of partners can be calculated and be used to rank the partners as exhibited in Table 11.



Table 11 shows an example of the rankings of the partners based on this participant's preferences. For example, Partner 5 is the most desirable partner with a utility of 1520. This also shows that the most desirable partners for these participants are small governmental entities. In addition, it seems that this participant does not value the experience or the surge capacity of the partners as critical requirements for a disaster response. Finally, the experts were asked to fill out the second questionnaire. An example of the accumulated data is exhibited in Table 12.


**Table 12.** An example of the result of the optimise resource-based decision-making.

Table 12 shows that for example, in this scenario the total available resources N42 = Doctors, are 0.0372 for each 100 people. However, the number of required doctors is more than 0.515 for 100 people. Although due to the scarcity of this resource, and the fact that the decision maker needs all the helps s/he could get, it is still possible to rank the Suppliers based on the decision maker's preference. As you see, the utility of the doctors that Supplier 2 can provide (0.133) is greater than the number doctors that Supplier 1 can provide (0.0112). In addition, as can be seen in this case the utility of the health cluster (0.103) is more than the other clusters. The utility of the shelter cluster is 0.099, whilst the utility of the nutrition is 0.017 and WASH is 0.0007. Therefore, if a decision maker must decide which need to prioritise, s/he should first consider choosing the Suppliers who can provide the doctors, nurses, etcetera, rather than the Suppliers who can provide, food, water, or shelter.

#### 5.2.3. Ranking Suppliers Based on Their MAUT

To get a better understanding about how the Suppliers in different scenarios for different decision makers may differ, an example is presented in Table 13.

**Table 13.** An example of the Suppliers ranked based on MAUT.


Table 13 shows the ranking of the Suppliers based on the highest utility to the lowest for this example. Based on the preferences of decision maker 2 and the needs predicted in scenario 1, Supplier 153 with a total utility of 1.13 is the best option followed by Supplier 41 with 1.09 utility, etc.

#### **6. Discussion**

The present research is designed to provide a technique for Supplier ranking/selection in disaster response. The research employs various techniques including analysing the archival data, and decision support tools including Linear programming optimisation, Analytical Hierarchy processing (AHP), Multi-attribute utility theory (MAUT) to develop several decision techniques based on secondary data. This research provides an approach to Supplier configuration in disaster situation in two phases. The ESTIMATION process answers the first research questions is "how to estimate the needs of the affected population at the time of the disaster strike?". Using various resources, the minimum standard requirements for a disaster response in four humanitarian clusters (WASH, Nutrition, Health and Shelter) was defined. This estimation was used as the basis for estimating the demand of the affected population in disasters. This exceeds the use of minimum standard requirements provided by the Sphere project because it draws upon various sources to provide the data about the required units of medical help and nutrition.

This framework could also further be developed to provide data about fatality management, evacuation, and required well contamination teams. This also complements the existing literature on provide the priority of the disaster type, and tasks during each disaster type. Even though some linguistic priorities are practiced in the literature [76], the numerical priorities that can contribute to the quantification of the needs were missing. The priorities suggested in this research are required to be investigated further with the fuzzy logic analysis of the experts' opinions regarding the priorities of each, task/need for each disaster type/country. However, this is another extensive research in its own merit and is out of scope of this research. The OPTIMISATION process answers the second question "how to optimise the selected set of suppliers (and their resources) based on the decision maker' preferences. This is a framework for disaster response supplier selection using the principles of utility theory. In this step, the Suppliers are ranked based on their importance for hypothetical decision makers. Using the AHP technique, a matrix of hypothetical decision makers' preferences is built and used to find the value of each Supplier in the eye of the decision maker. This step can be defined as a resource allocation problem with the target of optimising the utility of the response Suppliers' set for each decision maker. The optimisation here is like a variety of supplier selections based on MCDM [125,126]. The variable which needs to be maximised is the utility of the suppliers in the eye of the agent (here the decision maker).

The EDIS can be complementary to the abundance of existing methods for task allocation and scheduling techniques [71,127,128] in disaster management, as a quicker data feed. Furthermore, the research shows that comparing to the existing decision models in humanitarian sector the EDIS could prevails the existing guideline based on highly specialised data in HAZUS [129] or highly subjective decision maker's preferences in HISS [130] from The European Interagency Security Forum (IESF). In a sense, EDIS gives numerical estimations, and clearly expressed choices of suppliers whilst it is using simple available data. Contribution to theory is that it provides a unique insight into the growing body of research a part of decision-making under uncertainty where it is attempted to reduce the uncertainty by "gaining accumulated access" to other firms' resources meaning that every member has access to the resources of all the other members. This is based on the principles of resource- dependency theory and through collaboration. Because the collaboration act in practice is no guarantee of a successful disaster response due to the interaction of contributors, the most suitable group of suppliers to accumulate and share their resources are selected based on the optimisation technique using the principles of the utility theory.

Methodological contribution is that this model provides a design to simulate the decision-making under uncertainty in the disaster situation by considering the opinion of human agents (decision maker). It also uses mathematical optimisation in addition to the opinion of human agents, which integrates the heuristic and mathematical approaches of decision-making. This also complements the existing literature by drawing upon various studies to provide the priority of the disaster type, and tasks during each disaster type. Even though some linguistic priorities are practiced in the literature (Sphere project, 2011), the numerical priorities that can contribute to the quantification of the needs were missing. Practical contribution is that by providing a range of it enables the decision maker to decide based on their budget limitations and personal preferences. It also gives different humanitarian organisations the chance to customise the model using their own database if required. The practical contribution of the article is the needs estimation tool. This framework uses various resources to articulate the minimum standard requirements for disaster response in four humanitarian charter clusters (WASH, Nutrition, Health, and Shelter).

The humanitarian organisations could use this tool to estimate the resources required to response to the needs of the affected population before the MIRA report is released. The significance is threefold. First, it is the first decision framework of its type that enables the decision maker to estimate the needs and select the partners using the data that are readily available for each country at the time of the disaster. Reliance on the available data at the time of the disaster, which are freely available to the public would reduce the cost of the data gathering, and the time required for collecting and analysing these data. Consequently, it speeds up the response time of the operation to the disaster by almost 72 h, which is vital at the time of the disaster. In addition, it is the only existing framework not limited to a certain type of disaster (although it just considers the five types of disasters) or geographical or chronological order. Another contribution is that the model has the capability of accommodating the socioeconomic characteristics of the affected population, which hugely influences the required aid in humanitarian response practices. The authors also believes that this model in long-term could facilitate establishing a centralised database for humanitarian response which is long overdue.

#### **7. Limitations and Future Research Direction**

The first limitation of this work is the lack of secondary data regarding the specific requirements of non-key-life-saving activities which led to the exclusion of them from the study. However, the principles of this research can be extrapolated to non-key life-saving activities when the data is available. However, data collection on this scale requires the cooperation of various humanitarian organisations including the UN, IFRC, and government related organisations, in addition to the private and public humanitarian organisations and charities (like the process in the sphere project) and is out with the scope of the current research. management, evacuation, and required well contamination teams.

Second, the priorities suggested in this research are required to be investigated further with the fuzzy logic analysis of the experts' opinions regarding the priorities of each, task/need for each disaster type/country. However, this is out of scope of this research. Nevertheless, this research provides the preliminary basis for the further development of such framework.

The third limitation is that the EDIS model is based on two major assumptions. The first assumption is that a data base for humanitarian suppliers already exists. However, creating and maintaining such a database requires the cooperation of the international humanitarian bodies. The model cannot be fully tested before the creation of a standardised accredited database containing data on humanitarian suppliers, their selection criteria, and regular updates of the database. This project can be further discussed with international humanitarian entities with regard to the applicability of launching a universal initiative for gathering data and building a universal humanitarian database. The model is built upon

secondary data from various sources amongst others in which the data varies from case to case. Therefore, the model is only as accurate as its data feed.

The fourth limitation is that the optimisation constraints in this model are just the resources, the optimisation could be improved if other constraints such as time and cost could be considered. This could also be improved if the tasks can be separately defined in detail, and then the task allocation and resources related to the allocated task of each supplier could be optimised. Although the contribution of the current study is its model, further empirical research is required to develop an extensive database for the potential humanitarian suppliers at the industry level. The future research direction could follow different paths. For example, the EDIS model is based on the resources-based optimisation, it considers the decision makers' preference and characteristics in various other criteria such as experience, type, and size of the organisation, its surge capacity, and international expansion.

Further research is required to identify the actual non-resource based determinants of supplier selection in disaster response. Another suggestion is to provide a holistic research study involving all humanitarian actors to further identify and standardise the minimum requirements in a disaster response by considering the actual disaster type, and the geographical location and culture of each potential affected county. Another path could be the application of the EDIS model to various case studies and analyse the result and the areas of improvement. In addition, the EDIS model could be more accurately customised if it could analyse the data for each individual country, where it is possible to define exact scenarios for each disaster type, and the needs and suppliers required. This also may greatly improve the quality of estimations. The EDIS model is based on two major assumptions. The first assumption is that a pool of humanitarian partners already exists. However, creating and maintaining it requires the cooperation of the international humanitarian bodies.

The model cannot be fully tested before the creation of a standardised accredited database containing data on humanitarian partners, their selection criteria, and regular updates of the database. This project can be further discussed with international humanitarian entities with regard to the applicability of launching a universal initiative for gathering data and building a universal humanitarian database. Second, the model is built upon secondary data from various sources (Emdat.be, 2014, Munichre.com, 2014; ReliefWeb, 2013a; Gdacs.org, 2014), amongst others in which the data varies from case to case. Therefore, the model is only as accurate as its data feed. Although the contribution of the current study is its model, further empirical research is required to develop an extensive database for the potential humanitarian partners at the industry level.

The future research direction could follow different paths. For example, the EDIS model is based on the resources-based optimisation, it takes into account the decision makers' preference and characteristics in various other criteria such as experience, type, and size of the organisation, its surge capacity, and international expansion. Further research is required to identify the actual non-resource-based determinants of partner selection in collaborative networks with the focus on disaster response. Another suggestion is to provide a holistic research study involving all humanitarian actors in order to further identify and standardise the minimum requirements in a disaster response by considering the actual disaster type, and the geographical location and culture of each potential affected county. Another path could be the application of the EDIS model to various case studies and analyse the result and the areas of improvement.

**Author Contributions:** Conceptualisation: S.R.; methodology: S.R.; validation: S.R.; formal analysis: S.R.; investigation: S.R.; resources: S.R.; writing—original draft preparation: S.R.; Writing—review and editing: E.A. and S.R.; visualitaion: S.R.; supervision: E.A.; project administration: S.R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki, and approved by the Ethics Committee of Brunel University as part of a PhD dissertation approved in September 2014.

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy of the humanitarian participants.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **A Fuzzy Linguistic Multi-Criteria Decision-Making Approach to Assess Emergency Suppliers**

**Huilin Li \*, Jiaqi Yang and Ziquan Xiang**

School of Transportation and Logistics Engineering, Wuhan University of Technology, Wuhan 430063, China **\*** Correspondence: huilin.li@whut.edu.cn

**Abstract:** Under the influence of COVID-19, various emergency supplies have repeatedly broken links, seriously affecting normal life and hindering the sustainable development of enterprises and society. Therefore, suitable emergency suppliers are crucial. To prioritize and select suitable emergency suppliers, key indicators were determined, and evaluation models were established based on the characteristics of epidemic situations and epidemic prevention materials. The application of the MCDM (multi-criteria decision-making) issue combining fuzzy SWARA (the stepwise weight assessment ratio analysis) and the actor analysis method to emergency supplier selection studies, called the fuzzy SWARA-based actor analysis method, is used to identify appropriate suppliers for optimizing pre-preparation. Results of evaluation system weight computations by the Fuzzy SWARA-based actor analysis method show that the overall prioritization of four non-economic factors in ranking orders are "Delivery Capacity", "Flexible Supply Capacity", "Quality", and "Social Evaluation and Reputation". For the inclusion of conflicting standards and the unquantifiable feature, economic and non-economic factors were discussed separately and evaluated by language variables. Additionally, the fuzzy actor analysis indicated that economic factors and non-economic factors need to be considered comprehensively for emergency supplier selection. This method has good operability and reference value, convenient for the final choice making according to actual situation.

**Keywords:** emergency supplies; multi-criteria decision making; fuzzy set; actor analysis method; linguistic variables

#### **1. Introduction**

In recent years, the frequent occurrence of various natural disasters and emergencies has caused varying degrees of casualties and property loss. Especially in the past two years under COVID-19 outbreak, the supply chain phenomenon seriously affected the normal life of the masses, hindered the development speed of enterprises and society, and deepened scholars' thinking of the emergency supply chain of logistics management research.

Emergency management operations generally consist of four parts: prevention, preparation, response, and recovery. The process of emergency supply chain system is shown in Figure 1. The main work in the prevention stage is the establishment of relevant emergency mechanisms, laws, and regulations by the main government departments in society, to reduce hidden dangers and strengthen the ability to deal with emergency events. The preparation stage is to advance deployment and arrangement to resist possible emergencies and ensure the effectiveness of rescue after the occurrence of the event to the greatest extent, such as the advance purchase of epidemic prevention materials, the location of emergency supplies reserve centers, the deployment of emergency facilities, and other issues. The response stage is the key element of emergency management. Various rescue methods are needed to reduce the losses and casualties caused by emergencies and reduce the negative impact on society as a whole after the incident, such as the distribution of emergency relief supplies, transportation, and scheduling of emergency relief supplies. The recovery stage involves the reconstruction of disaster areas and the recovery of people's

**Citation:** Li, H.; Yang, J.; Xiang, Z. A Fuzzy Linguistic Multi-Criteria Decision-Making Approach to Assess Emergency Suppliers. *Sustainability* **2022**, *14*, 13114. https://doi.org/ 10.3390/su142013114

Academic Editor: Krzysztof Goniewicz

Received: 7 September 2022 Accepted: 10 October 2022 Published: 13 October 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

lives after the response stage. Strictly, the prevention stage does not belong to the category of logistics management. In the field of emergency logistics management, the first step should be the preservation of emergency materials, that is, the preparation stage. In the preparation stage, to ensure the best rescue effect after an emergency, reliable suppliers should be selected from numerous emergency suppliers and a good supply system should be established. In disaster relief practices, a good cooperative relationship between relief agencies and suppliers can simplify the procurement process and improve the availability and rapid delivery of supplies. In addition, establishing a close relationship with suppliers can achieve discounts in bulk pricing. Regardless of the scale and importance of procurement in emergency logistics, only a few studies focus on the issue of emergency supply procurement decisions [1].

**Figure 1.** Emergency supply chain system flow chart.

Suppliers are the critical link to any supply chain as an important strategic decision, and supplier selection helps achieve a solid relationship between supply and demand [2,3]. Consequently, the selection of emergency suppliers is an important part of the emergency supply chain management, which is a typical problem. There is a large body of literature on supplier selection decision-making in the commercial supply chain, such as supplier selection criteria [4]. However, not much attention has been paid to these factors in emergency logistics management, because disaster management is more closely related to the relationship between economic and non-economic factors. Previous standards in the commercial supply chain can provide guidance for this study, and the emergency supply chain also uses some of the same indicators, including price, quality, delivery capacity, etc. The contributions of this paper are as follows:

Firstly, the evaluation index system of emergency suppliers for large emergencies was established, and 20 evaluation indicators for emergency suppliers were listed in a relatively comprehensive way, which has targeted and comprehensive coverage, and further improves the evaluation index system of emergency suppliers.

Secondly, different from other fuzzy multi-criteria decision-making methods, this paper focused on the decision preferences of economic factors and non-economic factors and adopted the decision weights of experts to evaluate and select emergency suppliers by the fuzzy SWARA-based actor analysis method. Through the corresponding relationship between triangular fuzzy and decision language variables, the scores of qualitative indicators of different experts were converted into objective values, and the weights of non-economic factors were obtained.

Finally, sensitivity analysis was used to verify the influence of economic and noneconomic factors on the preference decision of emergency supplier selection, and the priority ranking under different decision preferences was obtained.

In this study, the scientific selection of emergency suppliers is emphasized. The evaluation indicators and evaluation method are two key research points in the evaluation and selection of emergency suppliers. Based on the characteristics of an emergency, by initiating the application of the MCDM issue combining fuzzy stepwise weight assessment ratio analysis (SWARA) and the actor analysis method to emergency supplier selection studies, this study bears significance for it illustratively identifies the evaluation system that is critical to prioritization and selection of alternative suppliers. In the process of supplier selection and evaluation, qualitative indicators need to be quantified because many qualitative indicators are involved. Therefore, linguistic variables are introduced in this method, and the corresponding relationship between linguistic variables and fuzzy sets is established to transform the evaluation of qualitative indicators by experts. Linguistic variables were used to determine standard ratings expressed as fuzzy numbers. The evaluation indicator weights of emergency suppliers were determined using the SWARA method. The ranking of each alternative supplier was determined by the actor analysis method on fuzzy sets, which considers non-economic factors.

The remainder of this paper is organized as follows. Section 2 presents the literature review and outlines the innovative points and contributions of this study. Section 3 puts forward the key evaluation indicators for emergency suppliers. Section 4 describes the methods and processes. Section 5 applies the method to numerical examples of emergency supplier prioritization. Section 6 presents the sensitivity analysis. Section 7 discusses managerial implications. Finally, Section 8 concludes the study and offers future research directions.

#### **2. Literature Review**

We focus our attention on the literature on supplier evaluation decision-making methods and evaluation indicators for the criteria.

In the field of emergency logistics management, the first step should be to reserve emergency materials. To ensure the best rescue effect after an emergency, reliable suppliers must be selected. In general, a reliable supplier should follow the principles of right price, right time, right place, right quality, and right quantity. At present, research on the evaluation or selection of commercial suppliers has produced rich results. Several evaluation indicators were proposed, including quantity discounts, transportation costs, carbon taxes, price discounts, delivery times, service levels, supplier capabilities, and delivery times [5–9]. Wang and Su [10] proposed a generic DSS framework based on activity-based costing to evaluate and select suppliers. According to the characteristics of logistics service outsourcing enterprises, Peng [11] established a logistics service outsourcing supplier evaluation and selection index system as measured by cost, operational efficiency, basic quality, and technical level, aiming for the evaluation and selection of logistics outsourcing service suppliers based on the hierarchical analysis method. However, for different industries, the selection basis of suppliers is different; in particular, the selection of emergency material suppliers is more special, and must be considered in terms of the material quality guarantee and timely supply capacity as the main factor. Hu and Dong [12] considered humanitarian assistance extremely important in supplier selection and incorporated it into the selection strategy. The supplier selection criteria include price discounts offered by suppliers based on order quantity, required delivery time, and physical inventory. Ruan et al. [13] built a balanced "helicopter and vehicle" intermodal network by selecting emergency distribution centers (EDCs) and allocating medical assistance points, considering helicopter travel time, transfer time, and vehicle delivery time.

Both quantifiable economic and qualitative non-economic factors are involved in supplier selection decisions; the conflict between the indicators is the existence, which is a typical multi-criteria decision-making problem. The multi-criteria decision-making (MCDM) approach, based on the evaluation of multiple conflict guidelines, provides an effective framework for supplier comparison. Evaluation methods, such as AHP, ANP, TOPSIS, DEA, TCO, and GRA, are widely used in the supplier selection problem [14]. TOPSIS is fully called Technique for Order Preference by Similarity to an Ideal Solution. The basic principle is to rank the distance between the evaluation object and the optimal solution and the worst solution [15–17]. In Boran's [18] study, the TOPSIS method combined with an intuitive fuzzy set is proposed, and it was used in a group decision-making environment to select appropriate suppliers. Based on a set of standards applicable to the Industry 4.0 environment, Kaur and Singh [19] used the fuzzy analytic hierarchy process and the ideal scheme similarity ranking technique (FAHP-TOPSIS) method to evaluate suppliers. Çalık [20] developed a new group decision-making approach based on Industry 4.0 components for selecting the best green supplier by integrating AHP and TOPSIS methods under the Pythagorean fuzzy environment. Chen [21] proposed a novel decision-making model of TOPSIS integrated entropy-AHP weights to select the appropriate supplier. Zhang et al. [22] solved the uncertain attribute values and weights in MCDM problems by combining the ER approach and stochastic multi-criteria acceptability analysis-2 (SMAA-2). Bai et al. [23] used the gray-BWM-TODIM method to evaluate and select socially sustainable suppliers. Social sustainability attribute weights were determined using the gray-BWM approach, and then the gray-TODIM method was used to rank suppliers. Nekooie et al. [24] proposed a fuzzy objective planning method with soft priority between the objectives. Wang and Cai [25] built a distance-based VIKOR multi-criteria group decision-making (MCGDM) model for processing heterogeneous information to appropriately and flexibly solve the problem of emergency supplier selection with a compromise solution, which is more acceptable and suitable in practice. Badi [26] used a hybrid grey theory-MARCOS method for decision-making regarding the selection of suppliers in the Libyan Iron and Steel Company (LISCO) to help it compete. Tavana [27] proposed an integrated approach for supplier selection by combining the fuzzy AHP method with the fuzzy multiplicative multi-objective optimization based on ratio analysis. Giannakis [28] developed a sustainability performance measurement framework for supplier evaluation and selection by the Analytic Network Process (ANP) method. Chou and Chang [29] used linguistic values to evaluate the ratings and weights of selection factors and proposed a strategy alignment fuzzy simple multiple attribute rating (SMART) technique to solve the supplier selection problem. Weng [30] presented the analytic hierarchy process (AHP) and grey relational analysis (GRA) as potential multi-criteria decision-making (MCDM) methods for spare parts planning (SPP) software selection. Bakeshlou et al. [31] established a multi-objective fuzzy linear planning model with 17 criteria and divided it into five clusters, solved by a mixed fuzzy multi-objective decision model (MODM). Fallahpour et al. [32] improved the existing DEA-AI model, introduced a new artificial intelligence method for supplier selection, and integrated the Kourosh and Arash methods into a robust DEA model obtained by genetic programming (GP).

This is a sophisticated problem because supplier selection is often a multi-standard group decision-making problem involving conflicting standards. Fuzzy set theory has been widely used in management decision making. The fuzzy set theory proposed by Zadeh [33] provides an effective method for addressing fuzzy problems. The judgment of decision makers is represented by fuzzy numbers, thereby quantifying the evaluation level. Muneeb [34] proposed a decentralized bi-level VSP where demand and supply are normal random variables and objectives are fuzzy in nature. Many others have solved evaluation and selection problems using fuzzy set theory [35–38]. Based on this, aiming at the fuzzy concepts that often appear in decision-making problems, a new multi-criteria decision-making method is proposed to solve the supplier selection problem.

In summary, most previous studies have focused on the evaluation or selection of suppliers, and the fields of application are mostly in commercial supply areas, using classic evaluation methods. A comparison of supplier selection methods is shown in Table 1. Additionally, many scholars have made innovations from the perspective of fuzzy theory, and a variety of fuzzy multi-criteria decision-making methods have been formed. Meanwhile, the emergency supplier selection decision issues as a multi-standard group decision-making problem involving conflicting standards and unquantifiable features. There are many non-economic factors to be considered, economic and non-economic factors should be discussed separately. In order to fully demonstrate the importance of non-economic factors and their mutual comparison, the fuzzy SWARA-based actor analysis method is used to

evaluate emergency suppliers. Meanwhile, the lack of research on emergency supply evaluation fields thus makes it necessary to conduct an emergency suppliers' criteria system and method, and fuzzy set theory is suitable for this issue. Therefore, the actor analysis method combined with the fuzzy SWARA method is proposed to solve the multiple-criteria decision-making (MCDM) problem, which evaluates unquantifiable indicators using language variables.


**Table 1.** Comparison of supplier selection methods.

#### **3. Evaluation Indicators Analysis**

Compared with ordinary materials, epidemic prevention materials are highly irreplaceable, with more uncertainties and high timeliness in the delivery process, that need more reliable channels [39–42]. If the quality of supplies is not guaranteed, insufficient quantity, or a low qualified rate, it may cause problems in the subsequent rescue response stage. As the supplier of emergency supplies, it should have a better supply capacity and a higher response capacity in both delivery time and quantity. In addition, a high response level in the supply chain ensures the effectiveness and supply of emergency supplies. The emergency suppliers' evaluation and selection criteria system established in this study is shown in Figure 2, which includes five main indicators: flexible supply capacity, delivery capacity, price, quality, social evaluation, and reputation. The following is an explanation of these indicators.

**Figure 2.** The emergency suppliers' evaluation and selection criteria.

#### 1. Flexible supply capacity.

After the replenishment demand for emergency materials is issued, different enterprises have different emergency response speeds and resource allocation capabilities. In addition, when encountering material damage or other technical problems, the response capabilities of different companies also differ. Therefore, it is necessary to choose suppliers with more flexible supply capabilities.

2. Delivery capacity.

Priority is given to suppliers with strong delivery capacity due to differences in delivery quantity, timeliness, completion rate, and accuracy.

3. Price.

This indicator is used to measure the economic factors in the procurement cost of emergency material reserves. Even if the emergency work itself is weakly economical, the more efficient the rescue, the higher the economic cost, but the price and cost factors must be comprehensively considered. This includes price stability, bulk agreement preferential price, etc. Here, refers to the unit cost of the material allocated to the distribution center.

4. Quality.

The quality of emergency materials determines the quality of the rescue after emergencies. This includes the product qualification rate, quality certification system, engineering technology level, etc.

5. Social evaluation and reputation.

The evaluation and reputation of enterprises in society must be considered, including whether they have a good image in the hearts of the people and a high social reputation. The difference is that disaster relief has a strong public welfare nature. If the social evaluation degree of the suppliers is not high, it may cause unfair doubts in the public.

In general, the evaluation and selection indicators of the emergency materials suppliers should closely focus on the characteristics of the emergency rescue work, consider the connection between the indicators and the working process, and highlight the emergency ability of the supplier enterprises, so as to choose. At the same time, the above analysis shows that the indicators selected by emergency materials suppliers can be basically divided into two categories; one is economical indicators, where the smaller the evaluation value, the better. Price is an economic indicator. The other is the non-economical indicator; the

greater the evaluation value, the better. Flexible supply capacity, delivery capacity, quality, social evaluation, and reputation are non-economical indicators.

#### **4. Methodology**

*4.1. Fuzzy Set Theory*

**Definition 1.** *R is a real number set, F*(*R*) *represents all the fuzzy sets, and a fuzzy set M* ∈ *F*(*R*) *on R is called a fuzzy number [43].*


**Definition 2.** *In fuzzy mathematics, the membership function of fuzzy sets can be represented by a triangular distribution:*

$$\mu\_M(\mathbf{x}) = \begin{cases} \frac{\frac{\mathbf{x}}{m-l} - \frac{l}{m-l}}{\frac{\mathbf{x}}{m-u} - \frac{u}{m-u}} \mathbf{x} \in [l, m] \\ \frac{\frac{\mathbf{x}}{m-u} - \frac{u}{m-u}}{0} \mathbf{x} \in [m, u] \\ 0 \quad \text{others} \end{cases} \tag{1}$$

*where, l* ≤ *m* ≤ *u, l and u represent the lower limit and upper limit of M, respectively, and m is the most likely value.*

The triangular fuzzy number can be defined by (*l* < *x* < *u*), and represents the non-fuzzy number when *l*, *m*, and *u* are equal. *M* = {*x* ∈ *R*|*l* ≤ *m* ≤ *u*}.

Its image is shown in Figure 3.

**Figure 3.** Triangular distribution function.

The fuzzy number of the triangular distribution is represented as *M* = [*l*, *m*, *u*](*l* ≤ *m* ≤ *u*). If the size of the fuzzy number is compared, it needs to be de-fuzzy, and the average comprehensive representation method is selected to de-fuzzy. According to the Equation (2), the defuzzification value *P*(*M*) is as follows.

$$P(M) = (l + m + u)/3\tag{2}$$

**Definition 3.** *Set up triangular fuzzy numbers M*<sup>1</sup> *and M*2, *M*<sup>1</sup> = (*l*1, *m*1, *u*1), *M*<sup>2</sup> = (*l*2, *m*2, *u*2), *M*<sup>1</sup> + *M*<sup>2</sup> = (*l*<sup>1</sup> + *l*2, *m*<sup>1</sup> + *m*2, *u*<sup>1</sup> + *u*2), *M*<sup>1</sup> − *M*<sup>2</sup> = (*l*<sup>1</sup> − *l*2, *m*<sup>1</sup> − *m*2, *u*<sup>1</sup> − *u*2), *M*<sup>1</sup> ∗ *M*<sup>2</sup> = (*l*1*l*2, *m*1*m*2, *u*1*u*2), *γ* ∗ *M*<sup>1</sup> = (*γl*1, *γm*1, *γu*1).

#### *4.2. Fuzzy SWARA Method*

The stepwise weight assessment ratio analysis (SWARA) method is a new multicriteria decision-making method proposed by Kersuliene et al. [44] to determine standard weights [45]. An important feature of SWARA is the ability to assess the accuracy of experts regarding the weighting criteria in the methodological process. Experts play a crucial role

in the process of judging the criteria and weights. Each expert sets the priority of each criterion, and then considers the total results to rank all factors. In this method, the highest priority will be assigned to the most valuable indicator, and the lowest priority will be assigned to the lowest value evaluation indicator.

Considering that the knowledge, experience, and information of experts are different, their scores directly affect the accuracy of the final results in the evaluation process. In order to weaken the decisive role of subjective factors in the traditional SWARA method and reduce the influence of a single decision maker's preference, the fuzzy SWARA method is adopted in this paper. According to the level of knowledge and experience of experts, combined with the fuzzy set, different experts are given the weight and the indicators weight are obtained. Here, is a description of fuzzy SWARA.

Step 1. Relative importance of different indicators and the corresponding order of defuzzification values. Each decision expert expresses the relative importance of each indicator. The triangular fuzzy number for each indicator can be obtained according to the corresponding linguistic variable. The defuzzification value of each indicator is then obtained [46–49]. The defuzzification values of the different indicators are arranged in descending order [50,51].

Step 2. The correlation parameter *sj* (*j* ≥ 2) between two adjacent indicators before and after is determined. The correlation parameter *sj* (*j* ≥ 2) can be determined according to different rules. In this study, the difference between the defuzzification values of two adjacent indicators is used to calculate the correlation parameter.

Step 3. Calculate the comparison coefficient *kj* according to Equation (3).

$$k\_{\vec{j}} = \begin{cases} & 1, \\ & s\_{\vec{j}} + 1, \ j > 1 \end{cases} \tag{3}$$

Step 4. Calculate the relative weight *qj* according to Equation (4).

$$q\_j = \begin{cases} \frac{1}{q\_{j-1}} & j = 1\\ \frac{q\_{j-1}}{k\_j}, & j > 1 \end{cases} \tag{4}$$

Step 5. Calculate the final weight *ω<sup>j</sup>* according to Equation (5).

$$
\omega\_{\vec{j}} = \frac{q\_{\vec{j}}}{\sum\_{k=1}^{n} q\_k} \tag{5}
$$

#### *4.3. Actor Analysis Method*

Actor analysis is a comprehensive factor evaluation method. The economic and noneconomic factors are unified according to their relative importance, and the factors are comprehensively analyzed from different degrees [52]. In this study, the fuzzy SWARA method was combined with the actor analysis method to determine the priority of the alternatives.

Step 1: Calculation of the importance value of economic factors *Tj*.

$$T\_{\bar{j}} = \frac{\frac{1}{c\_{\bar{j}}}}{\sum\_{j=1}^{n} \frac{1}{c\_{\bar{j}}}} \tag{6}$$

There are *n* alternatives, and *cj* is the cost value reflected by the economic factors of the alternative, which is the economic cost. The higher the cost, the worse the economy; therefore, taking the reciprocal for comparison, the larger the result, the better is the economy.

Step 2: Calculation of the importance value of non-economic factors *Tf*.

(1) The pairs of alternatives are compared using a single factor. According to the importance evaluation value given by the experts, the proportion value of the better one is 1 point, and the worst one is 0. Therefore, the relative importance value *Tdi* of every single non-economic factor for different alternatives is obtained. *Gi* is the specific gravity value of the alternatives for a single factor.

$$T\_{di} = \frac{G\_i}{\sum\_{j=1}^{n} G\_i} \tag{7}$$

(2) The relative importance value *Tdi* of every single non-economic factor is multiplied by its weight value *Wi* and accumulated to obtain the importance factor *Tf*. The number of non-economic factors is *m*.

$$T\_f = \sum\_{i=1}^{m} \mathcal{W}\_i T\_{di} \tag{8}$$

Step 3: Calculation of the importance values *Fi*.

The importance values of the alternatives are superimposed according to economic and non-economic factors to obtain the ranking of alternatives. *M*, *N* are the relative importance of economic factors (objective factors) and non-economic factors (subjective factors) respectively, *M* + *N* = 1.

$$F\_i = MT\_j + NT\_f \tag{9}$$

#### **5. Case Analysis**

A schematic of the research methodology is shown in Figure 4. First, the weights of the experts were determined according to the triangular fuzzy set method. The SWARA method of triangular fuzzy sets was then used to determine the weights of the evaluation indicators. Finally, the actor analysis method was used to determine the priority of each alternative enterprise.

**Figure 4.** The schematic diagram of the research methodology.

It is assumed that city J needed to carry out reserve work of emergency relief materials, and the cooperative emergency suppliers needed to be determined. Originally, ten companies are selected. After a preliminary judgment and evaluation by three experienced experts in the emergency industry, the remaining five enterprises served as alternatives.

The indicator set was determined as *C* = {*C*1, *C*2, *C*3, *C*4, *C*5}, *C*<sup>1</sup> corresponding to flexible supply capacity, *C*<sup>2</sup> to delivery capacity, *C*<sup>3</sup> to price, *C*<sup>4</sup> to quality, and *C*<sup>5</sup> to social evaluation and reputation, respectively. Here, *C*<sup>3</sup> is the economic indicator, *C*1, *C*2, *C*4, and *C*<sup>5</sup> are non-economic indicators. The unit prices of the five alternatives are 18, 22, 30, 15, and 20. A questionnaire for the evaluation of indicators was established and sent to three experienced experts. The evaluation values in the questionnaire were designed according to the importance scale tables in Tables 2 and 3 [53].

**Table 2.** Importance scales for evaluating decision makers.


**Table 3.** Correspondence of linguistic variable values.


Phase 1: Determining decision maker set and corresponding weight, alternative enterprise set, and evaluation indicators set.

The alternative enterprise set is *E* = {*E*1, *E*2, *E*3, *E*4, *E*5}. The evaluation indicators set is *C* = {*C*1, *C*2, *C*3, *C*4, *C*5}. The set of decision-makers is represented by *A* = {*A*1, *A*2, *A*3}, and the relative importance value of decision-makers are calculated according to the importance scale of Table 2. The order *ε* = (*ε*1,*ε*2,*ε*3) *<sup>T</sup>* as the importance weight of the expert group. The weight of the three decision makers can be obtained according to Table 2 and Equation (10). The results are presented in Table 4.

$$\varepsilon\_{k} = \frac{P(\mathcal{M}\_{k})}{\sum\_{k=1}^{p} P(\mathcal{M}\_{k})}, \ k = 1, 2, \dots, p \tag{10}$$

$$\boldsymbol{\varepsilon} = (\varepsilon\_1, \varepsilon\_2, \varepsilon\_3)^T = (0.3750, 0.2917, 0.3333) \tag{11}$$

**Table 4.** Weight of decision makers.



Decision makers assigned the importance of the indicators based on the linguistic variable values in Table 3. The aggregation triangular fuzzy number was obtained using Equation (11). The defuzzification value was calculated using Equation (2). The obtained defuzzification value P(*Cj*) was sorted in descending order, according to Equations (3)–(5), as shown in Table 5. The weights of four non-economic indicators were obtained as follows:

$$\mathcal{W} = (\mathcal{W}\_1, \mathcal{W}\_2, \mathcal{W}\_4, \mathcal{W}\_5) = (0.2604, 0.2767, 0.2583, 0.2046) \tag{12}$$

**Table 5.** Significance of the evaluation indicators.


Phase 3: Calculate the importance values.

According to the Equations (6)–(8), Tables 6–8, and the price, the importance values of economic and non-economic factors were calculated.

$$T\_{j\to 1\prime}T\_{j\to 2\prime}T\_{j\to 3\prime}T\_{j\to 4\prime}T\_{j\to 5\prime} = (0.2214, 0.1812, 0.1326, 0.2656, 0.1991) \tag{13}$$

$$T\_{f\to 1\prime}T\_{f\to 2\prime}T\_{f\to 3\prime}T\_{f\to 4\prime}T\_{f\to 5} = (0.2981, 0.0928, 0.2912, 0.1351, 0.1828) \tag{14}$$

**Table 6.** Decision makers' evaluation grades of alternative enterprises.


**Table 7.** Comparison results on non-economic factors.



**Table 8.** Comparison results summary.

Let *M* = *N* = 0.5. The importance values of the alternatives were obtained. and the order of importance was *E*<sup>1</sup> > *E*<sup>3</sup> > *E*<sup>4</sup> > *E*<sup>5</sup> > *E*2. Therefore, *E*<sup>1</sup> is the best choice.

$$F\_{\rm E1}, F\_{\rm E2}, F\_{\rm E3}, F\_{\rm E4}, F\_{\rm E5} = (0.2598, 0.1370, 0.2119, 0.2003, 0.1910) \tag{15}$$

#### **6. Sensitivity Analysis**

To verify the effectiveness of the method, sensitivity analysis is carried out in this section. The relative importance of economic factors and non-economic factors are adjusted, and the remaining indicators are kept unchanged to test the stability of the fuzzy linguistic multi-criteria decision-making method. Make the scenario S1 = Tj:Tf = (0.1, 0.9), the relative importance weights of the economic factor indicators Tj and Tf are set to 0.1, 0.9, respectively. There are nine scenarios, S1 = Tj:Tf = (0.1, 0.9), S2 = Tj:Tf = (0.2, 0.8), S3 = Tj:Tf = (0.3, 0.7), S4 = Tj:Tf = (0.4, 0.6), S5 = Tj:Tf = (0.5, 0.5),S6 = Tj:Tf = (0.6, 0.4),S7 = Tj:Tf = (0.7, 0.3), S8 = Tj:Tf = (0.8, 0.2), S9 = Tj:Tf = (0.9, 0.1).

In each scenario, the importance values of alternative emergency suppliers were calculated respectively, which are shown in Table 9 and Figure 5. As can be seen from the results, the obtained enterprise priorities are not exactly the same in the nine different scenarios. When the important values of economic factors and non-economic factors are the same, *E*<sup>1</sup> > *E*<sup>3</sup> > *E*<sup>4</sup> > *E*<sup>5</sup> > *E*<sup>2</sup> can be obtained. When the importance value of economic factors is higher and decision-making preference is toward economic factors, 1 and 4 have higher priority. When the importance value of non-economic factors is higher and the decision preference is toward non-economic factors, the priority of 1 and 3 is high.


**Table 9.** Sensitivity analysis of rankings by *FEi*.

It can be concluded that the triangle fuzzy SWARA factor analysis method used in emergency supplies supplier selection is reliable, can not only reflect the importance weights of different experts themselves and the ratings of the target enterprise, more can adjust the economic factors and non-economic factors to reflect the decision-making preference in practical application. It is convenient for decision-making departments to make final decisions, which has good operability and reference value.

**Figure 5.** Sensitivity analysis of rankings by *FEi*.

#### **7. Discussion**

Various natural disasters and emergencies occurred frequently in recent years; after the disaster, to minimize adverse effects, we must attach importance to disaster relief work. Therefore, it is necessary to establish an effective emergency supply chain, among which a reasonable selection of emergency suppliers is an important link for all departments to cope with new challenges and build a modern emergency support mode. Based on the analysis of the evaluation indicators of emergency suppliers, the fuzzy SWARA method was used to give the indicators' weights, and the fuzzy SWARA-based actor analysis method was used to establish the evaluation model of emergency suppliers considering the final decision preference. Thus, the best choice under different decision-making preferences can be obtained, which provides a scientific theoretical basis for decision-making departments to make final decisions, and to ensure the smooth development of emergency rescue work. This study has the following management implications:

(1) The evaluation index system of emergency suppliers proposed in this paper is developed from five aspects of flexible supply capacity, delivery capacity, price, quality, social evaluation, and reputation. The quantifiable economic factors and non-quantifiable noneconomic factors are fully considered, which can provide more comprehensive reference for management decision makers. Among them, the flexible supply capacity, emergency delivery capacity, and quality factors are closely related to the timeliness, stability, and reliability of supplies delivery. The acceptable price level is determined by the financial expenditure capacity of the management department, and the social evaluation and reputation will involve the public's view on the fairness and credibility of the management department. That means there are antinomic relationships among some indicators. In the decision-making process, it should be fully considered and relatively appropriate suppliers should be chosen to avoid some disadvantages of suppliers, which can not only guarantee the rescue work, but also maintain a good social image of the emergency management department.

(2) Since January 2020, novel coronavirus pneumonia has been spreading worldwide. Novel coronavirus pneumonia is a new type of public health emergency. It needs comprehensive emergency management and needs a coordinated response from different countries and regions. It is necessary to strengthen international macroeconomic policy coordination, maintain a stable and smooth supply chain of the global industrial chain, and jointly cope with the new crown pneumonia epidemic. At the same time, in the process of emergency management, reserve inventory, emergency demand, and supplier supply capacity should be deeply analyzed, and the uncertain impact of emergency inventory management should

be considered. In order to maximize the efficiency of emergency resource allocation, strong unified organization and implementation are needed in terms of material sources, material distribution, social security, and other measures. Emergency supplier evaluation and selection is an important link affecting the efficiency of emergency resource allocation, which has an important impact on the response and efficiency of the whole emergency supply chain.

(3) Under uncertain conditions, the total cost input of emergency rescue will increase with the improvement of the requirements on service level and reliability. Therefore, in the practical decision making of emergency supplier selection, the final decision maker should give certain decision-making preferences and fully consider the financial expenditure ability, so as to achieve the ideal decision-making goal within the reasonable total cost budget.

#### **8. Conclusions**

Due to the uncertainty and abruptness of natural disasters and emergencies, coupled with the complexity and changeability of the rescue process, the emergency rescue management has put forward high requirements. In order to respond quickly and effectively, emergency suppliers can be determined in advance, and the emergency material supply plan can be arranged to ensure emergency supply.

(1) This paper combined the fuzzy theory and the actor analysis method; fuzzy numbers are used to represent uncertainty and fuzziness, which improves the scientific and feasibility of decision making. Fuzzy numbers were used to convert the evaluation language of experts and establish a triangular fuzzy actor analysis method using the constraint nature of triangular fuzzy numbers. To get closer to the actual decision-making situation, the weight of each indicator was determined according to the situation of the experts. The weight of each indicator is determined using the fuzzy SWARA method. The triangle fuzzy actor analysis method determined the priority ranking of different emergency suppliers, which fully considered the decision preference for economic and non-economic factors.

Additionally, the specific application process of the method is given through a numerical example in this paper, and the optimal selection strategy of emergency supplies suppliers under different decision preferences is obtained by combining the sensitivity analysis of economic and non-economic factors. This method provides an evaluation method for emergency suppliers selection, which has reference value in practice. The results show that when the preferences of economic factors and non-economic factors are different, the optimal choice is different. When the important values of economic factors and non-economic factors are the same, we can obtain *E*<sup>1</sup> > *E*<sup>3</sup> > *E*<sup>4</sup> > *E*<sup>5</sup> > *E*2. When the decision preference is toward economic factors, *E*<sup>1</sup> and *E*<sup>4</sup> have higher priority. When the decision preference is toward non-economic factors, *E*<sup>1</sup> and *E*<sup>3</sup> have higher priority. This also indicates that in practical decision-making, disaster needs and financial situation need to be closely linked to achieve the best supply of materials within the range of reasonable economic expenditure.

Different from other qualitative or quantitative evaluation methods, the results of AHP, COPRAS, SWARA, and TOPSIS are mainly determined by the subjective evaluation of external experts, but they cannot directly reflect the antinomic relationship between economic and non-economic factors, which is not conducive to the reference and choice of the final decision-making departments. The method proposed in this paper can not only reflect the importance weights of different experts themselves and the ratings of the target enterprise, but it can also adjust the important value of the economic factors and economy factors to reflect the actual application of the decision preference. This method has good operability and reference value, which is convenient for the decision-making departments to make the final choice according to their own actual situation.

(2) The evaluation scope of emergency supplies suppliers has a wide range, especially for different types of suppliers involved in different indicators. This paper mainly puts forward a single supplier selection scheme from the perspective of epidemic prevention supplies, and there is still a lot of research space. In future research, the following aspects should be studied. First, it can be considered to increase the evaluation and analysis of specific materials, to solve the problem of supplier selection of emergency materials in a more targeted manner. Second, considering the impact scope of large emergencies, a single supplier may not be able to meet the actual demand, so the selection and configuration of emergency suppliers can be carried out from the perspective of multi-supplier collaborative supply. Third, it can study the emergency supply chain of multi-product, multi-level, and multi-inventory, considering the existing inventory and the supply guarantee ability of suppliers. The dynamic demand can be expressed by appropriate random function, which fully reflects the behavior of multi-level supply chain and completes the allocation of emergency materials. Finally, in the future, it can be considered to further extend the research from trapezoidal fuzzy sets, intuitionistic fuzzy sets, interval intuitionistic fuzzy sets, Z-number theory, and other fuzzy sets, so as to express uncertainty and fuzziness more precisely [54–56].

**Author Contributions:** Conceptualization, Z.X. and H.L.; formal analysis, H.L.; funding acquisition, J.Y.; methodology, Z.X. and H.L.; project administration, J.Y.; software, H.L. and Z.X.; supervision, J.Y.; writing—original draft, Z.X. and H.L.; writing—review and editing, H.L. and Z.X. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by the National Natural Science Foundation of China (51279153) and supported by the Fundamental Research Funds for the Central Universities (2021-zy-010).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **The Effect of Competency-Based Triage Education Application on Emergency Nurses' Triage Competency and Performance**

**Sun-Hee Moon and In-Young Cho \***

College of Nursing, Chonnam National University, 160 Baekseo-ro, Dong-gu, Gwang-ju 61469, Korea; sunnymon@jnu.ac.kr

**\*** Correspondence: kikiin1024@jnu.ac.kr

**Abstract:** The Korean Triage and Acuity Scale (KTAS) is used to determine emergency patient priority. The purpose of this study was to develop the Competency-Based Triage Education Application (CTEA) using KTAS and evaluate its effectiveness on emergency nurses' triage competency and performance. The developed CTEA mobile application comprised 4 lectures, 12 text-based cases, and 8 video-based triage scenarios. A quasi-experimental pre-post design with a comparison group (CG) was used to evaluate the effectiveness of the CTEA. Thirty-one participants were assigned to an intervention group (IG) and used the application for at least 100 min over one week. Thirty-five participants were assigned to a CG and underwent book-based learning, which covered the same content as the CTEA. Triage competency (t = 2.55, *p* = 0.013) and performance (t = 2.11, *p* = 0.039) were significantly improved in the IG. The IG's undertriage error was significantly reduced compared to that of the CG (t = 2.08, *p* = 0.041). These results indicated that the CTEA was effective in improving the emergency nurses' triage competency and performance. This application will be useful as a program for providing repeated and continuous triage education.

**Keywords:** triage; mobile applications; education; distance; competency-based education; KTAS

#### **1. Introduction**

In modern emergency departments (EDs), the triage process is applied to quickly identify patients' acuity and classify their priority based on the severity of their conditions [1,2]. In EDs, emergency treatment is provided based on urgency rather than on a first-come, first-served basis; therefore, triage, which guides the allocation of limited medical resources—including health care providers and available medical equipment—according to patient's conditions, is an essential system for patient safety [1,2]. Typically, nurses are the main practitioners of triage in EDs, acting as gatekeepers at ED entrances [2,3]. Hence, educational support to ensure sufficient triage competency among emergency nurses is imperative.

Due to its proliferation, triage scales have been developed; in particular, a five-tier triage scale that divides patient urgency into five levels is mainly applied in modern EDs [4]. The triage scales used worldwide include the Canadian Triage and Acuity Scale (CTAS), the U.S. Emergency Severity Index (ESI), the Australasian Triage Scale (ATS), and the U.K. Manchester Triage System (MTS) [4]. In addition to these, a triage scale suitable for each country's emergency medical system has recently been developed. In 2016, the Korean Triage and Acuity Scale (KTAS) was established based on CTAS and used in EDs nationwide [5]. To ensure that nurses can use the KTAS to make accurate and rapid decisions, appropriate educational support is required.

Various training methods have been applied to assist triage nurses in making accurate and rapid decisions. According to existing review studies on triage education, the methods used primarily include brief lectures with case studies, simulations, live actors, computerized scenarios, and game-based learning [6–8]. Since the goal of triage education is to improve emergency nurses' competence in making accurate and rapid decisions through

**Citation:** Moon, S.-H.; Cho, I.-Y. The Effect of Competency-Based Triage Education Application on Emergency Nurses' Triage Competency and Performance. *Healthcare* **2022**, *10*, 596. https://doi.org/10.3390/healthcare 10040596

Academic Editors: Amir Khorram-Manesh, Krzysztof Goniewicz and Holger Muehlan

Received: 23 January 2022 Accepted: 21 March 2022 Published: 22 March 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

various educational methods [2,7,9], there is a need for more efficient and learner-friendly methods to be developed. Countries that have created a representative triage scale offer efficient triage education online [6]; recently, various techniques, such as virtual reality and augmented reality, have been introduced [10,11].

A few KTAS-based education programs have been developed, including the official hospital-KTAS provider course, an offline training program consisting of lectures and text-based case studies [12]. However, one-time education programs offered offline suffer from certain limitations. Specifically, to improve their triage competencies, emergency nurses need to experience triage repeatedly and continuously [3], which one-time courses do not offer. According to extant review studies, smartphone-based education significantly improved nursing and medical science practitioners' clinical competency, knowledge, attitude, performance, and confidence [13,14]. Since online education via mobile devices can be accessed freely and repeatedly without spatial and temporal constraints, a positive effect can be expected for app-based triage education centered on the KTAS.

Therefore, the purpose of this study was to develop a mobile-app-based online education program to improve emergency nurses' triage competency and verify the effectiveness of the education program. This study hypothesized that participants using the Competency-Based Triage Education Application (CTEA) would have greater (1) triage competency and (2) triage performance, compared to those learning triage using a book.

#### **2. Materials and Methods**

#### *2.1. Design*

This was a pre- and post-test quasi-experimental study with a comparison group, designed to evaluate the effect of the CTEA on emergency nurses' triage competency and performance.

#### *2.2. The CTEA Development*

The basic framework of the CTEA was designed based on a literature review and a qualitative study conducted by the research team to explore the educational needs of emergency nurses. The development of the preliminary educational contents was led by the first author, who has 12 years of experience working in EDs and more than 6 years of triage experience. The lecture content was developed with reference to four triage books from the Emergency Nurses Association and KTAS committees and one emergency medicine book [15–19]. The educational content included four lectures, 12 text-based case studies (covering abdominal pain, dyspnea, fever, vomiting, epistaxis, simple treatment, cough, behavior, dizziness, vaginal bleeding, and traffic accidents), and eight conversational scenarios (covering acute myocardial infarction, acute stroke, major trauma, congestive heart failure, acute hemorrhage, urticarial rash, appendicitis, and hyperventilation) (Figure 1). The case study was designed to equip the learners to solve the competency targeted KTAS quizzes. The CTEA was structured such that learners could ask the operator questions about triage cases and communicate freely with each other (learner to learner).

An expert group consisting of two emergency medicine professors and four emergency nursing professors who evaluated the validity of the educational content using content validity indices (CVIs). The overall CVI score of the educational contents was 0.92. In accordance with the experts' opinions, the vital signs of the triage cases and the dialogue of the scenario were modified.

The educational content was then converted to video form. Specifically, the first author created a 50 min video containing four lectures. The actors filmed the conversational scenarios in 8 videos, each 2–3 min long. The developed educational content was loaded into an Android-based hybrid mobile app (Figure 2). Learners were able to score "energy" points when viewing triage lectures, taking quizzes, and reviewing cases using the app, and they were able to check their ranking against other learners.

**Figure 1.** Triage education program. KTAS = Korean Triage and Acuity Scale, V/S = vital sign, AMI = acute myocardial infarction, EMT = emergency medical technician, EMR = electronic medical records, ECG = electrocardiogram, BST = blood sugar test.

**Figure 2.** The Competency-Based Triage Education Application.

#### *2.3. Outcome Measurements*

To determine the nurses' KTAS competency, we measured four outcomes—critical thinking disposition, triage competency, triage knowledge, and triage performance.

Critical thinking disposition was the propensity to think critically to lead decisionmaking and problem-solving tasks [20]. In this study, critical thinking disposition was measured using a 5-point Likert scale developed by Kwon et al. in 2006 [20]. The scale consists of 35 items with potential total scores ranging from 35 to 175 points, with higher

scores indicating higher critical thinking disposition. At the time of development, the scale received a Cronbach's α of 0.89 [20]. In this study, it received a Cronbach's α of 0.83.

Triage competency is the ability to allocate medical resources efficiently by determining care priority according to patients' health statuses [1]. In this study, triage competency was measured using a 5-point Likert scale developed by Moon et al. in 2018 [21]. The triage competency measure comprises 30 items with five sub-factors, including clinical judgment, expert assessment, management of medical resources, timely decisions, and communication [21]. The potential scores of the scale range from zero to 150, with higher scores indicating higher triage competency. The Cronbach's α of the scale in Moon et al.'s study was 0.91 [21], while in this study it was 0.96.

To measure triage knowledge, we developed a scale consisting of 32 preliminary items based on the four triage books mentioned previously. The content validity verification of the developed items was conducted by a panel of experts who participated in the CTEA's content validity verification. Among the 32 preliminary items, one item with a CVI score of 0.67 was deleted, resulting in 31 final items; the CVI of the final triage knowledge scale was 0.96. The items were answered by selecting one true/false answer for each of the 31 questions, and triage knowledge was calculated by totaling up the correct answers. The KR-20 of the triage knowledge scale was 0.56.

To measure triage performance we created 10 triage scenarios, drafted based on the experience of the first author and revised through research meetings. An expert panel verified the content validity of the scenarios written in conversational format. The CVI of the triage scenarios was 0.93. Videos, which were approximately 2 min in length, were created for the 10 triage scenarios, to which participants had to respond by assigning the videos a KTAS-based score from level 1 to level 5. The number of correct answers would then be summed up, overtriage error totaled up, and undertriage error totaled up. The KR-20 of the triage performance scale was 0.71.

#### *2.4. Participants and Data Collection*

Six EDs in four cities participated in this study from July 2020 to January 2021. The emergency medical system in Korea consists of the regional emergency center (REC), which is in charge of treating critical emergency patients, and the local emergency center (LEC), which is in charge of treating emergency patients in their local area. Of the EDs participating in this study, two were RECs and four were LECs. All EDs have been performing triage using KTAS since 2016. Three EDs each were assigned to the intervention and the comparison group (Figure 3). Thirty-eight emergency nurses were assigned to the intervention group and 37 to the comparison group. In total, 75 emergency nurses in the EDs responded to our online survey comprising six measures—critical thinking, triage competency, triage knowledge, triage performance, demographic factor (age, gender, educational level), and clinical experience (total nursing, emergency nursing, triage) before intervention.

The participants in the intervention group were provided with instructions on installing the CTEA. The participants read the guide and then downloaded and installed the CTEA from Play Store. Android smartphones were rented for Apple smartphone users. The participants in the intervention group underwent triage training using the CTEA for over 100 min in one week. The lengths of time the intervention group participants spent learning via the CTEA were saved in the program so that the administrator could monitor them. Five of the learners who studied for less than 100 min over the week were excluded from the intervention group. After one week of using the CTEA, the intervention was deemed complete and a post-survey in the same format as the pre-survey was conducted online. Of the 33 participants who completed learning in the intervention group, two participants did not respond to the post-survey and were excluded. The remaining 31 participants' data were analyzed.

**Figure 3.** Participant's flow chart.

For the 37 participants in the comparison group, the 4 lecture materials and 12 textbased case studies (containing the same content as the CTEA material) were provided as a book. The comparison group participants completed 100 min of self-learning over one week using the triage book. The participants measured and reported their learning time to the researchers. After completing the learning, a post-survey in the same format as the pre-survey was conducted. Two of the comparison group participants were excluded due to insufficient self-learning time. The remaining 35 participants' data were analyzed.

The sample size calculation yielded G\*Power 3.1.9.4. According to a meta-analysis on virtual patient education for health professionals, the effect size *d* was 0.8 [22]. In a twotailed significance test with a power of 80% and an alpha level of 0.05, the sample size of each group was calculated to be 26. Therefore, the data used in this study's analysis met this criterion.

#### *2.5. Data Analysis*

The data were analyzed using IBM SPSS software, version 25.0(Armonk, NY: IBM Corp). The variables were analyzed based on frequencies, percentages, or means. We compared the baseline data of the two groups using chi tests for the categorical variables and *t*-tests for the continuous variables. To test the differences in critical thinking disposition, triage competency, triage knowledge, and triage performance, we calculated the difference between the pre-post means within each group and then compared the differences between the two groups using the independent *t*-test. As the pre-test result presented a difference in baseline triage knowledge, ANCOVA was applied to verify the difference in post-knowledge between the two groups using pre-knowledge as a covariate.

#### *2.6. Ethical Approval*

This study received approval from the University Institutional Review Board (No. 7001066-202002-HR-006). All participants willingly engaged in the data collection and intervention process and signed online informed consent forms. As a token of appreciation, all participants were awarded a \$50 gift card at the end of the study.

#### **3. Results**

#### *3.1. Baseline Characteristics*

The average age of the participants was 32.15 ± 7.47 years, and most of them (56) were women (Table 1). The average total nursing experience was 8.85 ± 7.54 years. In accordance with the Career Development Model of Nurses, a clinical career was defined in four-stages: novice (under 1 year of nursing experience), advanced beginner (1~3 years of nursing experience), competent (3~7 years of nursing experience), and proficient nurses (7 or more years of nursing experience) [23]. The participants' average emergency experience was 4.04 ± 3.54 years, with 24 (36.4%) of them falling within the competent level. The participants' triage experience was 1.25 ± 1.46 years. Except for the triage knowledge score, the baseline variables of the two groups before intervention did not differ (t = 2.56, *p* = 0.013) (Table 2).

**Characteristics Classification IG (n = 31) n (%) or Mean** ± **SD CG (n = 35) n (%) or Mean** ± **SD Total (N = 66) n (%) or Mean** ± **SD** *χ***2 or t (***p***)** Age Total (years) 34.09 <sup>±</sup> 8.19 30.42 <sup>±</sup> 6.41 32.15 <sup>±</sup> 7.47 2.00 (0.050) 20~29 11 (16.7) 20 (30.3) 31 (47.0) 4.66 (0.097) 30~39 12 (18.2) 12 (18.2) 24 (36.4) ≥40 8 (12.1) 3 (4.5) 11 (16.7) Gender Female 26 (39.4) 30 (45.5) 56 (84.8) 0.04 Male 5 (7.6) 5 (7.6) 10 (15.2) (0.834) Education level Associate degree 11 (16.7) 5 (7.6) 16 (24.2) 4.93 (0.085) Bachelor's degree 18 (27.3) 29 (43.9) 47 (71.2) Over master's degree 2 (3.0) 1 (1.5) 3 (4.5) Experience in nursing Total (year) 10.46 <sup>±</sup> 8.19 7.42 <sup>±</sup> 6.72 8.85 <sup>±</sup> 7.54 1.65 (0.104) Novice (<1) 2 (3.0) 2 (3.0) 4 (6.1) 2.53 (0.471) Advanced beginner (1≤~<3) 4 (6.1) 7 (10.6) 11 (16.7) Competent (3≤~<7) 6 (9.1) 11 (16.7) 17 (25.8) Proficient (≥7) 19 (28.8) 15 (22.7) 34 (51.5) Experience in the ED Total (year) 4.06 <sup>±</sup> 3.12 4.02 <sup>±</sup> 3.93 4.04 <sup>±</sup> 3.54 0.05 (0.960) Novice (<1) 5 (7.6) 5 (7.6) 10 (15.2) 0.13 (0.989) Advanced beginner (1≤~<3) 9 (13.6) 11 (16.7) 20 (30.3) Competent (3≤~<7) 11 (16.7) 13 (19.7) 24 (36.4) Proficient (≥7) 6 (9.1) 6 (9.1) 12 (18.2) Experience of triage (year) 1.27 <sup>±</sup> 1.77 1.23 <sup>±</sup> 1.14 1.25 <sup>±</sup> 1.46 0.10 (0.920)

**Table 1.** Baseline characteristics of participants.

IG = intervention group, CG = comparison group, SD = standard deviation, ED = emergency department.

#### *3.2. Comparison of Outcomes*

The first hypothesis in this study posited that the intervention group participants (those using the CTEA) would show higher triage competency than those in the comparison group. The intervention group's pre-post triage competency levels showed significant improvement compared with that of the comparison group (t = 2.55, *p* = 0.013) (Table 3). Specifically, after examining the sub-factors of the pre-post triage competency, the intervention group participants' clinical judgment (t = 2.39, *p* = 0.021) and timely decisions (t = 2.89, *p* = 0.005) showed significant improvement.


**Table 2.** Baseline outcome variables of participants.

IG = intervention group, CG = comparison group, SD = standard deviation, ED = emergency department, \* *p* < 0.05.



IG = intervention group, CG = comparison group, SD = standard deviation, \* *p* < 0.05.

The second hypothesis of this study was that the intervention group participants would show higher triage accuracy than those in the comparison group. When the pre-post triage accuracy of the two groups was verified, the intervention group showed significant improvement (t = 2.11, *p* = 0.039). To evaluate the cause of triage error, the averages of the two groups' undertriage and overtriage were investigated and compared (Figure 4). The results revealed a significant reduction in the intervention group's undertriage error, compared to the comparison group (t = 2.08, *p* = 0.041). However, both groups showed a decrease in the average of overtriage error.

**Figure 4.** Triage accuracy and causes of triage error. (**a**) Triage accuracy. (**b**) Undertriage. (**c**) Overtriage.

There was no significant difference between the two groups regarding critical thinking disposition when the pre- and post-test values were compared (t = −0.47, *p* = 0.633). For triage knowledge, after controlling the baseline values, that is, the covariate, no difference was found (F = 3.52, *p* = 0.065).

#### **4. Discussion**

Due to the spread of COVID-19, various non face-to-face education methods have recently been developed and utilized. Smartphone-based learning is one of the representative non face-to-face education methods that emphasizes learner accessibility. Recent research has reported that smartphone-based education improves clinical competency, knowledge, performance, attitude, and confidence in nursing and medical science practice [13,14]. Since a smartphone-based triage education program was developed, distance training and repeat clinical decision-making practice was rendered possible [24,25]. The significance of this study was the development of a mobile education app for strengthening the triage competency of emergency nurses based on the KTAS and the verification of its effectiveness. This KTAS-based educational mobile app development and effectiveness verification was the first attempt in Korea. Furthermore, since the importance of repetitive education is emphasized for the advancement of triage [3], the CTEA can be a complementary method to official triage education that allows iterative and convenient learning.

The Emergency Nurses Association (ENA) establishes practice guidelines for triage qualifications; thus, it is necessary to strengthen triage competency by improving knowledge and skills through appropriate triage education courses [26]. Extant research also reported competency and knowledge improvement as the main effect of triage education. Previous research reported that nursing students' competency and knowledge significantly improved after they were provided with education, including role plays and lectures based on the Simple Triage and Rapid Treatment (START) triage [27]. In a disaster nursing education program comprising lectures and practical tasks, nursing students' disaster nursing knowledge, disaster triage performance, and disaster readiness increased [28]. In this study, after undergoing CTEA-based training, the participants' triage competency improved. However, although the mean difference in triage knowledge between the two groups in this study was significant, there was no significant difference when the baseline score was adjusted using ANCOVA. Compared to the previous face-to-face research, the CTEA was conducted in a non face-to-face manner, which, except for observing learning time, translated to the limited monitoring of learners. Thus, triage knowledge may not have improved. Additionally, the intervention group's triage knowledge pre-test score and the comparison group's triage knowledge pre- and post-test score were almost similar. This may be due to the ceiling effect.

Triage competency comprises five attributes: clinical judgment, expert assessment, management of medical resources, timely decisions, and communication [1]. Among the sub-factors of triage competency, clinical judgment and timely decisions showed significant

improvement in this study. When emergency nurses perform triage, rapid and accurate decision-making is important, making it a crucial objective that must be achieved through triage education [1,7,18,26]. Therefore, improvements in clinical judgment and the ability to make timely decisions through the CTEA could be referred to as educational goals. However, sub-factors such as communication and expert assessment did not significantly improve in this study. As physical examination requires direct practice, it may have been difficult to achieve the same in mobile-based non face-to-face research. Other than the chat function, the CTEA did not have a feature that allowed participants to communicate with patients or medical staff. Due to these development limitations, communication ability may not have been improved. In cases where artificial intelligence plays the role of an emergency patient or where learners perform triage simulations with standard patients in a virtual space, improvements in learners' communication in non face-to-face education could be expected.

According to a review study, triage accuracy was reported to be 56.2~82.9% when analyzed based on written case scenarios or medical record review results [29]. Based on electronic medical records in the EDs using KTAS, the weighted kappa value was 0.69~0.83 for triage accuracy, which increased to 0.84 after problem-based learning [30,31]. In this study, triage accuracy significantly increased after using the CTEA; however, compared to a previous study [29], this was not high. Although the mobile-based education in this study had a limited effect on triage accuracy compared to face-to-face education, such education may be a good alternative; as triage education must be accessible and administered continuously and repeatedly to maintain triage accuracy [3] and mobile devices allow for such conveniences. Thus, if various mobile-based triage education programs are developed, triage quality can eventually be achieved and maintained.

There are two types of triage errors: undertriage and overtriage [32,33]. Undertriage is when treatment time for emergency patients is delayed due to underestimating the severity of the patients' condition at triage, potentially jeopardizing patients' safety [33]. Overtriage is when patients' conditions are overestimated, which can cause overcrowding in the ED and jeopardize patients' safety by endangering urgent patients [32]. Mistriage in the Korean EDs using KTAS resulted in an undertriage of 70.4% and overtriage of 29.6%, which was reported to have a higher underestimation error [31]. In this study, the incidence of undertriage was higher than that of overtriage when assessed using the video-based triage scenarios. The nurses' triage errors were reported to decrease when the web-based video education program developed using the KTAS was applied [34]. Similarly, in this study, when the mobile-based video scenario was applied, undertriage decreased significantly, and overtriage showed a decreasing pattern, although it was not statistically significant. Therefore, it could be said that the use of video-based scenarios for triage learning was effective and that it is necessary to develop various triage cases to ensure that learners can repeatedly access education more conveniently. In addition, since the incidence of undertriage in Korea's EDs using the KTAS was high, it may be necessary to develop an education program focused on reducing underestimation.

Gamification is the application of a game design in non-game contexts and has been widely used in education programs to motivate users, enhance psychological outcomes, and encourage behavioral change [35–37]. As in many other educational programs, gamification has also been introduced to triage education. A study on trauma triage education using serious game technology reported improvement in emergency physicians' decision heuristics [38]. According to review studies, game features commonly used in healthcare gamification include points, social interactions, leaderboards, progress statuses, levels, and immediate feedback [36]. The gamification strategies used in this study were "energy points," social interaction through chatting, levels, and immediate feedback. In a metaanalysis study verifying the effect of gamification in medical education, knowledge improvement and long-term knowledge retention were reported [39]. Although it was not possible to directly verify the educational effect of the gamification in this study design, if the same gamification strategy is used in future triage education, we expect continuity in the training effect.

This study had a few limitations. First, it measured the effectiveness of the CTEA program immediately after its use. Therefore, it was not possible to estimate the continuation of the educational effect on the participants and the interval requiring repeated education. If the effect of continuous education through the CTEA is confirmed, the application can be used more effectively as a complementary program for the official KTAS education. Second, this was a quasi-experimental study and participants were not randomly assigned. Randomization and blinding can assert strong causality and are suitable in research settings such as laboratories [40]. Many randomized studies examining the effectiveness of medical education have raised validity and reliability concerns; therefore, it has been suggested that they be categorized as quasi-experimental studies [40]. In this study, in accordance with the suggestions of a review [40], a randomized controlled trial, which was considered difficult in the educational field, was not unreasonably followed. Although random assignment could not be performed in this study, we attempted to maintain validity by assigning groups based on hospitals to avoid influence between participants and conducted a homogeneity test. Third, the book provided to the CG included 4 lectures and 12 text-based cases, except for the 8 videos provided to the IG. Although video-based contents could not be included in a paper-based book, differences in the amount of content provided may have affected the results. Fourth, this study was conducted only in six EDs in Korea; therefore, there was a limit to the generalizability of the results.

#### **5. Conclusions**

The development of the CTEA and its application to emergency nurses in this study improved triage competency and performance in the intervention group. Moreover, the CTEA was effective in reducing triage error. Therefore, the CTEA can be used as an educational program for continuous triage education because it is mobile-based and enables repetitive and convenient learning.

**Author Contributions:** Conceptualization, S.-H.M.; data curation, S.-H.M. and I.-Y.C.; investigation, S.-H.M.; methodology, S.-H.M.; project administration, S.-H.M.; software, S.-H.M. and I.-Y.C.; supervision, S.-H.M.; validation, S.-H.M.; visualization, I.-Y.C.; writing—review & editing, S.-H.M. and I.-Y.C.; funding acquisition, S.-H.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2017R1C1B5074027).

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board (IRB) of Changwon National University IRB (No. 7001066-202002-HR-006).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

#### **References**


## *Article* **Addressing Uncertainty by Designing an Intelligent Fuzzy System to Help Decision Support Systems for Winter Road Maintenance**

**Mahshid Hatamzad 1,\*, Geanette Polanco Pinerez <sup>1</sup> and Johan Casselgren <sup>2</sup>**


**Abstract:** One of the main challenges in developing efficient and effective winter road maintenance is to design an accurate prediction model for the road surface friction coefficient. A reliable and accurate prediction model of road surface friction coefficient can help decision support systems to significantly increase traffic safety, while saving time and cost. High dynamicity in weather and road surface conditions can lead to the presence of uncertainties in historical data extracted by sensors. To overcome this issue, this study uses an adaptive neuro-fuzzy inference system that can appropriately address uncertainty using fuzzy logic neural networks. To investigate the ability of the proposed model to predict the road surface friction coefficient, real data were measured at equal time intervals using optical sensors and road-mounted sensors. Then, the most critical features were selected based on the Pearson correlation coefficient, and the dataset was split into two independent training and test datasets. Next, the input variables were fuzzified by generating a fuzzy inference system using the fuzzy c-means clustering method. After training the model, a testing set was used to validate the trained model. The model was evaluated by means of graphical and numerical metrics. The results show that the constructed adaptive neuro-fuzzy model has an excellent ability to learn and accurately predict the road surface friction coefficient.

**Keywords:** adaptive neuro-fuzzy inference system (ANFIS); prediction methods; road surface friction; road transportation safety; winter road maintenance

#### **1. Introduction**

#### *1.1. Motivation*

Low temperatures and heavy snowfall can be problematic for road users, especially in countries with long and harsh cold-weather conditions. Road safety can be significantly reduced due to slippery surface conditions and poor visibility [1]. Adverse weather can decrease the reliability and productivity of the surface transportation system. In addition, it increases traffic delays and the likelihood of vehicle accidents that may lead to severe injuries and fatalities [2]. To minimize these negative impacts, roads must be kept clear of ice and snow through chemical (salting) and mechanical (plowing) operations, referred to as winter road maintenance (WRM) techniques. WRM helps to increase the friction between tires and the road surface to prepare the road for normal traffic flow, meaning that the road users can drive as fast as in summer (effective WRM); however, this can result in high expenses (inefficient WRM).

Prediction of the road surface friction coefficient (RSFC) can help decision makers to plan in advance for the type and timing of WRM to improve decision support systems (DSS). The prediction of RSFC is a comprehensive calculation from different performance aspects using multiple physical dynamic variables (e.g., weather temperature, ice layer, snow

**Citation:** Hatamzad, M.; Polanco Pinerez, G.; Casselgren, J. Addressing Uncertainty by Designing an Intelligent Fuzzy System to Help Decision Support Systems for Winter Road Maintenance. *Safety* **2022**, *8*, 14. https://doi.org/10.3390/ safety8010014

Academic Editors: Amir Khorram Manesh and Tom Brijs

Received: 1 November 2021 Accepted: 15 February 2022 Published: 17 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

height, etc.), which often shows a nonlinear relationship between road surface friction and the considered variables. In fact, sensors are not always able to measure road surface conditions (RSC) precisely, due to various reasons such as signal noise. Therefore, uncertainties need to be considered to enhance the accuracy of prediction models and improve the WRM performance.

#### *1.2. Significance of the Topic*

An accurate RSFC prediction model helps decision makers to effectively prepare for and respond to severe road conditions in winter (such as snowstorms and sharp reductions in RSFC), in order to maximize road safety for road users. Addressing uncertainty in a prediction model allows decision makers to systematically prevent hazards (such as fatal vehicle accidents) and strategically organize WRM resources to mitigate the detrimental impacts that a disaster can cause. In addition, avoiding the use of extra salt (chemicals) on the road surface minimizes cost and environmental impacts. In fact, high WRM quality and using an optimal salt quantity minimizes damage to vehicles, road infrastructure, vegetation, lakes, rivers, etc.

#### *1.3. State-of-the-Art Method*

There are several studies that present various methods to reach effective and efficient WRM. Shao and Lister [3], the authors used air temperature, wind speed, surface temperature, and dew-point as input variables to develop an automated road ice prediction model. Mohseni [4] applied a regression model to select the strongest features (pavement temperature, latitude, air temperature, elevation, and depth into the asphalt concrete layer) to develop a prediction model for low pavement temperature based on laboratory tests. Kangas et al. [5] developed a simulation model named RoadSurf using numerical weather forecasts as input to predict road surface temperature and condition. Moreover, developing sensor technologies has had a significant impact on monitoring road surface conditions in winter. Ye et al. [6] presented a review about developing and implementing advanced technologies to achieve safe and efficient WRM. Ewan et al. [7] investigated the reliability of an optical sensor to measure snow depth, water depth, and surface state (dry, wet, icy, etc.). Feng and Fu [8] investigated the performance of pavement sensors, and their results show that a sensor cannot always detect friction precisely. However, WRM effectiveness and efficiency can be improved by developing data-driven approaches using historical data collected by sensors. Ahabchane et al. [9] proposed a data-driven regression model using geometry, weather, and telemetry data to predict the amount of salt and abrasives in street segments. Pan et al. [10] applied deep neural networks to classify RSC according to images. Liu et al. [11] utilized machine learning algorithms (gradient-boosting) to develop road surface temperature forecasting. Roychowdhury et al. [12] applied neural networks to design a methodology to estimate RFSC. Panahandeh et al. [13] employed machine learning (ML) classification algorithms to predict RFCS for connected vehicles. Pu et al. [14] developed a daily RSFC prediction model using a long–short-term memory neural network based on the following three scenarios: (i) considering only daily friction data, (ii) selecting water thickness as an input variable, and (iii) selecting road surface temperature and water thickness as predictors. Their results showed that the second scenario had the highest accuracy. ML algorithms are powerful techniques to predict different nonlinear problems. Optical and road-mounted sensors are mostly used to measure data-related road surface conditions. Sometimes, numerical data derived from sensors can be associated with uncertainty due to imprecision, vagueness, or ambiguity. Song et al. [15] estimated maximum RSFC under uncertainty using deep learning. Matusko et al. [16] presented a new approach by adding neural networks to the friction estimator model, to enhance the estimation quality by compensating for the impacts of uncertainties. Kim et al. [17] designed a system for composite friction control, which included friction uncertainty using recurrent fuzzy neural networks. While previous research studies have contributed significantly to developing different dimensions of WRM, there has, thus far, been no study to predict RSFC by designing an

adaptive neuro-fuzzy inference system (ANFIS), which is able to handle the uncertainty hidden in historical data extracted by sensors. An adaptive fuzzy RSFC prediction model with high accuracy plays an important role in making WRM plans in advance, in order to achieve effective and efficient WRM.

#### *1.4. Contributions*

Reviewing previous studies on WRM reveals that it is not easy to establish an accurate data-driven RSFC prediction model, due to dynamic weather conditions that can lead to variation in road surface conditions. In addition, historical data collected by sensors can be associated with uncertainty, which must be modeled. To model this complex problem, the main contribution of this study is the design of an ANFIS model to predict RSFC using real data measured by optical and road-mounted sensors. The ANFIS model fuzzifies the crisp data for simulating this complex problem, associated with uncertainty.

#### *1.5. Outline of the Paper*

The remainder of this paper is organized as follows. Section 2 explains the summary of ANFIS. The data and methods are defined in Section 3. In Section 4, we present results. Finally, in Section 5, a conclusion is drawn.

#### **2. Adaptive Neuro-Fuzzy Inference System (ANFIS)**

Crisp numerical data points can be fuzzified and represented by membership functions (MFs) [18]. In recent years, artificial intelligence methods, including fuzzy intelligent techniques, have been extensively used in different fields such as economics, medicine, and engineering. The ANFIS model was proposed by Jang in the 1990s [19] and can be considered as a universal estimator for predicting long- and short-term effects [20]. ANFIS is a five-layer adaptive network that illustrates the relationship between inputs and outputs to simulate complex problems associated with uncertainties by creating fuzzy variables [21]. The ANFIS network utilizes the learning ability of neural network concepts and the reasoning mechanisms of the Takagi–Sugeno fuzzy interference system (FIS) [22]. Due to using both fuzzy logic and neural networks, ANFIS can benefit from both models' principles in a single model. The inference system employs fuzzy "if-then" rules, which have a learning ability to estimate nonlinear functions [23].

#### **3. Data and Methods**

Figure 1 shows the framework of this study and its different steps, which are explained in this section.

#### *3.1. Data Collection*

Although there is an obvious relationship between road surface conditions and weather conditions, the proposed model's input variables are defined as conditions that impact the WRM performance. In addition, the output variable is defined as a result gained from the input variables. One of the major benefits of the data-driven model is that different kinds of inputs and outputs are allowed to be included in the model without having special relationships. Moreover, all of the input variables have a similar opportunity to affect the road conditions. Therefore, here, air temperature, surface temperature, ice layer, snow layer, water thickness, and snow height were chosen as input variables and RSFC was chosen as the output variable. Historical data of these variables were measured every 10 min and collected from the Swedish Transport Administration's RWIS station on a European road at test site E18 in February 2019. The test site E18 is located in Sweden, Northern Europe, between Enköping and Västerås, the latitude and longitude are approximately 59.724 and17.029 [24], and the type of pavement was asphalt. The road weather station measured air temperature. Optical sensors measured ice layer (mm), snow layer (mm) (new snow), and RSFC. A road-mounted sensor measured surface temperature (◦C), snow height

(mm) (both new and old snow), and water thickness (mm). Table 1 shows the statistical description of the dataset.

**Figure 1.** The framework used this study.

**Table 1.** Statistical description of the dataset.


In Table 1, the first column shows the number of observations, which are equal for all variables. The second and third columns illustrate the mean value and standard deviation of observations, respectively. The fourth and eighth columns are the minimum and maximum values for each variable. The fifth, sixth, and seventh columns demonstrate the 25th percentile (the lower or first quartile), 50th percentile (the median), and 75th percentile (the upper or third quartile), respectively.

#### *3.2. Feature Selection*

In the previous step, variables influencing RSFC were initially chosen. However, reducing the number of input variables decreases the model complexity and enhances the training process, which can lead to enhancing the model accuracy. Therefore, we used the Pearson correlation coefficient to select the most significant predictors. Table 2 shows the absolute value of correlation between the input variables and RSFC. Out of six input variables, four variables (ice layer, snow layer, water thickness, and snow height) were highly correlated with RSFC. Thus, these four input variables were used to design an RSFC prediction model.


**Table 2.** Pearson correlation coefficients between input variables and RSFC.

#### *3.3. Dividing the Dataset into Training and Testing Sets*

The dataset needed to be divided into training and testing sets. The training dataset optimizes the parameters of the model, and the test dataset evaluates the model performance to predict RSFC. In this study, 70% of the observations were considered for training the ANFIS model and the rest of the observations were applied to test the model, since the testing set needs to be large enough to lead to meaningful statistical results. Table 3 shows the statistical information of the testing set.

**Table 3.** Statistical description of the testing set.


#### *3.4. Generating Basic Fuzzy Inference System*

In this stage, input variables were fuzzified by using Genfis3 in Matlab software. Genfis3 generates a structure based on the fuzzy inference system (FIS) using the fuzzy c-means (FCM) clustering method by extracting some fuzzy rules, which model the data behavior. The number of clusters specifies the number of rules and membership functions. We selected the 'Sugeno' type because Sugeno is more flexible to design a system more precisely [20]. The number of clusters was five, and the clustering (FCM) options were selected according to the default values in Matlab. The types of the input and output MFs were Gaussian and Linear, respectively. The number of input and output MFs (clusters) was five, equal to the number of fuzzy rules.

#### *3.5. Training Using ANFIS*

The training epoch number was set as 200, the initial step size was 0.01 with a decrease rate of 0.9 and an increase rate of 1.1, and the hybrid method was selected as the optimization method [25]. Increasing and decreasing the step sizes balances the exploration and exploitation to enhance the convergence speed and drive the process from the local minimum solution.

#### *3.6. Evaluating Performance of ANFIS*

A total of 3847 data points were considered in this study, of which 2693 (70%) observations were for training and 1154 (30%) were for testing the model. Figure 2 and Table 4 depict the structure of the ANFIS network designed in this study to predict the RSFC in winter. In fact, five layers built the ANFIS model based on node functions. The first layer was the "if part", or fuzzification; the second layer was implications (rules); the third layer was normalization; the fourth layer was the "then part", or defuzzification; and the fifth layer was the summation part (output) [20]. According to the collected road dataset, the fuzzy clustering of the predictors for a one-month period is presented in Figure 3. The degree of membership was between 0 and 1. A degree of membership of 0 means that the value does not belong to the given fuzzy set. A degree of membership of 1 means the value certainly belongs to the given fuzzy set. However, if the value of membership is between 0 and 1, it demonstrates the degree of uncertainty with which the value belongs in the given fuzzy set. The information and parameters of the MFs for each input (i.e., mean and standard deviation) and output (coefficients and constant) are shown in Tables 5–7. We considered five clusters for each input, and the output based on trial and error, which demonstrated the best results.

**Figure 2.** Structure of the ANFIS network applied in this study to predict the RSFC in winter.



**Figure 3.** The fuzzy membership functions (Gaussian) selected for (**a**) ice layer, (**b**) snow layer, (**c**) water thickness, and (**d**) snow height.

**Table 5.** Information about selected inputs and output.


**Table 6.** Parameters (Mean and Std values) of each Gaussian mf (cluster) for inputs.



**Table 7.** Parameters (coefficients and constant values) of each linear mf for the output.

After building the base FIS, the model was trained by ANFIS. Figure 4 shows a 3D view of the relationship between the ice layer, snow layer, and RSFC. A 3D view of the relationship between parameters helps us to extract the relationship between effective variables to predict RSFC. For instance, in Figure 4, if the value of the ice layer is less than 0.3, the value of the snow layer has no measurable impact on the created value of the RSFC. The ANFIS rule viewer (for trained data) is shown in Figure 5. Each input column displays five Gaussian membership functions for each input variable and each row shows a particular rule. Hence, each membership function has a specific rule and maps the values of each input variable to rule input values. The output column indicates how various rules can be applied to the RSFC (output variable) [26].

**Figure 4.** A 3D view of the relationship between the ice layer, snow layer and RSFC.

**Figure 5.** View of the fuzzy rule base for the designed intelligent fuzzy system for RSFC prediction (trained if-then rules).

Figures 6–10 visualize ANFIS performance for the training and test datasets. The graphs demonstrate that predicted values are close to the real values most of the time. Only a few numbers show obvious errors, which could be due to (i) these targets not being scientifically justifiable, or (ii) this method not being suitable to predict these targets. Figure 10 shows the residual plots that show the difference between real values and predicted values for both the training and testing datasets. As is clear, the value of 0 in the residual plots has the highest number and the residual plots are normally distributed, which means that ANFIS is the correct selection for our dataset. In addition, MSE and RMSE were selected as the evaluation metrics to evaluate the model performance (Table 8). RMSE values for the training and test datasets are 0.035 and 0.038, respectively. The low error of the test set indicates that the ANFIS model has a good generalization performance to effectively predict RSFC based on historical data for an unseen dataset.

**Figure 7.** Comparison between real and predicted values of the special interval for (**a**) training dataset, and (**b**) testing dataset.

**Figure 8.** (**a**) Training errors, and (**b**) testing errors by the ANFIS model with Gaussian membership functions.

**Figure 9.** (**a**) Training errors, and (**b**) testing errors for the special interval.

**Figure 10.** Residual distribution plot for (**a**) training set, and (**b**) testing set.

**Table 8.** MSE and RMSE values achieved by the ANFIS model for both training and testing datasets.


#### **4. Analytic Results**

Analysis of the variables affecting the RSFC prediction model enables us to drive future insights to accurately predict different road surface conditions at a particular time. The RSFC prediction model helps us to discover the relationship between variables and eventually leads to an improvement in decision-making procedures. We utilized the most informative scatter chart (Figure 11) to plot the RSFC, ice layer, and the amount of chemicals used for WRM to extract valuable findings.

Generally, driving conditions are divided into the following three categories: (i) normal road conditions (RSFC ≥ 0.3), (ii) bad conditions (0.15 < RSFC < 0.3), and (iii) very bad conditions (RSFC ≤ 0.15). When the friction coefficient is under 0.15, the rate of accidents can be four times higher than in conditions with a friction coefficient of 0.35–0.44 [27]. In the previous figure, the data points are shown based on different values of the friction coefficient. It is clear that, with an increase in the thickness of the ice layer on the road surface, the RSFC drops sharply. If no chemicals (e.g., salt) are used on the road surface, this leads to a drastic reduction in road safety. Therefore, the RSFC prediction model contributes to detecting these dangerous situations in advance and taking action to both prevent dangerous vehicle accidents on the road and mitigate their associated severe consequences. Moreover, when the ice layer is thinner than 0.2 mm, using a small amount of salt contributes to increasing friction on the road surface. When the ice layer is almost 0 mm, using a high quantity of chemicals (salt) on the ground leads to extra expense (including materials, trucks, and truck drivers). Chemicals (salt) are not only the main reason for rust and corrosion on vehicles, but also exacerbate the harm to road infrastructure such as concrete bridges. Furthermore, salt has negative impacts on the environment, caused by melting into rivers, lakes, and into soil, damaging vegetation.

**Figure 11.** Scatter plot for the ice layer, chemicals, and RSFC.

#### **5. Conclusions**

In this paper, ANFIS was used to design a data-driven model to accurately manage the uncertainty hidden in historical data and predict the road surface friction coefficient in winter. The model was implemented in MATLAB software using real data, measured by a road weather information system, optical sensors, and road-mounted sensors at test site E18 in Sweden in February 2019. The graphical and numerical results of ANFIS modeling demonstrate the high reliability and accuracy of the model in handling uncertainty and predicting the road surface friction coefficient. This model can be considered as a main computational component in decision support systems, to assist decisions made about the type and time of winter road maintenance in a quantitative manner. Thus, the findings of this paper can be used to develop a winter road maintenance strategy for both pre-disaster and post-disaster periods. This accurate prediction model can help decision makers to make plans in advance, which will lead to optimizing the level of service. Preparing the optimal number of trucks and materials in real-time to treat snowy and icy roads leads to improved road transportation safety (by increasing the friction between tires and road surface), traffic flow (by removing snow and ice on the road), and economic productivity (by avoiding the use of extra materials and trucks). However, ANFIS demands computational power, and its performance is significantly dependent on data quantity and quality and specifying the number of member functions for input and output variables. Hence, in future, researchers should search for alternative mathematical methods that are less dependent on data.

**Author Contributions:** Conceptualization, M.H.; methodology, M.H.; software, M.H.; validation, M.H.; formal analysis, M.H.; investigation, M.H.; resources, M.H.; data curation, J.C.; writing original draft preparation, M.H.; writing—review and editing, M.H. and G.P.P.; visualization, M.H.; supervision, G.P.P. and J.C.; project administration, G.P.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Ministry of Education and Research, Norway, grant number 470079.

**Data Availability Statement:** We have got the data from the Swedish transport administration's RWIS station at Test site E18. https://www.trafikverket.se/resa-och-trafik/forskning-och-innovation/aktuellforskning/transport-pa-vag/testsite-e18--en-vagforskningsstation/ (accessed on 1 January 2020).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **The Service Capability of Primary Health Institutions under the Hierarchical Medical System**

**Shuo Liu, Jintao Lin, Yiwen He and Junfang Xu \***

School of Public Health, Zhejiang University School of Medicine, Hangzhou 310058, China; a1147465914@163.com (S.L.); ljt181151104@163.com (J.L.); 3190104273@zju.edu.cn (Y.H.) **\*** Correspondence: xujf2019@zju.edu.cn

**Abstract:** Background: Primary health institutions (PHIs) are the foundation of the whole health system and the basic link to achieve the goal of all people enjoying primary health care. However, the service capability of primary health institutions is not under the hierarchical medical system. Method: Data were collected from the *China Health Statistics Yearbook* between 2014 and 2020. PHIs included community health centres, community health stations, and township hospitals in our study. The service capability of primary health institutions was analysed from the perspective of structure, process, and results. Structure capability was evaluated using the number of beds, number of personnel, number of health technicians, and proportion of the number of personnel in PHIs accounting for the total number of health personnel. Process capability was evaluated using the number of general practitioners. The number of outpatients and inpatients, medical income, the proportion of drug income, and the average number of patients and beds served by physicians in PHIs per day were employed to evaluate the resulting capability. Results: From 2014 to 2020, the number of community health service centres/stations increased, while the number of township health centres decreased. In the aspect of structure capability, the total number of personnel and health technicians in community health centres/stations and township hospitals both increased during 2014 and 2020. However, the increasing rate in PHIs was a little bit less than that of general medical institutions. The proportion of male health technicians in community health centres and township hospitals both decreased, while the proportion of female technicians in both increased. From 2014 to 2020, the number of beds in PHIs also increased from 138.12 <sup>×</sup> <sup>10</sup><sup>4</sup> to 164.94 <sup>×</sup> 104. However, the proportion of beds in PHIs accounting for the total number of beds in medical institutions decreased. For the resulting capability, from 2014 to 2019, the proportion of diagnosis and treatment times in PHIs decreased from 57.41% to 51.96%, although it increased in 2020. The proportion of inpatients in PHIs decreased from 20.03% to 16.11%. From 2014 to 2020, the utilisation rate of hospital beds in PHIs decreased (from 55.6% to 34% for community health centres and 60.5% to 53.6% for township hospitals). The average daily bed days of doctors in township hospitals was higher than that of doctors in community health service centres. However, the average medical cost of outpatients and the per capita medical cost of inpatients in community health service centres were higher than in township hospitals. Conclusion: In recent years, although the service capability showed an increasing trend in PHIs, the growth rate was lower than the general health institutions. The utilisation rates of PHIs, including beds and physicians, were decreased. Among PHIs, the utilisation in township hospitals was higher than in community health centres with a relatively low price. Under the hierarchical medical system and normalisation period of the COVID-19 epidemic, it is important to improve the service capability to achieve its goal of increasing PHI utilisation and decreasing secondary and tertiary hospital utilisation.

**Keywords:** primary health institution; service capability; hierarchical medical system

**Citation:** Liu, S.; Lin, J.; He, Y.; Xu, J. The Service Capability of Primary Health Institutions under the Hierarchical Medical System. *Healthcare* **2022**, *10*, 335. https:// doi.org/10.3390/healthcare10020335

Academic Editors: Amir Khorram-Manesh and Krzysztof Goniewicz

Received: 24 December 2021 Accepted: 5 February 2022 Published: 10 February 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

#### **1. Introduction**

Chronic diseases have become the biggest disease burden in China in the past 30 years. The primary goal of China's health system is to prevent and control chronic diseases, especially among the elderly. This reflects the urgency of speeding up the capacity-building of primary health institutions (PHIs) [1,2] as the foundations of the whole health system, which undertake the responsibility of providing basic health services, including public health, to people [1]. The ageing and growing burden of chronic diseases in China also highlights the need for a continuous and coordinated system based on primary health services [2]. To improve the health service level of the hierarchical health system, health resources should be effectively provided to the primary institutions, so as to promote the optimisation and adjustment of the structure, function, and layout of the health system as a whole.

The importance of primary healthcare has also been highlighted after the pandemic of COVID-19 [3]. For example, more than 4 million personnel from PHIs across the country participated in preliminary screening, diagnosis, and referral, which has greatly alleviated the pressure faced by general hospitals [4]. In addition, the importance of PHIs is not only to be reflected in the public health of disease prevention, but also in the diagnosis and treatment of some common diseases, especially during the pandemic. Currently, during the epidemic normalisation period, all the patients with a fever can be found in time through the PHIs. The fever patients can be transferred to the fever clinics at the higher-level hospitals to receive timely and effective treatment, which has played a positive role in the effective control of COVID-19. Although the application of tiered medical services in the prevention and control of sudden epidemic situations is quite effective, some problems have also emerged, such as the capability of PHIs in China.

Continuous monitoring and health management of COVID-19 patients after they are discharged from the hospital are carried out by PHIs. The COVID-19 epidemic is a huge test for the construction of medical alliances encouraged by China in recent years, which fully exposes the weak service level of PHIs. Since the outbreak of the epidemic, a considerable number of PHIs have been confronted with problems such as insufficient medical supplies, insufficient manpower and skills, and loopholes in public information systems. They are basically not equipped with screening and isolation capabilities, making it difficult to implement screening and classified isolation for community residents.

The Chinese government has recently issued a series of policies to improve primary health care. On 4 December 2020, the National Health Commission of China issued the notice on deepening the "Internet plus Health", and encouraged the development of the information system for PHIs [5]. On 24 May 2021, the State Council issued the 2021 key tasks for deepening the reform of the health system. One of the tasks is to accelerate the improvement of the basic infrastructure of community health centres and township hospitals [4]. It can be seen that the hierarchical medical system attaches unprecedented importance to the construction of the service capability of PHIs. At the same time, the implementation of a two-way referral policy, which requires the first diagnosis at PHIs, provides another chance to improve primary health care. The service capability of PHIs is directly related to the construction of the hierarchical health system and universal health coverage. Moreover, it also influences the social satisfaction of residents as a result of it often being difficult and expensive to see a doctor in China [6].

However, the current status of PHIs' service capability or the impacts of hierarchical medical system construction on PHIs' capability are unclear; for example, whether the capability of PHIs is increasing and in line with the goals of the hierarchical medical system. Scholars have performed a lot of research on the improvement paths of the medical service capacity of PHIs in China. However, most studies used one place or several indicators; the analysis of the change of the medical service capacity of PHIs from a comprehensive and national perspective is limited. To find the problems existing in the service of PHIs in China and provide evidence for improving the capability of primary health care, we aim to analyse the capability of PHIs from 2014 to 2020 from structural, process, and result perspectives.

#### **2. Methods**

#### *2.1. Data Source*

The related data of PHIs were collected from the *China Health Statistics Yearbook* [7] from 2014 to 2020, respectively, which included number of PHIs, personnel including health technicians in PHIs, number of health technicians in PHIs, beds and equipment above 10,000 yuan in medical institutions and PHIs, general practitioners in China, number of diagnoses and treatment times in PHIs, utilisation rate of hospital beds in PHIs, income of PHIs averaged by number of doctors' visits per day in PHIs, average daily bed days of doctors in PHIs, and medical expenses of patients.

#### *2.2. Measurement of Service Capability*

In the study, PHIs included community health centres, community health stations, and township hospitals. The service capability of PHIs was divided into structural service capability, process service capability, and result service capability. The indexes measuring structural service capability included the number of health technicians in China, the number of primary health technicians, the proportion of primary health technicians (the number of primary health technicians in PHIs to the number of health technicians in total medical institutions), the number of beds in PHIs, the proportion of beds in PHIs (the number of beds in PHIs to the number of beds in total medical institutions), and the utilisation rate of beds in PHIs. Number of general practitioners was used to evaluate the process service capability of PHIs. The indexes of result service capability contained the number of outpatients and inpatients in PHIs, medical income of PHIs, the proportion of drug income (the income of drugs in PHIs to the total drug income of total medical institutions), and the average number of patients and beds served by physicians in PHIs per day.

#### *2.3. Statistical Analysis*

Descriptive statistical analysis (i.e., frequency and percentage) was used to analyse the structure, process, and outcome service capability indicators of PHIs from 2014 to 2020. The trend and comparative analysis of the service capability of PHIs from 2014 to 2020 were also conducted. All data analysis was based on the statistical software SPSS 23.0 (IBM, Armonk, NY, USA). Variables with *p* < 0.05 were considered as statistically significant.

#### **3. Results**

#### *3.1. Structure Service Capability in PHIs from 2014 to 2020*

The number of community health service centres/stations increased from 34,238 in 2014 to 35,365 in 2020, while the number of township health centres decreased from 36,902 in 2014 to 35,762 in 2020 (Figure 1).

**Figure 1.** The number of PHIs in China from 2014 to 2020.

From 2014 to 2020, the total number of personnel and the number of health technicians in community health centres/stations both increased (Figure 2). The proportion of health technicians accounting for the total number of personnel in community health centres/stations remained almost unchanged before 2020 and increased slightly in 2020 (86.12%) compared to 2014 (85.42%). Similarly, the total number of personnel and the number of health technicians in township hospitals also increased, and the proportion of health technicians accounting for the total number of personnel in township hospitals remained almost unchanged before 2020, and increased slightly to 85.55% in 2020 compared to 84.45% in 2014.

**Figure 2.** Personnel including health technicians in PHIs.

From Figure 3, we can see that the number of health technicians younger than 45 years old was reduced, and the proportion of male health technicians in PHIs, including community health centres and township hospitals, also showed a decreasing trend, while the proportion of females increased. Moreover, most health technicians (~60%) had an education level lower than undergraduate.

Although the number of beds in PHIs also increased to 164.94 × 104 in 2020 from 138.12 × <sup>10</sup><sup>4</sup> in 2014, the proportion of beds in PHIs accounting for the total number of beds in medical institutions decreased from 20.92% in 2014 to 18.12% in 2020. The equipment valued above 10,000 RMB showed a similar trend with beds in the number and proportion (Figure 4).

**Figure 4.** Beds and equipment above 10,000 yuan in medical institutions and PHIs.

#### *3.2. Process Service Capability of PHIs*

The number of general practitioners in China increased from 172,597 in 2014 to 308,740 in 2018 (Figure 5), as did the number of general practitioners per 10,000 people (1.26 in 2014 to 2.22 in 2018).

**Figure 5.** General practitioners in China from 2014 to 2018.

#### *3.3. Result Service Capability of PHIs*

From 2014 to 2019 (Figure 6), the number of diagnoses and treatment times by total medical institutions increased from 760,186.6 (10,000 person-times) to 872,000 (10,000 persontimes), but it decreased to 774,000 (10,000 person-times) in 2020. For PHIs, the number of diagnoses and treatment declined from 453,087.1 (10,000 person-times) to 412,000 (10,000 person-times). Among PHIs, the number of diagnoses and treatment times in township hospitals was much higher than that of community health centres.

**Figure 6.** Number of diagnoses and treatments in PHIs (10,000 person-times).

From 2016 to 2018, the income of PHIs increased from 48,293,753 (10,000 yuan) to 61,246,366 (10,000 yuan), and the proportion of the income of PHIs accounting for the total income in medical institutions increased slightly from 14.56% to 14.9%. Among PHIs, the income in township hospitals was higher than that of community health centres. Moreover, the medical income in PHIs was much higher than government subsidies (Figure 7).

Figure 8 shows that the utilisation of beds in township hospitals was also higher than that of community health centres. However, both showed a decreasing trend during 2014 and 2020. The bed days per doctor's service also had a similar trend. The medical expenses per patient in PHIs were much lower than those of general medical institutions, and compared with community health centres, they were lower in township hospitals. However, the medical expenses per patient in PHIs increased in the last six years.

**Figure 8.** Beds utilisation, days served by doctors, and medical expenses of patients in PHIs.

#### **4. Discussion**

Since the implementation of the new medical reform, the service capability of PHIs has been improved, especially by the structure service capability. In the number of PHIs, community health service centres increased, while the number of township hospitals

decreased. This is related to the fast urbanisation, which means many rural areas turned to urban areas and rural populations migrated to cities in China in recent years [8]. From the point of view of the structural service capability of the PHIs, the structural service capability of the PHIs in China has been improved as a whole. From 2015 to 2020, the number of health technicians in PHIs gradually increased, but the proportion of health technicians remained almost unchanged. This is because the number of total staff in PHIs also increased. In addition, from 2015 to 2018, the health technicians in community health service centres mainly had junior college degrees and undergraduate degrees, and the health technicians in township health centres mainly had secondary school degrees and junior college degrees, while there were very few with graduate degrees, suggesting that the construction of the talent teams in PHIs urgently needs to be improved [9]. From 2014 to 2020, the number of beds in PHIs also increased year by year, but the proportion of beds in PHIs decreased year by year. Moreover, the proportion of equipment above 10,000 RMB in PHIs remained almost unchanged. The number of health technicians younger than 35 years old was reduced, which may be due to the number of young people choosing to be health technicians reducing gradually. The proportion of male health technicians in PHIs also showed a decreasing trend, while the proportion of females increased. This may be related to safe working practices and low salaries in primary health facilities. Such characteristics make women who need more care for their families choose this job. These results indicated that the government should pay more attention to the resource allocation of PHIs. Moreover, reviewing the course of the COVID-19 epidemic, we can also find that the improvement of PHIs is the "most economical and effective" measure to control infectious diseases. The capability of PHIs is also the key to realising tiered diagnosis and treatment. To sum up, the structural service capability of PHIs in China has been improved as a whole, but there are still deficiencies that need to be improved.

As the gatekeeper of primary medical care, the number of general practitioners in China increased. The first consultation system of general practitioners is the key to realising the function of the gatekeeper [10]. The establishment of the first consultation system of general practitioners can give full play to the core role of rational allocation and effective utilisation of health resources [11]. General practitioners guide patients to seek medical treatment in a scientific and orderly manner and on-demand [12]. This also showed that under the hierarchical health system, primary healthcare and its capabilities were gradually improved. However, due to the lack of data, it is temporarily impossible to analyse the contract services of general practitioners in PHIs.

From the point of result service capability, the progress of the result service capability of the PHIs in China was not strong, and the utilisation of diagnoses and treatment at the PHIs level was low, although evidence has shown that 80% of diseases can receive effective treatment in the PHIs. From 2014 to 2019, the number of diagnoses and treatment times in PHIs almost remained the same and decreased in 2020. In addition, from 2019 to 2020, the number of diagnoses and treatment times by total medical institutions decreased, too. This has to do with the fact that COVID-19 has kept people indoors [13]. In addition, the proportion of diagnoses and treatment times in PHIs accounting for the total also decreased. The low utilisation of PHIs may also affect whether the COVID-19 epidemic can be effectively controlled. The main reason for the low utilisation may be the relatively low capacity in PHIs. In addition, the development of PHIs still lags behind the development of general hospitals, and the relatively weak situation of PHIs has not been substantially improved, which has been clearly shown in our results regarding service capability.

Among PHIs, the number of diagnoses and treatment times in township hospitals was much higher than that of community health centres. One explanation is that people in rural areas are more likely to choose township hospitals over large urban hospitals due to traffic constraints [6]. What is more, the average daily bed days of doctors in PHIs also did not increase. These indicated that patients tended to seek treatment in the general hospitals rather than PHIs. The government financial subsidy income and medical income both increased year by year, which showed that the new medical reform attaches enough importance to the financial support of PHIs and has achieved initial results [14]. Although the income of PHIs in China has improved, in order to achieve long-term development, PHIs need to make good use of this income, improve service capability, and create more value [15]. All in all, the outcome service capability of PHIs in China has been improved on the whole, but the effect is weak, and the reform efforts need to be further strengthened [16–19]. On the one hand, they should promote the effective integration of medical resources and improve the sharing system of medical resources in the region to strengthen the ability of PHIs. On the other hand, they should improve the helper mechanism of an integrated health care system by encouraging hospital professionals working in the PHIs regularly.

This study was subjected to some limitations. First, some aspects may benefit a lot from a qualitative study rather than quantitative research. Second, considering the significant difference in geography and economic levels, the service capability of PHIs are different within China, but this study lacks regional analysis. Third, due to the lack of some data, the process service capability cannot be analysed in depth. Moreover, future studies may benefit from a deep statistics analysis.

#### **5. Conclusions**

Under the hierarchical health system and normalisation period of the COVID-19 epidemic, to improve the capability of primary health services, many measures have been issued, including PHIs' functional orientation, talent team construction and financial compensation mechanism, structural service capability, process service ability, and the result service ability. These were improved, but on the whole, failed to achieve the expected effect, and some indicators even showed a downward trend, showing the weakening of medical service capabilities and the shrinking of medical service scope. This may significantly affect the carrying out of high-quality and effective health systems, as well as the emergency management of infectious diseases.

**Author Contributions:** All authors were responsible for the structure of this paper. J.L. drafted the paper and Y.H. and S.L. contributed the data collection and literature review. J.X. contributed to the study's conception and design, interpretation of the data, and critical revisions of the paper. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the Zhejiang Soft Science Program (No. 2021C35015).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** All of the main data have been included in the results. Additional materials with details may be obtained from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Effect of the Strategic Thinking, Problem Solving Skills, and Grit on the Disaster Triage Ability of Emergency Room Nurses**

**Jina Yang <sup>1</sup> and Kon Hee Kim 2,\***


**\*** Correspondence: konhee@ewha.ac.kr; Tel.: +82-2-3277-4489

**Abstract:** In this descriptive study, we aimed to identify factors related to emergency room nurses' disaster triage ability. A total of 166 nurses who worked for emergency departments of general hospitals completed a structured questionnaire consisting of the Disaster Triage Ability Scale (DTAS), the Strategic Thinking Scale (STS), the Problem-Solving Inventory (PSI), and the Original Grit Scale (Grit-O). The data were analyzed using SPSS/WIN 25.0 by means of descriptive statistics, *t*-test, one-way ANOVA, the Scheffé post hoc test, Pearson's correlation coefficients, and stepwise multiple regression. Participants' DTAS averaged 14.03 ± 4.28 (Range 0–20) and showed a statistically significant difference according to their experience of triage education (t = 2.26, *p* = 0.022) as a disaster triage-related attribute. There were significant correlations among DTAS and confidence in the PSI (r = 0.30, *p* < 0.001), the approach-avoidance style in the PSI (r = −0.28, *p* < 0.001), and futurism in the STS (r = 0.19, *p* = 0.019). The strongest predictor was confidence in the PSI; in addition, 14.1% of the DTAS was explained by confidence in the PSI, approach-avoidance in the PSI, and futurism in the STS. Emergency room nurses who received triage education showed a higher level of the DTAS and their DTAS could be explained by problem-solving skills and strategic thinking. Therefore, it is necessary to develop and implement triage education programs integrated with stress management to improve the approach-avoidance style to ensure better problem-solving skills and to utilize various training methods to enhance confidence to improve problem-solving skills and futurism as part of strategic thinking.

**Keywords:** thinking; problem-solving; grit; triage; nurse

#### **1. Introduction**

Disasters are no longer uncommon. In the case of a disaster with multiple casualties, severity classification is essential for efficient use of limited manpower and resources [1]. Nurses are mainly responsible for the triage role in the clinical setting. Rapid information collection and accurate decision-making—that is, clinical reasoning ability—are essential for severity classification. In order to acquire this ability, continuous and systematic education is needed, and for the development and application of severity classification education programs, it is necessary to investigate the degree of nurses' severity classification abilities and their influencing factors.

Understanding and implementing severity classification in disasters with mass-casualty incidents is vital in order to disperse any limited resources among victims with the highest possibility of survival. Recent previous mass-casualty incident studies have found that improper first aid and victim transfer distribution results in re-transfer due to inadequate severity classifications at the mass-casualty incident site [2,3]. Therefore, it is essential to perform proper severity classification in mass-casualty incident situations because it directly affects the decline of resources and the health of casualties.

The individual who performs a severity classification at the disaster site must be a competent medical professional who can identify the incident site, allocate resources, set

**Citation:** Yang, J.; Kim, K.H. Effect of the Strategic Thinking, Problem Solving Skills, and Grit on the Disaster Triage Ability of Emergency Room Nurses. *IJERPH* **2022**, *19*, 987. https://doi.org/10.3390/ ijerph19020987

Academic Editors: Amir Khorram-Manesh and Krzysztof Goniewicz

Received: 22 December 2021 Accepted: 13 January 2022 Published: 16 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

priorities, and transfer the victims according to patient severity [4]. Emergency room triage nurses (ERTNs) are exemplar healthcare professionals who can perform such tasks [5]. Research has revealed that more than 90% of the medical professionals who perform severity classification using the Korea Triage and Acuity Scale (KTAS) are nurses. Taken together, these data suggest that ERTNs are the ideal medical professionals to make the most appropriate choices according to victim health problems and the amount of resources available in disastrous situations.

Severity classification requires strategic thinking, which involves deciding upon new and appropriate actions based on existing methods and facts learned from specific situations in order to achieve the best goal in a given situation [6]. Strategic thinking provides indisputable facts through formal analysis and refers to making a choice among different solutions that systematically minimizes the risk factors when damage occurs to an organization [7]. One study revealed that classification strategies affect severity classifications and, subsequently, the health outcomes of patients in disastrous pediatric events [8]. Therefore, strategic thinking is required in order to determine the method and range of severity classifications using the type and disaster scales and assessing the health state of casualties.

Severity classification requires an understanding of the overall situation of a disaster, resource availability, the number and condition of victims, the classification of skills based on strategic and critical thinking, and professional medical knowledge, which must be used in a timely manner. Problem-solving skills in nursing refer to the ability to resolve health issues promptly through critical thinking, based on knowledge and experience [9,10]. Problem-solving skills require self-confidence, approach-avoidance style, and control. Confidence has a positive effect on the problem-solving process and outcome. Approachavoidance style refers to the approach style of working hard to avoid bad things, and it is similar to coping with stress. In the process of solving problems, control over one's consciousness and behavior is required [11]. In severity classifications, problem solving skills are essential and affect accuracy through strategic and critical thinking. A previous study has shown that problem-solving skills improve the accuracy of severity classifications [12].

In a disastrous situation that is full of uncertainty, personal characteristics such as passion, persistency, and courage in spite of failure are required in those who perform severity classification. Grit refers to the characteristic of courage or determination that is required for successful goal achievement [13]. It does not mean simply passion and perseverance, but also includes the courage and persistence necessary to do the job in spite of failures without being discouraged [14]. Previous study results have shown high levels of stress in ERTNs who performed severity classification in mass-casualty incident simulation training. Therefore, grit is necessary to perform severity classification accurately in a stressful disaster situation.

Disaster triage ability (DTA) includes making the most appropriate choice for the best outcome according to the health issues of the victims and limited resources, and demonstrating courage, passion, and persistence in order to practice successful leadership. Furthermore, severity classification results influence the disaster cycle in relation to recovery; therefore, there is a need for ERTNs to have DTA that is based on specialized knowledge and skills in emergency situations. In this study, we investigated strategic thinking, problem-solving skills, grit, and DTA in ERTNs. Additionally, we examined these factors' relationships and influences on DTA, and aimed to provide a data-based framework for the development of severity classification training.

#### **2. Materials and Methods**

#### *2.1. Design*

This study was a descriptive study that sought to demonstrate the effects of strategic thinking, problem solving skills, and grit on DTA.

#### *2.2. Participants*

The participants were chosen from ERTNs in general hospitals located in metropolitan areas who had a minimum of 1 year of experience, who could efficiently and promptly classify patients based on emergency medical knowledge [15,16]. The sample size was calculated using G\*Power 3.1.9.4 under the conditions of a significance level (α) of 0.05 and power (1 − β) of 0.80 for multiple regression analysis [17]. According to the Cohen criterion, the medium effect size (f2) and predictors were set to 0.15 and 22, respectively, to calculate the number of samples required, which was 163. Considering the dropout rate, 195 questionnaires were distributed and 86.6% (169) copies were collected [18]. Three insufficient responses were excluded. Consequently, 166 responses from ERTNs were used in this study.

#### *2.3. Measurements*

#### 2.3.1. General and Severity Classification-Related Characteristics

A questionnaire assessing the general (gender, year of birth, education level, clinical experience, and ER experience) and severity-classification-related (disaster education participation, disaster triage education participation, disaster triage experience, KTAS qualification) characteristics were used.

#### 2.3.2. Strategic Thinking

The Strategic Thinking Scale (STS), which was developed by Salavati et al. [7], was translated and used in this study, with the authors' permission. Translation was conducted according to the WHO translation guidelines [19], in the following sequence: forward translation; expert panel back translation; pre-testing and cognitive interviewing; and the final version. Briefly, a bilingual United States registered nurse with over ten years of ER experience conducted the forward translation. The tool translated into Korean was then back-translated by a professional translator with a PhD in translation (Korean-English). To determine culturally different expressions and the degree of correspondence between the original tool and the translated version, a panel supervised the document. This panel included a nurse with at least ten years of ER clinical experience, a professional translator with a PhD in Korean-English translation, and an English native speaker. Their opinion indicated that there were no cultural differences; however, some word changes were required and the tool was modified accordingly. Next, pre-testing was conducted on five nurses with at least one year of ER experience. The final tool was approved through cognitive interviewing, which confirmed that the content was appropriate and did not contain any issues in understanding and responding. This five-point Likert scale had four sub-areas with 26 items: system thinking, conceptual thinking, futurism, and intelligent opportunism. Higher scores indicated a higher strategic thinking ability. Cronbach's α was 0.85 when the tool was developed [7], and was 0.71 in this study.

#### 2.3.3. Problem-Solving Skills

Problem-solving skills (PSS) was measured using the Problem Solving Inventory (PSI) developed by Heppner and Petersen [20], translated by Jeon [21]. The tool, a five-point Likert scale, consisted of 21 items with three sub-areas: confidence, approach-avoidance style, and person control. The higher the score, the higher the PSS. Cronbach's α was 0.89 when the tool was developed [20], 0.90 in the research of Jeon [21], and 0.86 in this study.

#### 2.3.4. Grit

The Original Grit Scale (Grit-O), which was developed by Duckworth et al. [14] and translated by Lee and Son [22], was used in this study. This five-point Likert scale consists of two sub-areas with twelve items: consistency of interests and perseverance of efforts. Higher scores indicated higher grit. Cronbach's α was 0.81 when the tool was developed [14], 0.79 in the research of Lee and Son [22], and 0.86 in this study.

#### 2.3.5. Disaster Triage Ability

With the authors' permission, 20 items were extracted, revised, and supplemented from the Disaster Triage Education in Korea Disaster Life Support—Basic (KDLS-Basic) scale, which was developed by the Central Emergency Medical Center [23]. The content validity was verified by five experts (two emergency medicine specialists, two emergency structural science professors, and one ER specialist nurse) using the Content Validity Index (CVI) with a four-point Likert scale. The questions were adopted after confirming the minimal CVI of all 20 questions (0.99). Correct answers counted for one point, whereas incorrect answers obtained zero points. Higher scores indicated higher DTA, with a perfect score of 20.

#### *2.4. Data Collection*

During May 2019, following the Institutional Review Board (IRB) approval, we explained the purpose and method of the study to the head of the institution, the nursing department, and emergency unit for approval. Next, consent forms and structured questionnaires were distributed to voluntary participants, who were informed of the study's purpose and method. Informed consent forms and questionnaires were collected in separate collection boxes and small souvenirs were offered to the participants.

#### *2.5. Data Analysis*

Data were analyzed with the SPSS/WIN 25.0 program, using descriptive statistics, the independent *t*-test, one-way ANOVA, and Scheffé post-hoc analysis. The correlations between ST, PSS, Grit, and DTA were analyzed using Pearson's correlation coefficients. The effects of ST, PSS, and Grit on DTA were analyzed using stepwise multiple linear regression analysis.

#### *2.6. Ethical Consideration*

This study was conducted following approval by the IRB (MJH 2019-04-010-001). To acquire permission for this survey, the author explained the purpose and methods of our study to the head of each institution and the research manager of the nursing department in all participating institutions. The purpose and methods of the study were explained to the voluntary study participants before the informed consent forms and questionnaires were distributed. It was explained to participants that they can withdraw their participation in the study at any time with no disadvantages. To protect their privacy, the study participants were not specified; the consent forms and questionnaire were collected autonomously. In addition, these collections occurred simultaneously. The completed questionnaires were stored in a locked personal drawer and will be discarded immediately upon completion of the study. Furthermore, the computerized data is encrypted and stored in a personal computer accessible only to the primary investigator and will be deleted after three years of storage period.

#### **3. Results**

#### *3.1. General and Severity Classification-Related Characteristics*

The mean age of ERTNs was 29.12 ± 4.81, and '36+' showed the highest proportion, at 13.9%. There were 89.2% females, and 85.5% possessed a Bachelor's degree or less. The mean clinical experience was 70.22 ± 66.33 months, and over half (58.4%) had '<5 years' of experience. The mean experience in the ER was 63.66 ± 56.15 months, and 59.6% had worked '<60 months' in ER. There were 64.5% and 51.8% of participants who had undertaken disaster and disaster triage education, respectively. Finally, 38.6% had severity classification experience, and 60.8% possessed a KTAS qualification (Table 1).


**Table 1.** Participants' general characteristics (*N* = 166).

#### *3.2. Participants' Strategic Thinking, Problem-Solving Skills, Grit, and Disaster Triage Ability*

The mean ST score was 3.23 ± 0.19 points. When measuring the sub-areas, intelligent opportunism had the highest score (3.53 ± 0.35), followed by futurism (3.26 ± 0.48), conceptual thinking (3.25 ± 0.24), and system thinking (2.86 ± 0.18). The mean PSS score was 3.41 ± 0.39, and by sub-areas, approach-avoidance style 3.60 ± 0.37, confidence 3.47 ± 0.52, and person control 2.83 ± 0.74. The mean grit score was 3.02 ± 0.47, and by sub-areas, perseverance of efforts was 3.13 ± 0.48 and consistency of interests was 2.85 ± 0.60. The mean DTA score was 14.03 ± 4.28 (range, 2–19; Table 2).

**Table 2.** Strategic thinking, problem-solving skills, grit, and disaster triage ability. (*N* = 166).


#### *3.3. Association of Disaster Triage Ability with General and Severity Classification-Related Characteristics*

There was no significant difference in DTA according to the general characteristics; however, there was a significant effect of disaster triage education on DTA (t = 2.26, *p* = 0.022); nurses who participated in disaster triage training had significantly higher DTA scores (Table 1).

#### *3.4. Association between Strategic Thinking, Problem-Solving Skills, Grit, and Disaster Triage Ability*

DTA had a positive correlation with futurism (r = 0.19, *p* = 0.019), a sub-area of STS. Additionally, there was a significant positive and negative correlation with confidence (r = 0.30, *p* < 0.001) and approach-avoidance style (r = −0.28, *p* < 0.001), which are sub-areas of PSI; however, this was only to a weak degree (Table 3).

**Table 3.** Correlations among strategic thinking, problem-solving skills, grit, and disaster triage ability (*N*= 166).


<sup>1</sup> Disaster triage ability. <sup>2</sup> Strategic thinking. <sup>3</sup> Conceptual thinking. <sup>4</sup> System thinking. <sup>5</sup> Futurism. <sup>6</sup> Intelligent opportunism. <sup>7</sup> Problem-solving skills. <sup>8</sup> Confidence. <sup>9</sup> Approach-avoidance style. <sup>10</sup> Person control. <sup>11</sup> Consistency of interests. <sup>12</sup> Perseverance of efforts.

#### *3.5. Factors Influencing Disaster Triage Ability*

Multiple regression analysis was conducted to identify the factors influencing DTA in ERTNs. Futurism, confidence, and approach-avoidance style were used as independent variables. Multicollinearity between tolerance limits and independent variables, as well as mutual independence between residuals, were identified. Tolerance was 0.55–0.88, the variance inflation factor (VIF) was <10 (range, 1.14–1.81). This confirmed the low correlation between independent variables without multicollinearity. The Durbin–Watson coefficient was closed to two (range, 1.91–2.01) [20], confirming the independence of the residuals.

The factors influencing DTA were identified as the approach-avoidance style, a subarea of PSI (β = −0.27, *p* < 0.001); confidence, a sub-area of PSI (β = 0.22, *p* = 0.004); and futurism, a sub-area of STS (β = 0.17, *p* = 0.030). These variables explained 14.1% of the DTA. Among these three factors, approach-avoidance style was the most significant factor. Consequently, the DTA of ERTNs was considered higher when individuals reported lower approach-avoidance styles and higher confidence in PSI, and higher futurism in STS (Table 4).


**Table 4.** Influencing factors on disaster triage ability among emergency room nurses (*N* = 166).

<sup>1</sup> VIF: variance inflation factor.

#### **4. Discussion**

Interest in disaster medical care has increased and related activities due to the occurrence of various disaster accidents. This demand has strengthened the need for severity classification abilities, which influence the success and failure of response results. ERTNs are ideal medical professionals to make appropriate severity classification choices because of their experiences identifying the severity of casualties according to health issues and resource availability. This descriptive survey study sought to provide a new basis for the educational intervention for improved DTA in ERTNs by identifying the degrees of strategic thinking, problem-solving skills, grit, and disaster triage ability in ERTNs. We found that the mean DTA score of ERTNs was 14.03, which can be converted to over 70.15 points in a 100-point scale. A novel tool was used to measure DTA in this study; therefore, we converted the points calculated to a 100-point scale for comparison and analysis with previous studies. The result of this study was similar to a study with Canadian ERTNs, who had a mean DTA score of 72.2 points [24]. This study used a KTAS based on the Canada Triage and Acuity Scale (CTAS) and ERTNs were assessed using similar classification tools to those in the present study. Interestingly, a previous domestic severity classification study reported different results; DTA was lower in domestic military nursing personnel (63.5 points) [25] and higher in 119 paramedics (75.7 points) [26]. This indicates that nurses who perform different occupational duties show differences in their DTA.

A previous study reported that the mean DTA scores of 119 paramedics, which included nurses, were different according to age, clinical experience, and job title [27]. In contrast, there was no difference in DTA scores associated with general characteristics in this study. However, a direct comparison was difficult, as previous studies did not investigate general characteristics [24] or did not consider their relationship to DTA, although age, clinical experience, and job title have been investigated [28]. A previous study confirmed a significant difference in DTA according to the general characteristics, such as age, clinical experience, and job title of the ERTNs [29]. Taken together, these data suggest that a followup study assessing the relationship between DTA and general characteristics is required. These differences are significant for the development of DTA educational programs.

This study confirmed the significant difference in DTA according to the presence of disaster triage education; higher DTA scores were associated with disaster triage education participation. This result is similar to a study on US ERTNs, which showed a significant DTA improvement after a video simulation education course [28], and a study on 119 paramedics and public health care emergency teams, which revealed improvements in timing and DTA after education [30]. According to findings of the National Disaster Health Medical Education [31], opportunities for severity classification education are relatively limited because there are no separately operated severity classification courses, and these skills are only included as a part of disaster-related education. Therefore, specialized disaster triage education courses and programs are required.

Our results revealed that a higher DTA score was associated with enhanced confidence in PSI, higher futurism in STS, and a lower level of the approach-avoidance style. These results were similar to those of a previous study showing that higher confidence and job performance was correlated with a higher DTA score in ERTNs in US general hospitals [28]. Confidence in PSI refers to the ability to choose and apply a solution among various options [20]. It is dependent on basic knowledge and performance frequency [31]. This suggests that education should combine theory and practice. Futurism in STS refers to the future assessment ability and is improved through practice [7]. Approach-avoidance style refers to stress-coping skills, which are dependent on stress defense mechanisms [11]. Research has indicated that enhanced DTA is associated with a well-formed approach-avoidance style, which suggests the need for an approach-avoidance style-forming programs. Furthermore, our results differ from research showing that other emergency medical professionals who use appropriate approach-avoidance style defense mechanisms are healthier in disaster situations [11]. This may be due to a limitation of the appropriate approach-avoidance style defense mechanisms in ERTNs due to constant exposure to emergency situations. Hence, the application of appropriate approach-avoidance style defense mechanism formation education in ERTNs is necessary.

The explanatory power of the factors influencing DTA in ERTNs assessed in this study (approach-avoidance style and confidence in PSI, futurism in STS) was 14.1%. Similarly, previous study results have revealed that confidence in PSI [32] and futurism in STS [33] significantly influence DTA. In contrast to our results, mass-casualty incident simulation training of ERTNs has shown that stresses associated with the approach-avoidance style alone significantly affect DTA [34]. DTA aims to distribute medical resources to casualties with higher probabilities of survival. It is responsible for evaluating patients, communicating with and between professionals; providing initial first aid; allocating medical resources; and monitoring, reassessing and managing the flow of the patient treatment [16]. A study on Swedish ERTNs analyzed the skills and influential factors relating to DTA and confirmed that confidence for PSI [32] and futurism in STS enabled the best choices in regard to future situations [33]. Furthermore, another study assessed mass-casualty incident situation training of nurses, doctors, and other healthcare workers from general hospitals in Thailand. They found that severity classification stress was significantly influenced by individual approach-avoidance style [34], which is in contrast with our study. Taken together, our study and previous data highlight that enhanced confidence in PSI, futurism in STS, and the influence of the approach-avoidance style in PSI can improve DTA.

This study confirmed that the influence of an approach-avoidance style and confidence in PSI, futurism in STS, and disaster triage education affect DTA. The approach-avoidance style in PSI is affected by stress coping skills, and confidence in PSI and futurism in STS are affected by education and practice; therefore, this study has provided data that can be used to develop further strategic thinking education programs.

There are some limitations of this study. We sought participation in a limited area; therefore, generalization is limited. Follow-up studies that use national random data are suggested. Furthermore, repeated research through objective data collection, such as measuring DTA using impartial observers, is suggested because it is difficult to exclude subjective effects with self-report-oriented data collection, as used in this study. Moreover, this study identified approach-avoidance style and confidence on PSI and futurism on STS as significant influencing factors; however, there was a low level of explanatory power. We suggest a follow-up study to elucidate the other influencing factors in DTA.

#### **5. Conclusions**

This study was conducted in order to provide data on the requirements for the development of DTA in ERTNs who play a major role in severity classification in emergency situations. This was performed by identifying the degree and influential factors on DTA in current ERTNs. Our data revealed at least an intermediate level of DTA in ERTNs, which was significantly increased with disaster triage education. Approach-avoidance style and confidence, sub-areas of PSI, and futurism, a sub area of STS, affected DTA, with an explanatory power of 14.1%.

When developing and applying an educational intervention for DTA, it is important to promote the approach-avoidance style as part of a stress coping program. The preparation and application of stress-coping programs such as mindfulness, meditation, relaxation, and so on, may be considered for ERTNs. Furthermore, the use of learning methods such as conceptual guidance, theory, and task-based learning are crucial. Education increases DTA; therefore, the continuous development and application of disaster triage education programs will be most efficient. It is also necessary to provide an opportunity to maintain and improve nurses' severity classification ability through repeated learning using virtual reality, augmented reality, and extended reality, and to verify the effectiveness of this approach. Additionally, as a follow-up study, the authors suggest an analysis of patient outcomes and medical costs according to the number of triage experiences and their success or failure. Considering the rapid nurse turnover rate due to the shortage of nursing manpower, we suggest a study to verify the effectiveness of severity classification education according to the period of clinical experience.

**Author Contributions:** Conceptualization, J.Y. and K.H.K.; methodology, J.Y. and K.H.K.; software, J.Y. and K.H.K.; validation, J.Y. and K.H.K.; formal analysis, J.Y. and K.H.K.; investigation, J.Y.; resources, J.Y. and K.H.K.; data curation, J.Y. and K.H.K.; writing—original draft preparation, J.Y. and K.H.K.; writing—review and editing, K.H.K.; visualization, J.Y. and K.H.K.; supervision, K.H.K.; project administration, K.H.K. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of Myongji Hospital (MJH 2019-04-010-001, 30 April 2019).

**Informed Consent Statement:** Informed consent was obtained from all participants involved in the study. Written informed consent was obtained from the participants to publish this paper.

**Data Availability Statement:** The data presented in this study are not publicly available due to participants' privacy.

**Acknowledgments:** The authors thank each of the participants in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


#### *Article*

## **Can a Healthcare Quality Improvement Initiative Reduce Disparity in the Treatment Delay among ST-Segment Elevation Myocardial Infarction Patients with Different Arrival Modes? Evidence from 33 General Hospitals and Their Anticipated Impact on Healthcare during Disasters and Public Health Emergencies**

**Na Li 1,2, Junxiong Ma 1,2, Shuduo Zhou 1,2, Xuejie Dong 1,2, Mailikezhati Maimaitiming 3, Yinzi Jin 1,2,\* and Zhijie Zheng 1,2**

**Citation:** Li, N.; Ma, J.; Zhou, S.; Dong, X.; Maimaitiming, M.; Jin, Y.; Zheng, Z. Can a Healthcare Quality Improvement Initiative Reduce Disparity in the Treatment Delay among ST-Segment Elevation Myocardial Infarction Patients with Different Arrival Modes? Evidence from 33 General Hospitals and Their Anticipated Impact on Healthcare during Disasters and Public Health Emergencies. *Healthcare* **2021**, *9*, 1462. https://doi.org/10.3390/ healthcare9111462

Academic Editors: Amir Khorram-Manesh and Krzysztof Goniewicz

Received: 7 September 2021 Accepted: 26 October 2021 Published: 28 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).


**Abstract:** (1) Background: Chest pain center accreditation has been associated with improved timelines of primary percutaneous coronary intervention (PCI) for ST-segment elevated myocardial infarction (STEMI). However, evidence from low- and middle-income regions was insufficient, and whether the sensitivity to improvements differs between walk-in and emergency medical service (EMS)-transported patients remained unclear. In this study, we aimed to examine the association of chest pain center accreditation status with door-to-balloon (D2B) time and the potential modification effect of arrival mode. (2) Methods: The associations were examined using generalized linear mixed models, and the effect modification of arrival mode was examined by incorporating an interaction term in the models. (3) Results: In 4186 STEMI patients, during and after accreditation were respectively associated with 65% (95% CI: 54%, 73%) and 71% (95% CI: 61%, 79%) reduced risk of D2B time being more than 90 min (using before accreditation as the reference). Decreases of 27.88 (95% CI: 19.57, 36.22) minutes and 26.55 (95% CI: 17.45, 35.70) minutes in D2B were also observed for the during and after accreditation groups, respectively. The impact of accreditation on timeline improvement was greater for EMS-transported patients than for walk-in patients. (4) Conclusions: EMS-transported patients were more sensitive to the shortened in-hospital delay associated with the initiative, which could exacerbate the existing disparity among patients with different arrival modes.

**Keywords:** chest pain center accreditation; healthcare quality improvement; door-to-balloon time; arrival mode; ST-segment elevation myocardial infarction

#### **1. Introduction**

ST-segment elevation myocardial infarction (STEMI) is the deadliest and most timesensitive acute cardiac event. Primary percutaneous coronary intervention (PCI) is the typically recommended treatment for STEMI cases. The door-to-balloon time, also referred to as the in-hospital delay, which denotes the interval from the patient's arrival at the emergency department to the first inflation of an angioplasty balloon in the occluded coronary artery, is widely used to assess the timeliness of primary PCI [1]. A door-to-balloon time of 90 min or less is given as the Class I (highest level) recommendation according to the American College of Cardiology/American Heart Association (ACC/AHA) and European Society of Cardiology (ESC) guidelines [2,3]. Despite the widespread promulgation and

endorsement of the guideline, their translation into clinical practice remains suboptimal. In China, only 32.6% of STEMI patients receive primary PCI within 90 min of arrival [4]. Moreover, there is a very pronounced gap in the door-to-balloon time between walk-in and emergency medical service (EMS)-transported STEMI patients undergoing primary PCI [5–8]. Therefore, implementation of a healthcare quality improvement initiative, ensuring that hospitals provide timely guideline-recommended clinical practice, is warranted to reduce the in-hospital delay.

A growing strand of studies indicate that accreditation of chest pain centers can facilitate the implementation of strategies to improve healthcare quality for STEMI. Accredited chest pain centers should follow the criteria according to the recommended guideline, accompanied with a number of quality improvement activities (e.g., establishing a standardized monitor system of healthcare performance, carrying out healthcare performance review and feedback). For instance, the United States and Germany have witnessed improvements in the management and clinical outcomes of STEMI patients after an extensive adoption of nationwide programs for chest pain center professional society accreditation [9–12]. Positive empirical evidence in developed countries has shown that chest pain center accreditation is associated with shortened in-hospital delay for STEMI care [10,13]. However, two key questions remained unanswered: First, does the positive effect of chest pain center accreditation on the in-hospital delay found in other settings apply to China, where the proportion of cases receiving guideline-recommended treatments remains low [4]? Second, given the large disparity in the in-hospital delay among patients with different arrival modes [5,6,8,14,15], is the association between chest pain center accreditation and the in-hospital delay different between walk-in and EMS-transported cases? If so, is chest pain center accreditation widening or narrowing the existing disparities in in-hospital delay? Furthermore, integration of prehospital and hospital care is one of the dimensions of chest pain center accreditation criteria in China. It requires that hospitals should establish a regional collaborative healthcare delivery system that integrates prehospital emergency systems and in-hospital green passages, coordination and division of labor between different hospital departments, and connections between hospitals and community healthcare centers. This also plays a key role in optimizing the capacity of public health infrastructure and/or systems to respond at times of public health emergencies and disasters. Therefore, if chest pain center accreditation is beneficial to timeliness for STEMI patients and other acute cardiac events, it would also contribute to the promptness of triage, transfer, and treatment at times of public health emergencies and disasters.

#### **2. Materials and Methods**

#### *2.1. Study Design and Population*

For this study, we utilized data from all the accredited hospitals with PCI capabilities in Beijing during January 2016 to June 2019. For our study, we recruited STEMI patients who met the following criteria:


Patients were excluded if they had an unknown mode of arrival (*N* = 15), missing hospital arrival time, or implausible door-to-balloon time, such as a negative value or time exceeding 24 h (*N* = 37).

#### *2.2. Measurement*

#### 2.2.1. Accreditation Status

Hospitals' accreditation status was authenticated by the National Health Commission of China in April 2018. Chest pain center accreditation is made available to all hospitals in Beijing, and hospitals voluntarily continue to apply for the accreditation in a staggered manner. To June 2019, there were a total of 33 hospitals with accredited chest pain centers in Beijing. Thus, not every hospital had enrolled chest pain patients for 18 months. It takes months to receive accreditation, which is based on a review of information from multiple sources, including self-assessment statements, data reports, and field surveys, jointly led by the National Health Commission of China and a specialist team. Data on individual patients were extracted from the electronic medical systems of each hospital, regarding the data elements which were selected based on the ACC/AHA clinical practice guideline [2]. The accreditation statuses of hospital were grouped as 'before accreditation' (had not applied for accreditation), 'during accreditation' (were applying for accreditation), and 'after accreditation' (had been accredited). Patients who were admitted to hospitals before, during, and after the corresponding date of accreditation were, respectively, classified into the 'before accreditation' group, the 'during accreditation' group, and the 'after accreditation' group.

#### 2.2.2. Outcomes

The primary outcome was the in-hospital delay, measured by the door-to-balloon time and the percentage of cases with door-to-balloon time of more than 90 min. Door-to-balloon time was defined as the interval from the STEMI patient's arrival at the hospital to inflation of the balloon to restore flow.

#### 2.2.3. Covariates

The EMS-transported patients were defined as patients transported to the hospital by EMS services. Walk-in patients were defined as those arriving at the hospital by self- or private transportation, taxi, public transportation, or walking to the hospital.

Patient-level covariates included age, sex (male or female), and signs and symptoms at presentation: whether the patient had sustainable chest pain (refers to the onset of chest pain that lasts more than 30 min and cannot be relieved by rest), intermittent chest pain (refers to chest pain that lasts a few minutes at a time and can be relieved by rest or elimination of the triggers), chest pain relief, abdominal pain, dyspnea, cardiogenic shock, heart failure, malignant arrhythmia, receiving prehospital cardiopulmonary resuscitation or not, heart rate (beats/min), blood pressure (mmHg), and Killip class (I to IV) [16]. Hospital-level characteristics included time of day of arrival (8:00 a.m. to 16:59 p.m., 17:00 p.m. to 11:59 p.m., 12:00 a.m. to 7:59 a.m.), weekday or off-day arrival (off-days include weekends and Chinese official holidays), hospital level, and region of hospital (urban or suburb). Hospital levels in China are divided into several levels according to the scale, facilities, and ability of hospitals: grade III A, B, and C; grade II A, B, and C; and grade I, with grade IIIA being the highest level. Hospitals of grade IIIA have high-level capacity for primary PCI, and the number of PCIs meets certain requirements, ensuring that emergency PCI operations are performed 24 h a day.

#### *2.3. Data Analysis*

The characteristics of patients and hospitals and the in-hospital delay were compared between walk-in and EMS-transported patients, with the use of the chi-square test for categorical variables and the *Kruskal–Wallis* test for continuous variables. Categorical variables are presented as counts and percentages. Quantitative variables are expressed as means ± standard deviations (SDs) or medians and interquartile ranges. *p* values of less than 0.05 were considered to indicate statistical significance. We describe the median of door-to-balloon times and the percentage of patients for whom door-to-balloon times were 90 min or more across arrival modes and accreditation status.

To account for clustering of patients within hospitals, we employed generalized linear mixed models with a random effect term for the hospital to examine the associations of arrival mode and accreditation status with in-hospital delay (Model 1). The logistic regression was performed for the percentage of cases with door-to-balloon time of more than 90 min, and the effect estimates are reported as odds ratios (ORs) and 95% CI. For the door-to-balloon time, the effect estimates were calculated from the linear regression and reported as changes in minutes. Variables included in the models were selected based on their physiological relevance and potential to be associated with outcomes. We initiated the model development with a crude model (no adjustment) and then added a range of covariates into the regression models based on previous studies in the literature [14,17–19]. All the models were adjusted for sex, age, signs and symptoms at presentation, heart rate, blood pressure and Killip class, arrival mode, time of day of arrival, day of arrival, class of hospital, region of hospital, and accreditation status. To examine the modification effect of arrival mode in the association of accreditation status with the in-hospital delay, an interaction term of arrival mode and accreditation status was incorporated into the model (Model 2). All the models were adjusted for covariates, with *p* < 0.05 considered the level of statistical significance. Hospital was added as a random effect term in the models to adjust for unobserved hospital-level factors. All statistical analyses were performed using R software.

#### **3. Results**

#### *3.1. Characteristics of Patients and Hospitals*

A total of 4186 STEMI patients undergoing PCI from 33 hospitals were enrolled in this study, including 1284 (30.7%) EMS-transported patients and 2902 (69.3%) walk-in patients. The overall median age of the patients was 60 years, and 80.5% were men. The patients admitted to hospitals before, during, and after accreditation accounted for 43.2%, 13.7, and 43.1%, respectively. A comparison of patient- and hospital-level characteristics by arrival mode is presented in Table 1. In general, compared with walk-in patients, EMS-transported patients had lower heart rate (73.5 versus 75.9 beats/min, *p* < 0.001), lower systolic blood pressure (122.7 versus 132.5 mmHg, *p* < 0.001), lower diastolic blood pressure (76 versus 81.3 mmHg, *p* < 0.001), and lower rate of Killip I status (81.3% versus 89.7%, *p* < 0.001). Regarding hospital-level characteristics, 72.2% of EMS-transported patients arrived at urban hospitals, slightly larger than the proportion of walk-in patients (67.1%, *p* < 0.001).

**Table 1.** Patient- and hospital-level characteristics of participants.



Abbreviations: EMS, emergency medical service; CPR, cardiopulmonary resuscitation; q1, the first quartile; q3, the third quartile; SD,

standard deviation. Notes: Sustainable chest pain refers to the onset of chest pain that lasts more than 30 min and cannot be relieved by rest. Intermittent chest pain refers to chest pain that lasts a few minutes at a time and can be relieved by rest or elimination of the triggers.

> The in-hospital delay also varied in the two groups of patients (Table 2) and by different status (Figure S1 and Table S1). The median door-to-balloon time (70 versus 85 min, *p* < 0.001) and the percentage of cases with door-to-balloon time more than 90 min (26.7% versus 43.9%, *p* < 0.001) in EMS-transported patients were lower than those in walk-in patients.


**Table 2.** Door-to-balloon time in patients with different arrival modes.

Abbreviations: EMS, emergency medical service; q1, the first quartile; q3, the third quartile.

#### *3.2. Association between Accreditation Status and In-Hospital Delay*

Figure 1 shows the results of generalized linear mixed models of the likelihood of door-to-balloon time being more than 90 min. According to the full adjustment model, compared with the 'before accreditation' group, the risk of the door-to-balloon time being more than 90 min was significantly lower in both the 'during accreditation' group (OR: 0.35, 95% CI: 0.27, 0.46) and the 'after accreditation' group (OR: 0.29, 95% CI: 0.21, 0.39). Arrival by EMS was associated with a lower risk of the door-to-balloon time being more than 90 min, compared with arrival by self (OR: 0.49, 95% CI: 0.41, 0.58). The results generated by crude models are presented in Table S2.

Figure 2 shows the results of generalized linear mixed models of the door-to-balloon time. Compared with the 'before accreditation' group, we observed significant decreases of 27.88 (95% CI: −36.22, −19.57) minutes and 26.55 (95% CI: −35.70, −17.45) minutes for the 'during accreditation' group and 'after accreditation' group, respectively. Those transported by EMS exhibited a 21.62 (95% CI: −27.27, −16.11) minute decrease in door-to-balloon time

compared with walk-in patients. The results generated by crude models are presented in Table S3.

**Figure 1.** Associations of in-hospital delay (door-to-balloon time of >90 min) with accreditation status and arrival mode. Abbreviations: 95% CI, 95% confidence interval; EMS, emergency medical service. Notes: Both Models 1 and 2 were adjusted for sex, age, signs and symptoms at presentation, heart rate, blood pressure and Killip class, arrival mode, time of day of arrival, day of arrival, class of hospital, region of hospital, and accreditation status. Model 1 did not contain an interaction term. Model 2 included an interaction term of arrival mode and accreditation status. The OR value of the interaction term indicates that the decreased likelihood of door-to-balloon time being more than 90 min associated with accreditation status for EMS-transported patients was (OR − 1) ∗ 100%-more than that for walk-in patients.

**Figure 2.** Associations of door-to-balloon time with accreditation status and arrival mode. Abbreviations: 95% CI, 95% confidence interval; EMS, emergency medical service. Notes: Both Model 1 and 2 were adjusted for sex, age, signs and symptoms at presentation, heart rate, blood pressure and Killip class, arrival mode, time of day of arrival, day of arrival, class of hospital, region of hospital, and accreditation status. Model 1 did not contain an interaction term. Model 2 included an interaction term of arrival mode and accreditation status. The β value of the interaction term indicates that the decrease in door-to-balloon time associated with accreditation status for EMS-transported patients was β minutes more than that for walk-in patients.

#### *3.3. Association between Accreditation Status and Disparity of In-Hospital Delay across Arrival Modes*

After adding the interaction term between accreditation status and arrival by EMS, the negative associations between 'after accreditation' status and in-hospital delay remained statistically significant and had larger coefficient sizes in the likelihood of door-to-balloon time being more than 90 min (OR: 0.25, 95% CI: 0.18, 0.34) and the door-to-balloon time (β: −32.90, 95% CI: −42.98, −22.90). In terms of differences in arrival mode, the coefficient sizes of arrival by EMS were also larger in the in-hospital delay: the likelihood of door-toballoon time being more than 90 min (OR: 0.42, 95% CI: 0.33, 0.54) and the door-to-balloon time (β: −30.40, 95% CI: −38.77, −22.22).

The OR of the interaction term of arrival by EMS and 'during accreditation' was statistically insignificant, suggesting that arrival by EMS did not modify the effect of accreditation process on in-hospital delay. The OR of the interaction term of arrival by EMS and 'after accreditation' was 1.59 (95% CI: 1.10, 2.30), indicating that completed accreditation widened the disparity in the in-hospital delay between walk-in and EMStransported patients. The impact of completed accreditation on the likelihood of door-toballoon time being more than 90 min for EMS-transported patients was 59% greater than that for walk-in patients (Figure 1). Similar patterns were also found for the door-to-balloon time. A β value of 18.01 was found for the interaction term of arrival by EMS and 'after accreditation', suggesting that the reduction in the door-to-balloon time associated with completed accreditation for EMS-transported patients was 18.01 min more than that for walk-in patients (Figure 2).

#### **4. Discussion**

To the best of our knowledge, this was the first study to identify the modifying role of arrival mode in associations of chest pain center accreditation with in-hospital delay. Our results reflect an overall significant reduction in the in-hospital delay among STEMI patients after hospitals were accredited; however, the improvement was inconsistent between walk-in and EMS-transported patients. Our findings suggest that the healthcare quality improvement initiative may widen the disparity in treatment delay for patients with different arrival modes, providing implications for the optimization of implementation strategies for the continuous quality improvement of healthcare for acute chest pain.

Our results showed that compared with 'before accreditation', both 'during accreditation' and 'after accreditation' statuses were associated with lower prevalence of in-hospital delay among STEMI patients undergoing primary PCI. This finding is generally consistent with studies conducted in other regions, which indicated that chest pain center accreditation was associated with improved processes and outcomes for patients with STEMI [9,12,20]. Generally, chest pain center accreditation is a hospital-based, multifaceted, continuous quality improvement initiative from a multidisciplinary approach; it can be an efficient way to improve the in-hospital process and is of great significance to shortening the treatment time for STEMI patients. Furthermore, the negative associations of completed accreditation ('after accreditation' status) with in-hospital delay were more pronounced among EMS-transported patients than among walk-in patients.

There are several potential explanations for the observed disparity in sensitivity to chest pain center accreditation efforts. First, establishment of a regional collaborative healthcare network from a multiagency approach was emphasized in the current practice for achieving chest pain center accreditation criteria. Delivery of EMS always occurs across multiple sectors, including emergency departments, centers for prehospital care, ambulance stations, and day care or primary healthcare centers, and it requires at least two different services, with each service provided by different settings. Care coordination is critical for the delivery of EMS to ensure that healthcare professionals interact with each other to provide timely and efficient healthcare. A large number of studies have also shown that the transmission of prehospital electrocardiogram and prehospital diagnosis is the primary basis for a hospital to decide whether to bypass the emergency department and

cardiac care unit, which can reduce the in-hospital delay for EMS-transported STEMI patients [21–23]. Pre-hospital ECGs going directly to the hospital by bypassing the emergency department and even coronary care unit is a general practice for achieving this objective. The EMS-transported patients also had greater and significantly faster receipt of initial reperfusion therapies [24]. Second, the condition of patients who are transported to hospital by ambulance is generally considered to be more urgent and more severe. They are given more attention and a higher medical priority when arriving at emergency departments. As soon as the ambulance arrives, prompt diagnosis, triage, and treatment are provided by healthcare professionals who are on stand-by in advance. Third, for walk-in patients, they have to undergo the normal medical procedures after arriving at the hospital, such as consulting, registering, paying, and even waiting for treatment. They cannot get the rapid and priority healthcare that patients transported by EMS can have. As a result, the time interval between their arrival at the gate of the hospital and initiation of reperfusion is extended.

In addition to the integration of prehospital and hospital care, implementation measurements required by the chest pain center accreditation criteria could also provide a plausible explanation for the mitigation of in-hospital delay among STEMI patients requiring primary PCI. Accredited hospitals continuously report data on individual patients for quality monitoring and assessment. The indicators for measuring clinical performance quarterly and annually are reported, and a ranking is calculated based on the percentile of each indicator and a weighted composite score. In terms of auditing and feedback regarding clinical performance, an improvement in adherence to the guideline recommendations is facilitated through monthly and quarterly hospital-specific performance feedback reports. The hospital-specific data are compared against a variety of internal and external benchmarks, including the temporal trend in performance and comparison points to regional or national performance thresholds, led by the National Health Commission of China. A series of regular meetings and case management and case study meetings are carried out at least once every quarter to share 'best practice' clinical support tools in hospitals. Regarding educational outreach to clinicians, routine educational programs are organized, and the contents of training include rules and guidelines for chest pain center construction, clinical skills for the diagnosis and treatment of STEMI cases, and standardization and guidelines for real-time data reporting. These dimensions of implementation measurements required by the chest pain center accreditation criteria could also benefit the development of public health capacity and capability to respond to public health emergencies by saving resources for triage, promoting efficiency of transfer, and optimizing timeliness of treatment.

The current findings suggest that some attention should be channeled to walk-in patients in order to eliminate the inequality of the implementation effect of the healthcare quality improvement between patients with different arrival modes. The strategies to deal with this disparity might include, but are not restricted to, the following suggestions. From the patient level, health education on recognition of the onset symptoms of STEMI and awareness of seeking treatment by calling EMS should be encouraged and perhaps conducted by community healthcare centers and hospital-based chest pain centers. The existing evidence indicates that wider use of EMS by patients with acute chest pain may offer a considerable opportunity for improvement in public health [14,21,22,25,26]. From the level of healthcare professionals, physicians, general practitioners, and nurses in emergency department should pay close attention to walk-in patients whose main complaint is chest pain. On the one hand, healthcare professionals should improve capacity for rapid diagnostics and triage of STEMI requiring primary PCI. On the other hand, the hospital could set up a green channel to optimize the ambulatory treatment process for them so as to buy time for healthcare professionals. From the hospital level, it is warranted to reinforce the information sharing and communication between the emergency and cardiology departments and establish a multidisciplinary coordinated team of healthcare professionals for comprehensive triage, treatment, and transfer of STEMI cases.

There were some limitations to this study. First, the nature of the cross-sectional design of this study restricted us to making causal inferences between the chest pain center accreditation and decreased in-hospital delay. Rather, the associations found in the present study underscore the need for research to capitalize on chest pain center accreditation to mitigate in-hospital delay. Second, this study included patients who were undergoing PCI; therefore, the results cannot be generalized to all patients with STEMI. Third, we were unable to adjust for medical history and socioeconomic indicators (e.g., income, educational attainment, marriage status, etc.) due to the unavailability of relevant data for the patients. However, a previous publication suggested that these variables are not associated with in-hospital delay but might be associated with pre-hospital delay and mortality [27]. Moreover, in terms of measuring the effect on the in-hospital delay rather than clinical outcomes, we adjusted for Killip classification, which is positively associated with medical history of patients [16,28] and could account for the partial confounding effect of medical history. Finally, although our analysis included all 33 centers accredited during January 2016 to June 2019 in Beijing, it is inevitable to introduce heterogeneity regarding the quality of data collection. Voluntary participation of hospitals made it difficult for us to compare with those not seeking accreditation.

#### **5. Conclusions**

Among STEMI patients undergoing primary PCI, EMS-transported patients were more sensitive to the shortened in-hospital delay associated with chest pain center accreditation efforts. This effect might exacerbate the existing disparity in in-hospital delay among patients with different arrival modes. Thus, more attention should be paid to walk-in patients and more strategies for increasing the utilization of EMS should be considered in further healthcare quality improvement.

**Supplementary Materials:** The following are available online at https://www.mdpi.com/article/ 10.3390/healthcare9111462/s1, Figure S1: Median and interquartile range of door-to-balloon time by accreditation status and arrival mode. Table S1: Median and interquartile range of door-toballoon time (minutes) by accreditation status and arrival mode. Table S2: Odds ratio (OR) and 95% confidence interval (95% CI) of in-hospital delay associated with accreditation status and transfer mode. Table S3: Changes in door-to-balloon time and 95% confidence interval (95% CI) associated with accreditation status and transfer mode. Table S4: Odds ratio (OR) and 95% confidence interval (95% CI) of in-hospital delay associated with accreditation status by arrival mode. Table S5: Changes in door-to-balloon time and 95% confi-dence interval (95% CI) associated with accreditation status by arrival mode. Table S6: Odds ratio (OR) and 95% confidence interval (95% CI) of covariates obtained from fully adjusted models. Table S7: Changes in door-to-balloon time (minutes) and 95% confidence interval (95% CI) of co-variates obtained from fully adjusted models.

**Author Contributions:** Conceptualization, Y.J. and Z.Z.; Data curation, J.M., X.D. and M.M.; Formal analysis, N.L.; Funding acquisition, Y.J.; Investigation, S.Z.; Methodology, N.L. and Y.J.; Project administration, Y.J.; Software, N.L.; Supervision, Y.J. and Z.Z.; Validation, Z.Z.; Visualization, N.L.; Writing—original draft, N.L.; Writing—review and editing, Y.J. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Natural Science Foundation of Beijing Municipality, grant number 9204025 and the National Natural Science Foundation of China, grant number 71904004.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of Peking University Institutional Review Board (IRB000001052-21020).

**Informed Consent Statement:** Informed consents were obtained from hospitals for research approval to collect data without requiring individual patient informed consent. Patient confidentiality is protected in the following ways: (1) data are de-identified before their use in research and (2) the use of data for these purposes is closely overseen by the accredited hospitals and the Beijing Chest Pain Center Alliance.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to privacy.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **References**


## *Article* **Development and Validation of an Instrument to Measure Work-Related Stress among Rescue Workers in Traumatic Mass-Casualty Disasters**

**Yu-Long Chen 1,2, Wen-Chii Tzeng 3, En Chao <sup>4</sup> and Hui-Hsun Chiang 3,\***


**Abstract:** Rescue workers are a population at high-risk for mental problems as they are exposed to work-related stress from confrontation with traumatic events when responding to a disaster. A reliable measure is needed to assess rescue workers' work-related stress from their surveillance of a disaster scene to help prevent severe PTSD and depressive symptoms. The purpose of this study was to develop and validate the Work-Related Stress Scale (WRSS) designed to measure stress in rescue workers after responding to traumatic mass-casualty events. An exploratory sequential mixed methods procedure was employed. The qualitative phase of the item generation component involved in-depth interviews of 7 experienced rescue workers from multiple specialties who had taken part in 1 or 2 mass-casualty events: the 2018 Hualien earthquake or the 2016 Tainan earthquake. In the quantitative phase, a modified Delphi approach was used to achieve consensus ratings by the same 7 raters on the items and to assess content validity. Construct validity was determined by confirmatory factor analysis using a broader sample of 293 rescue workers who had taken part in 1 of 2 mass-casualty events: the 2018 Hualien earthquake or the 2021 Hualien train derailment. The final WRSS consists of 16 items total and 4 subscales: Physical Demands, Psychological Response, Environmental Interruption, and Leadership, with aggregated alphas of 0.74–0.88. The WRSS was found to have psychometric integrity as a measure of stress in rescue workers after responding to a disaster.

**Keywords:** work-related stress; disaster rescue workers; traumatic events; mass-casualty incidents; disaster management

#### **1. Introduction**

Traumatic disasters with mass casualties incur major human and healthcare costs in countries such as Taiwan that provide many opportunities for such events to occur [1]. The effects of these catastrophes are experienced not only by the victims at the disaster site; they also create a high risk of long-term negative psychological consequences in the rescue workers [2–4]. Rescue workers, especially firefighters, are first responders who provide immediate support to victims at a disaster site. Those who work with multiple other professionals on a rescue team (e.g., in a search and rescue operation) have been found to experience several specific physical–psychosocial–environmental–organizational challenges, such as problems related to food and drink, job-related conflicts with the control tower, lack of cooperation from other team members, confusion about who has what duties, and harsh work environments that require observing and identifying very badly damaged bodies of children and adults [5,6]. Despite these multiple challenges,

**Citation:** Chen, Y.-L.; Tzeng, W.-C.; Chao, E.; Chiang, H.-H. Development and Validation of an Instrument to Measure Work-Related Stress among Rescue Workers in Traumatic Mass-Casualty Disasters. *IJERPH* **2021**, *18*, 8340. https://doi.org/ 10.3390/ijerph18168340

Academic Editor: Paul B. Tchounwou

Received: 21 July 2021 Accepted: 5 August 2021 Published: 6 August 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

there are no instruments designed to measure stress surveillance in rescue workers after deployment to traumatic disasters [7].

Rescue workers, especially first responders, are at a high risk for experiencing high levels of psychosocial work stress and depression [8]. In this population, these psychosocial factors appear to be more complex and multifaceted than the environmental factors and the physical and mental strain evoked by work-related accidents [9]. Rescue workers must confront mass-fatality incidents involving many dead bodies and/or body parts as well as emotional involvement with the deceased victims, at times under the threat of injury to themselves [10]. Greater physical strain has been associated with unfair distribution of work tasks, and greater psychological strain has been associated with frequent differences of opinion with other workers or supervisors that interfere with the work [8]. Other research on the stress experienced by rescue workers has shown these stressors to include environmental factors and ineffective cooperation with team members or those in the control tower in the aftermath of a disaster [5]. Direct or indirect exposure to such work environments plays an important role in creating negative psychological consequence in rescue workers when the circumstances are extreme and inhumane and the emotional reactions toward the victims are overwhelming [11].

Work-related stress has been significantly associated with depressive symptoms, burnout, and PTSD symptoms among rescue workers [11,12]. They are at a high risk for physical and mental problems, as they are confronted with traumatic events and endure work-related stress exposure [13]. It is important to develop an instrument for assessing work-related stress specifically for rescue workers who experience physical and psychological impairment after responding to a disaster [14]. The measures of workrelated stress currently available have been criticized for the heterogeneous nature of the populations to which they have been applied and their non-specificity re disaster-related events [15,16], especially traumatic mass-casualty events [13]. Our aim was to fill this knowledge gap by developing and evaluating the psychometric integrity of a measure of work-related stress designed specifically for rescue workers.

#### **2. Methods and Results**

We developed and validated the WRSS across different cultures and types of traumatic mass-casualty disasters, using a succession of qualitative and quantitative methods. In so doing, we followed the recommendations of Zhou [17], who proposed a mixed methods model that features application of both factor analysis and application of mixed-methods integration procedures to culture-relevant data obtained through qualitative interviews to assess the construct validity of a test instrument.

#### *2.1. Item Generation for the WRSS*

#### 2.1.1. Participant Recruitment and Eligibility Criteria

The original sample for the scale construction consisted of 7 disaster rescue personnel (2 females and 5 males) with various search and rescue (SAR) backgrounds: logistics, search, rescue, SAR dog controller, security officer, and commander. Each had taken part in the response to one of two traumatic mass-casualty events in Hualien, Taiwan: an earthquake on 6 February 2018 or a train derailment on 2 April 2021. The earthquake measured 6.4 on the momentary magnitude scale. It claimed at least 17 dead and 285 injured, and featured bodily mutilations [18]. The train derailment claimed 49 dead, most featuring bodily mutilation or body parts, and a further 247 injured and requiring medical treatment [19]. Both traumatic disasters had the potential to cause members of the SAR team to experience physical–psychological stress at the time and psychological and emotional trauma later [14]. The demographic data for the rescue workers are presented in Appendix A Table A1.

#### 2.1.2. The Interview (Qualitative Component)

Qualitative data were collected through in-depth face-to-face semi-structured interviews of the above 7 rescue workers, all of whom who agreed to be interviewed. The focus

of the interviews was first on participants' experiences of their most recent SAR operation and on how they experienced the work-related stress and the effects of the SAR procedures on them (see Appendix A Table A2). The interviews lasted 45 to 90 min.

The qualitative analysis of the interview data included several steps. In the first step, the researcher analyzed the verbatim transcriptions of the audio-taped interviews using thematic analysis [20]. The transcripts were first read to obtain a general sense of the topic and bring it into focus. There was no sorting or coding of the data in the first step. In the second step the transcripts were reread, and descriptive phrases were extracted. In the third step, the relevant phrases were coded, keeping the codes as similar to the participants' meanings as possible. In the fourth step, the researcher developed categories from the codes by aggregating similar codes. The codes and categories were examined and compared within and across items to identify relevant relationships. The researcher's inferences were analyzed and compared with those from the 7 interviewees using a form of peer debriefing.

The four thematic categories of the causes of the traumatic stress identified by the researcher in the third step of the analysis were: physiological demands (of the stress), psychological response (to the stress), leadership (by the team leader), and environmental interruption (at the disaster site). The researcher then generated the 69 original items of the WRSS so they would correspond to these themes.

#### 2.1.3. The Modified Delphi Approach (Quantitative Component)

The 69 items were further evaluated to achieve consensus from the same 7 rescue workers (raters) using a modified four-stage Delphi method [21]. In this study, the Delphi process required three rounds of item selection and modification. In Round 1, the judges were asked to independently rate each of the 69 statements on whether they agreed that it is a good measure of traumatic stress using a 5-point Likert scale ("strongly agree", "agree", "adequate but needs modification", "disagree", "strongly disagree") in the context of responding to a disaster. It has been demonstrated that 5-point scales produce stable findings in modified Delphi studies [22]. The raters were also provided an opportunity to elaborate or explain their ratings and contribute further ideas for each category through responses to open-ended questions provided by the researcher. Using this information, the researcher eliminated 35 items in which there was less than 70 percent of inter-rater agreement and reworded some of the remaining 34. The rescue workers then rated these 34 items in Round 2. The Round 1 process was repeated, which led the researcher to eliminate 6 more items according to the previous criterion. The entire process was repeated in Round 3, leading to the elimination of 9 more items. The ratings of each of the remaining 19 items met the criterion for consensus, defined as >70% of participants affirming the good quality of the item by choosing the response "agree" or "strongly agree". This definition of consensus has been considered appropriate in previous Delphi studies [23]. Results from the modified Delphi procedure are reported in Appendix A Table A3.

#### *2.2. Content Validity of the WRSS*

Content validity is the degree to which an instrument covers the conceptual domain of the construct it is intended to measure [24]. The content validity of the 19-item WRSS was assessed by a panel of 3 judges with professional expertise in disaster rescue and emergency management. All were instructors with the Disaster Medical Assistance Team (DMAT) and had personally witnessed several catastrophic disasters in Taiwan. They rated each item on whether it reflected traumatic stress using a 4-point Likert scale with the following response options: "agree very much", "agree", "agree a little bit", and "not agree at all". The content validity of the scale was found to be 0.94 by using the content validity index (CVI) to detect the validity of the items.

#### *2.3. Construct Validity of the WRSS*

#### 2.3.1. Participant Recruitment and Eligibility Criteria

Using an online sample-size calculator for structural equation models, it was estimated that 305 participants would be required to detect a moderate effect (*ρ* = 0.23) of high workrelated stress for rescue workers who had a high workload [13]. This allowed for 5 latent variables, 19 observed variables, power of 0.8, and alpha of 0.05 [25].

Purposive sampling was used to recruit the participants from March 2018 to May 2021. A sample of 305 rescue workers who were engaged with the 2018 Hualien earthquake or the 2021 Hualien train derailment disasters agreed to participate in the study after the purpose and procedure had been explained by the researchers, who were referred by the occupational nurse. Participants were assured that their participation was anonymous and would not influence their work. Of the 305 rescue workers recruited, 12 were unable to complete the questionnaire because of work-shift rotation problems. Data from the remaining 293 rescue workers were used for the analyses.

#### 2.3.2. Procedure and Data Analysis

After signing the consent form, it took about 15–20 min for the participants to complete the questionnaire in a quiet room in a comfortable environment. The questionnaire consisted of the 19-item WRSS and items requesting the same demographic information obtained from the rescue workers who participated in the 2018 Hualien earthquake or the 2021 Hualien train derailment disasters. We used two separate boxes to collect the consent forms and questionnaires to assure anonymity.

Table 1 presents the demographic characteristics of the sample. To summarize, most participants were men (*n* = 283, 96.6%), were married (*n* = 191, 65.2%), and were between 31–40 years old (*n* = 175, 59.7%). Most reported that their SAR experience was less than or equal to 5 years (*n* = 236, 80.5%), and that their service as firemen was less than 5 years (*n* = 200, 68.3%).

**Table 1.** Demographics of the rescue workers and differences in their WRSS scores (*n* = 293).


*Note.* <sup>a</sup> from independent *t* test or ANOVA; SAR: search and rescue.

Following the recommendations of Zhou [17], confirmatory factor analysis (CFA) was employed, without a prior exploratory factor analysis, to examine the construct-based validity of the 19-item WRSS using *Mplus* software version 8.2 [26]. The purpose was to confirm a common factor pattern and to determine whether the scale structure in fact corresponds to the four themes identified from the interviews. The determination of which items were to be retained for the final scale was based on their model fit with the factor pattern [27] and conformance to the guiding theoretical definitions of the work-related stress dimensions [2].

Results from the CFA indicated that a four-factor solution was best. After 3 items were removed from the scale because of cross-loadings of the variables, the remaining 16-items fell into four factors (or subscales) with appropriate names corresponding to the four themes derived from the interviews: (1) Environmental Interruption (four items); (2) Psychological Response (five items); (3) Leadership (four items); and (4) Physiological Demands (three items). Participants responded to each item on a four-point scale: (a) "not at all stressful"; (b) "a little bit stressful"; (c) "stressful"; and (d) "very stressful". Average variance extracted (AVE) was > 0.5 except for Psychological Response (0.43), but the composite reliability for this subscale was higher than the acceptable level of 0.6 [28]. Table 2 presents the CFA results for each item on each factor and Table 3 presents the items on the final scale.

**Table 2.** Summary of confirmatory factor analysis and reliability of Work-Related Stress Scale (*n* = 293).



**Table 2.** *Cont.*

*Note.* PHY: physical demands; PSY: psychological response; ENV: environmental interruption; LEAD: leadership. Fit indices: χ2(98) = 183.84, *p* ≤ 0.001; CFI = 0.96, TLI = 0.95; RMSEA = 0.06; SRMR = 0.05. CI: confidence interval; bootstrap = 1000; range: lower 2.5% to upper 2.5%. The criterion column refers to the raters' assessments and specifies how many of them classified each item as 1: strongly disagree; 2: disagree; 3: adequate but needs minor rewording; 4: agree; and 5: strongly agree. *n* = number of raters; %: percent of inter-rater agreement, defined as the percent of participants affirming the good quality of the item by choosing the response "agree" or "strongly agree"; *M*: mean rating assigned to each item. *<sup>a</sup>* deleted following confirmatory factor analysis; item 10: cross-loadings of ENV and PSY (MI = 52.47); item 21: cross-loadings of PSY and ENV (MI = 32.25); items 22: cross-loadings of ENV and PSY (MI = 15.65).

**Table 3.** Final items on the Work-Related Stress Scale and ratings from the modified Delphi method.


*Note.* ENV: environmental interruption; PSY: psychological response; LEAD: leadership domain; PHY: physical demands; SAR: search and rescue.

#### *2.4. Reliability of the WRSS*

The reliability of the 16-item WRSS was assessed through internal consistency analysis. Cronbach's alpha values for the four subscales ranged from 0.74 to 0.88 and the composite reliability values ranged from 0.79 to 0.88. For the total scale, Cronbach's alpha was 0.89 (see Table 2).

#### *2.5. Ethical Considerations*

In conformance to the ethics requirements, for each phase of scale construction and validation it was emphasized that participants' cooperation was voluntary and that their answers were confidential and would be used only for the purposes of this study. All participants provided their written informed consent. The Institutional Review Board of the Tri-Service General Hospital approved the study (Approval No. 1-107-05-061).

#### **3. Results and Discussion**

#### *3.1. Main Results*

The present study demonstrates that the WRSS has good psychometric integrity. The finding from the interviews that this work-related stress has four dimensions is noteworthy, and four subscales corresponding to these four dimensions (Physical Demands, Psychological Response, Environmental Interruption, and Leadership) were identified in the construct validation phase. We can thus conclude that the WRSS is a well-validated and reliable instrument that is suitable for assessing work-related stress in rescue workers responding to traumatic mass-casualty incidents.

Proper preparation has been shown to be crucial before experiencing the onset of a catastrophic event [13]. Our results indicate that rescue professionals and administrative officers need more specific information about the work-related stress that rescue workers experience and the associated challenges to prepare them to rescue and save the lives of victims. Moreover, in the interviews, the participants registered strong complaints against their administrative officers for not providing them with safety procedures and the necessary control of the environment during SAR events, as well as against the mass media. The findings of our research, which was specifically targeted to disasters, may provide information that will help rescue workers succeed in their rescue operations and improve the quality of disaster SAR. They therefore highlight the importance of developing a specific work-related stress measure for rescue workers.

#### *3.2. Comparisons with the Literature*

Our results are consistent with what has been found in previous studies that have explored the high physical and mental health risks faced by disaster responders [4,29]. In our interviews, participants expressed a lack of preparation for deployment, specifically a lack of rest and poor sleep quality. These results are consistent with previous research that has shown that rescue workers' health status, inability to fully regain energy, exhausted from disaster-support work, and perceived physical disturbances play an important role in the creation of work-related stress and aggravate the psychological burdens they face [30,31]. Unpredictable situations involving a threat to human life, such as those faced by disaster rescue workers, have been shown to cause an increase in cardiac sympathetic excitation [32].

High-quality team leadership has been shown to be significantly associated with a lowered risk of subsequent mental distress [33]. Good team leadership plays an important role in employee health and well-being as well as reducing burnout [34]. It is also an essential antecedent of occupational safety; rule-oriented leadership that formulates plans for future operations collaboratively with the workers improves the workers' motivation to comply with safety regulations and to participate in safety-promoting activities [35]. Our results indicate that rescue professionals and administrative officers need more specific information about the work-related stress that rescue workers experience and the associated challenges in order to prepare them for rescue missions and to save the lives of victims. The results of this study are consistent with those of a previous study demonstrating an increased risk of harmful consequences of simply being in the disaster environment; they include infectious disease, catastrophic injury to oneself or coworkers, severe burns, and psychological stress [36]. These results, as well as our own, call for increased attention to rescue workers' safety needs and the overall consequences of their work after the deployment.

#### *3.3. Strengths and Limitations*

Although the results of the present study are a valuable contribution to the literature, several limitations should be noted. Although the instrument has adequate overall psychometric integrity, the number of rescue workers available to serve as raters was

small because there are so few catastrophic events that workers respond to. The resulting small sample size in the scale construction phase could have impacted the accuracy of the estimates from the measurement models. Likewise, the samples of raters in both phases may not have been representative of the population of workers who respond to traumatic mass-casualty disasters, because the period of data collection was small (2018 to 2021). Moreover, most of the raters were young men, few of whom were from an organization, and all of them were from just one culture. These factors limit the generalizability of the findings. Future research should use samples from different populations. Finally, previous empirical studies on work-related stress in disaster rescue, including our own study, have been retrospective rather than prospective; future research should include prediction of the consequences of work-related stress in disaster rescue. The strength of employing an exploratory sequential mixed methods approach for constructing the WRSS is that it provided a better understanding of the experiences of the participants because of the inclusion of a qualitative component. The weakness of our application of this approach is the lack of generalizability of the findings.

#### **4. Conclusions**

In this paper, the construction of a scale measuring work-related stress in rescue workers with good validity and trustworthiness was described. This 16-item, four-factor instrument, for which there is evidence of good content and construct validity, enables a fast, comprehensive, and systematic assessment of the stress suffered by rescue workers from responding to traumatic mass-casualty disasters. The results should increase understanding of rescue workers' needs and the stresses they experience while responding to traumatic events; paying attention to the outcomes of this research is likely to be important for improving the efficiency and safety of disaster rescue workers. Rescue professionals and administrative officers need to pay more attention to monitoring the work-related stress of rescue workers and the associated challenges to prepare them for SAR and to save the lives of victims.

**Author Contributions:** All authors contributed to this manuscript. Y.-L.C. contributed to the acquisition of quantitative data and reviewed the article critically. W.-C.T. contributed to the acquisition of qualitative data and reviewed the article critically. E.C. contributed to the acquisition of quantitative data, interpreted data, and reviewed the article critically. H.-H.C. conceptualizing and designed the study, acquired the data, analyzed the data, interpreted the data, and reviewed the article critically for important intellectual content. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported in part by the Ministry of the National Defense Medical Affairs Bureau, Taiwan (MND-MAB-110-050), the Ministry of Science and Technology, Taiwan (MOST 108-3111-Y-016-014) and Taipei Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, Taiwan (TCRD-TPE-108-41).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of Tri-Service General Hospital (protocol code: 1-107-05-021).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** Datasets related to this article can be requested from corresponding author.

**Acknowledgments:** We are grateful to our participants for their time and effort. We are thankful to the New Taipei City Fire Department, the Taipei City Fire Department, and the Hualien County Fire Department for their cooperation in this study.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**


**Table A1.** Demographics of the qualitative interview raters (*n* = 7).

*Note.* SAR: search and rescue.


**Table A2.** *Cont.*


*Note.* SAR: search and rescue.

**Table A3.** Numbers of remaining items after rounds 1 to 3 of the Delphi procedure.


#### **References**


## *Review* **Decontamination Methods of N95 Respirators Contaminated with SARS-CoV-2**

**Thirumaaran Gopalan 1, Rabi'atul 'Adawiyah Mohd Yatim 1, Mohd Ridha Muhamad 1,2,\*, Nor Shafina Mohamed Nazari 3, N. Awanis Hashim 4, Jacob John <sup>3</sup> and Victor Chee Wai Hoe <sup>5</sup>**


**Abstract:** In the preparation and response to the COVID-19 pandemic, a sufficient supply of personal protective equipment (PPE), particularly the face mask, is essential. Shortage of PPE due to growing demand leaves health workers at significant risk as they fight this pandemic on the frontline. As a mitigation measure to overcome potential mask shortages, these masks could be decontaminated and prepared for reuse. This review explored past scientific research on various methods of decontamination of the N95-type respirators and their efficiency against the SARS-CoV-2 virus. Ultraviolet germicidal irradiation (UVGI) and hydrogen peroxide vapor (HPV) show great potential as an effective decontamination system. In addition, UVGI and HPV exhibit excellent effectiveness against the SARS-CoV-2 virus on the N95 respirator surfaces.

**Keywords:** decontamination; N95 respirators; SARS-CoV-2; COVID-19; ultraviolet germicidal irradiation (UVGI); hydrogen peroxide vapor (HPV); heat; microwave-generated steam (MGS); ethanol

#### **1. Introduction**

According to the WHO, COVID-19 human cases, which are caused by a novel coronavirus named SARS-CoV-2, were first reported in Wuhan City, China, in December 2019 [1]. Due to this unprecedented pandemic, the demand for face mask respirators has surged significantly. The WHO predicted that mask manufacturing industries need to increase manufacturing by 40 percent to meet the demand [2]. Frontline workers rely solely on PPE, especially N95 respirators, to protect themselves from being infected and infecting others. The N95 respirators should be disposed of after a sole patient visit, according to the Centers for Disease Control and Prevention.

Nevertheless, under acute PPE scarcity, it advises prolonged use of N95 respirators (using the same N95 respirator for many patient interactions) with limited reuse (keeping an N95 respirator during interactions for usage across several patients' visits). During the COVID-19 pandemic, due to a shortage of N95 masks, several emergency services have implemented various N95 prolonged use strategies. However, there is insufficient scientific proof that they were successful. In one investigation, researchers examined how often duckbill N95s and dome-shaped N95s masks failed by using fit-tests when they were reused. They concluded that healthcare systems must closely monitor N95 fit throughout extended usage or reuse and avoid using duckbill masks if better options are available [3].

**Citation:** Gopalan, T.; Mohd Yatim, R.'A.; Muhamad, M.R.; Mohamed Nazari, N.S.; Awanis Hashim, N.; John, J.; Wai Hoe, V.C. Decontamination Methods of N95 Respirators Contaminated with SARS-CoV-2. *Sustainability* **2021**, *13*, 12474. https://doi.org/10.3390/ su132212474

Academic Editors: Amir Khorram-Manesh and Krzysztof Goniewicz

Received: 13 September 2021 Accepted: 8 November 2021 Published: 11 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Among the available models of face masks, N95 respirators are designed and intended for healthcare usage [4].

Developing countries whose populations are mostly made up of people living in poverty, such as India, Pakistan, and Sri Lanka, face even greater challenges due to a shortage of masks. The slowed economies in these countries, coupled with a face mask price hike, made people prioritize daily necessities over face masks, promoting the risk of the COVID-19 pandemic still existing in the community [5]. Due to these shortages, health workers were forced to ration their face mask supply to one N95 mask per week with an additional surgical mask on top. In addition, healthcare facilities are restricted to performing some non-COVID-related medical care as these supply limitations are concentrated on COVID-related patients [6].

As a solution, extending the usage of N95 respirators can assist in overcoming the shortage of masks experienced worldwide. Decontamination procedures of face masks that reduce the pathogen burden show great potential to alleviate the shortage of mask issues. According to NIOSH, ultraviolet germicidal irradiation, vaporous hydrogen peroxide, and moist heat have shown the most potential procedures to decontaminate filtering facepiece respirators (FFR) [7].

In essence, the mask shortage problem during the pandemic needs to be addressed immediately. This review aimed to compare the decontamination procedures of the virus on the N95 respirator, particularly highlighting effective but economical methods.

#### **2. Methods**

Relevant studies were searched using the PubMed and Preprint platform (medRxiv) electronic databases using a combination of specified MeSH terms that were restricted from 2000 to 2021 (Table 1). Apart from the database searches, several studies were included based on the relevance to this review. In addition, regulatory documents related to the decontamination of N95 respirators were obtained from the official websites of the CDC, the FDA, the WHO, and 3M. Studies were selected for evaluation based on specified inclusion criteria: (a) studies reporting at least one of the selected N95 respirator decontamination procedures for this review (UVGI or HPV or heat or MGS or ethanol); (b) studies reporting at least one of the selected N95 respirator decontamination outcomes (reduction in pathogen load or mask performance or structural integrity of the mask).

**Table 1.** Studies search strategies and outcomes.


#### **3. SARS-CoV-2**

The WHO named the pathogen that causes coronavirus disease (COVID-19) SARS-CoV-2 on 12 February 2020. CoVs is a single-stranded positive-sense RNA (+ssRNA) virus [8]. The schematic structure of the SARS-CoV-2 virus is illustrated in Figure 1. The SARS-CoV-2 virus was reported to possess 80% similarity in the aspect of the genome to previous human coronaviruses. Bats were deduced as the vital host and transmitting medium of the SARS-CoV-2 virus [9]. It was concluded that SARS-CoV-2 is transmitted mainly via respiratory droplets and direct contact [10]. Evaluation of the stability of SARS-CoV-2 on different environmental conditions demonstrated that after seven days, a detectable level of the virus still presents on the outer layer of the surgical mask [11]. The FDA calls for a policy where at least three log reductions must be achieved to sterilize devices intended for skin contact [12].

**Figure 1.** Schematic structure of SARS-CoV-2 [13].

#### **4. The N95 Respirator**

The N95 respirator is a type of respiratory protective equipment with a specific design to tightly fit its user. This type of respirator undergoes a testing and evaluation process by NIOSH [14]. In comparison to other FFRs, the N95 respirator offers a minimum of 95% filtration efficiency against particulate aerosols [15]. Quantitative fit testing of FFRs proves the superior protection that the N95 respirator offers [16].

The N95 respirator is made up of four layers, namely, a coverweb, a shell, filter 1, and filter 2 as illustrated in Figure 2. The coverweb and the shell layers are made up of polyester; meanwhile, filter layers are made from polypropylene [4]. The filtration efficiency of the respirator is determined by the internal filtration layer, which is a high-efficiency melt-blown non-woven material [17].

**Figure 2.** Multilayer sandwich anatomy of N95 mask. (**A**) Environmental interface; (**B**) user interface; (**C**) from left to right: inner layer (shell), middle layers (filter 2 and filter 1), and outer layer (coverweb); (**D**) light microscope images of the four layers, with a lower row at four-fold higher magnification (3M model 8210). Adapted from [18] with permission.

#### **5. Decontamination Treatment for N95 Respirators**

#### *5.1. Ultraviolet Germicidal Irradiation (UVGI)*

UVGI is a scientifically proven decontamination method that can destroy the protein coating of the SARS-coronavirus, which possesses similar characteristics as the SARS-CoV-2 virus (COVID-19 virus) [19]. Ozog et al. [20] reported excellent decontamination results of the SARS-CoV-2 virus with a 1.5 J/cm<sup>2</sup> UV dose, which was achieved using a 4 UVC lamp set-up. Vo et al. [21] produced the required decontamination levels up to a three-log reduction with a UV dose of 4.32 J/cm<sup>2</sup> and complete decontamination with a ≥7.20 J/cm2 dosage against the MS2 virus. A relatively longer decontamination time was reported due to the low range of UV irradiation used in the research.

Lindsley et al. [22] tested a UV dose up to 950 J/cm2 on N95 respirators, which resulted in acceptable degradation on filtration performance and no effect in flow resistance. This study reported a perfect range for UVC-based decontamination treatment cycles. Ozog et al. [23] reported excellent fit testing results using N95 respirators with a total exposure of 60 J/cm2.

#### *5.2. Hydrogen Peroxide Vapor (HPV)*

HPV-based decontamination systems are regarded as some of the best decontamination systems due to their efficacy against various microorganisms and their rapid processing cycles [24]. Saini et al. [25] tested the N95 respirator's decontamination against three biological indicators: Escherichia coli, Mycobacterium smegmatis, and spores of Bacillus stearothermophilus using an HPV machine. Excellent decontamination results were reported where decontamination up to a seven-log reduction was achieved using 11–12% HPV against E.coli. Jatta et al. [26] performed decontamination with a 59% HPV concentration using a VPRO maX low-temperature sterilization system. These research results exhibited no significant effect on the filtration performance and fit of the N95 mask after exposure to 59% HPV up to 10 cycles. The range of treatment time reported in this study provides a solid foundation for an HPV-based decontamination system design.

#### *5.3. Heat*

#### 5.3.1. Moist Heat

Lore et al. [27] tested moist heat decontamination against the influenza virus applied on an N95 mask. In this study, a contaminated mask was heated to 65 + 5 ◦C for 3 h. The results show that the required decontamination level (>four-log reduction) was achieved. However, a relatively slow decontamination time can prove to be an inefficient decontamination procedure for everyday application. Rockey et al. [28] investigated the effect of humidity in virus heat inactivation against two bacteriophages (MS2 and phi6), a mouse coronavirus (murine hepatitis virus), and a recombinant human influenza A virus subtype H3N2 (IAV) using a humidity-controlled oven. Heat treatments illustrated greater decontamination results with increasing humidity, where six-log reductions were reported in humidity exceeding 50%.

Bopp et al. [29] tested multiple cycles of autoclaves on N95 respirators. Four different autoclave cycles (115 ◦C for one hour, 121.1 ◦C for 30 min, 130 ◦C for two minutes, and 130 ◦C for four minutes) were administered to N95 FFRs. N95 FFRs showed negligible differences in their functionality and integrity even after three cycles. Andregg et al. [30] applied heating decontamination to N95 respirators with moisture (85 ◦C, 60–85% humidity) in a polypropylene container and a convection oven setup. Post-decontamination N95 FFRs exhibited excellent results in both quantitative fit testing and filtration efficiency.

#### 5.3.2. Dry Heat

Xiang et al. [31] implemented dry heat pasteurization for one hour at 70 ◦C for the N95 respirator's decontamination. This study showed that this procedure can kill six species of respiratory bacteria and one fungi species and can inactivate the H1N1 indicator virus. In addition, neither the performance nor the integrity of N95 respirators showed significant degradation. This study shows that dry heat is capable of deactivating various pathogens but at a relatively slow rate. Pascoe et al. [32] successfully decontaminated pathogen (*S. aureus*) under dry heat of 70 ◦C by reducing log 4 in 90 min using a laboratory incubator. Despite strong decontamination results, the slow decontamination rate might prove to be the drawback of this method. Viscusi et al. [33] reported a slight increase in average penetration at N95 respirators when exposed to 80 ◦C after 60 min. These results can potentially act as a limitation for dry heat exposure to an N95 mask.

#### *5.4. Microwave Generated Steam (MGS)*

Fischer et al. [34] have proved up to a four-log reduction in bacteriophage MS2 pathogenic virus using sealed steam bags on a 1100-W-rated microwave for 90 s. In addition, tested N95 respirators also passed the minimum required filtration efficiency requirements of 95%. Zulauf et al. [35] reported a reduction greater than four logs measured in PFU on the N95 respirator. They tested MS2-phage-contaminated N95 respirators to microwavegenerated steam for 3 min. Moreover, the respirators exhibited the required filtration performance and integrity even after 20 cycles of 3 min.

#### *5.5. Ethanol*

By using ethanol, decontamination of pathogens happens by protein denaturation. At a concentration of 60%–80%, ethanol proves to be effective against lipophilic viruses and many hydrophilic viruses [36]. Liao et al. [37] tested N95 respirators using a 75% ethanol treatment, which was immersed and dried. The filtration efficiency of the N95 respirators were affected considerably with treatment, which indicates that ethanol treatment could not retain the mask's reusability properties.

#### *5.6. Other Methods*

N95 respirator decontamination procedures other than the methods selected for this review (UVGI or HPV or heat or MGS or ethanol) are highlighted based on their potential as a low-cost and accessible method. Lendvay et al. [38] tested SARS-CoV-2-inoculated N95 masks under methylene blue (MB) photochemical action for decontamination. They showed that MB activated by red or white light significantly inactivates SARS-CoV-2 on N95 mask surfaces without compromising the specimen's integrity. Excellent virucidal activity of 99.8%–>99% was reported, and preservation of mask integrity proved up to five treatment cycles. Their findings suggested a strategy for decontaminating PPE and masks for reuse that is accessible and inexpensive and that can be used in high-resource and lowresource situations amid supply disruptions. This is due to the worldwide availability of MB light at an affordable cost without using specialized instruments. In addition, the New York City Department of Health and Mental Hygiene has released passive decontamination guidance to health workers to use a paper bag or other clean, breathable containers to store used N95 respirators to prolong their efficiency over multiple usages. The method is as follows. Each day, the healthcare workers would use one N95 respirator with a tagged name and the number of the day used and would place it in a paper bag or a ventilated container at the end of the shift. The mask should be disposed of after the seventh day of use. Healthcare workers must be aware that the N95 respirator could be contaminated albeit at a substantially lower rate. Limited storage periods may be considered, although they may raise the chance of contamination. As the more rigorous disinfecting techniques become accessible, this strategy could be integrated for higher efficiency [39]. Heimbuch et al. [40] evaluated the ability of wipe products available commercially to clean filtering facepiece respirators (FFRs) contaminated with pathogenic or non-pathogenic aerosols. They examined the decontamination effect of benzalkonium chloride, hypochlorite, and nonantimicrobial wipes on the N95 FFRs. The highest particle penetration capacity was observed in benzalkonium chloride wipes. They reported effective decontamination results of S aureus up to 99.72% (exterior of N95) and 98.60% (interior of N95) using benzalkonium chloride (BAC) wipes. Decontamination using wipes is readily

available for public usage, but penetration of respirator due to wipe decontamination must be approached with caution.

#### *5.7. Comparison of Decontamination Treatments for N95 Respirators*

The reusability of a disinfected N95 respirator depends on several factors such as inactivation of the targeted organism, the safety of the user, and consistent filtration function and fit of the respirator. UVGI and HPV have demonstrated excellent results as an efficient decontamination method with effective elimination of SARS-CoV-2 virus while preserving the performance of the respirator. However, extensive studies are needed to incorporate HPV- and UVGI-based decontamination systems into a household-based portable commercial-ready product for commercial use. On the other hand, the MGS-based decontamination method exhibits great potential with rapid disinfection for household applications. Currently, there are still few studies about this method for decontamination application. Its rapid method enables a huge potential of applications. However, use in materials that are sensitive to steam could be a concern for material degradation. The other method includes the heat-based decontamination method, which has a major drawback for its time-consuming process and filtration performance degradation in extensive dosages. The conventional method of using ethanol has shown unavoidable degradation of the respirator by using this procedure. Table 2 demonstrates the effects of using a specified N95 decontamination treatment.


**Table 2.** Advantages and disadvantages of decontamination treatments for N95 respirators.

#### **6. Decontamination System Design for N95 Respirators**

#### *6.1. Ultraviolet Germicidal Irradiation*

Several factors must be taken into account when designing a UVGI-based decontamination system, namely, the wavelength of the ultraviolet rays, the irradiance, and the exposure time. The effectiveness of a UVGI-based decontamination system depends on the dosage of UVC administered to the N95 mask. A safe dosage range must be estimated beforehand because excessive dosage can affect the integrity of the mask. On the other hand, an insufficient dosage can lead to incomplete deactivation of the virus. The UV dose for a specific system can be calculated using Equation (1) [41]. The system specifications and outcomes of studies related to UVGI-based N95 decontamination are listed in Table 3.

$$\text{LUV dose} \left(\frac{\text{J}}{\text{cm}^2}\right) = \text{Irradiance} \left(\frac{\text{W}}{\text{cm}^2}\right) \times \text{Time (s)} \tag{1}$$

**Table 3.** UVGI-based decontamination system specifications and outcomes.



**Table 3.** *Cont.*

#### *6.2. Hydrogen Peroxide Vapor (HPV)*

Most of the studies reviewed here used commercially available HPV-based decontamination machines. The efficiency of HPV-based decontamination systems depends on the concentration of the HPV used coupled with the time of exposure to the N95 respirator. HPV traces on mask surfaces might induce health hazards. Therefore, each HPV-based decontamination system must be able to produce residue-free N95 respirators upon the decontamination cycle. The system specifications and outcomes of studies related to HPV-based N95 decontamination are listed in Table 4.


**Table 4.** HPV-based decontamination system specifications and outcomes.


**Table 4.** *Cont.*

#### *6.3. Heat*

Heat treatments can sterilize microbes by altering their membranes and denaturing proteins [68]. Heat-related decontaminations can be divided into two main classifications, namely, moist-heat and dry-heat decontamination. The efficiency of a heat-based decontamination system depends on the working temperature, the presence of humidity, and the exposure time. The existence of moisture in the heating procedure is proven to promote better decontamination results. The system specifications and outcomes of studies related to moist heat and dry heat-based N95 decontamination are listed in Tables 5 and 6 respectively.


**Table 5.** Moist-heat-based decontamination system specifications and outcomes.


**Table 5.** *Cont.*


**Table 5.** *Cont.*


#### **Table 6.** Dry-heat-based decontamination system specifications and outcomes.

#### *6.4. Microwave-Generated Steam (MGS)*

MGS-based decontamination has enormous potential for wide application as it can be done with household items. It offers a rapid disinfection rate with minimal expertise needed to perform this treatment. The efficiency of MGS-based decontamination is affected by exposure time and is specific to the design of the selected face mask model for the treatment. However, many protocols use commercial steam bags or special materials that are available in laboratories. The system specifications and outcomes of studies related to MGS-based N95 decontamination are listed in Table 7.


**Table 7.** Microwave-generated steam (MGS)-based decontamination system specifications and outcomes.

#### *6.5. Ethanol*

Ethanol-based disinfection is used widely around the world as an effective decontamination method. However, ethanol-based treatment does not produce an efficient result in the decontamination of N95 respirators. Ethanol is known to degrade the structure of the mask's filtration and thus affects the integrity and performance of treated N95 respirators. The system specifications and outcomes of studies related to ethanol-based N95 decontamination are listed in Table 8.

**Table 8.** Ethanol-based decontamination system specifications and outcomes.


#### **7. Effectiveness of Decontamination Systems against SARS-CoV-2**

The effectiveness of a specific decontamination system depends on critical parameters such as the exposure time. UVGI and HPV were investigated further in this review on their effectiveness against SARS-CoV-2, specifically from the surfaces of N95 respirators. The relationship between parameter control and effectiveness against the SARS-CoV-2 virus is illustrated in Figures 3 and 4.

**Figure 3.** Log reduction of viable SARS-CoV-2 virus with increasing UV dose (data represented in Figure 3 exhibit minimum log reduction achieved by specific dosage as upon reaching the limit of detection (LOD)—real data are not quantifiable).

**Figure 4.** Log reduction of viable SARS-CoV-2 virus with various HPV-based decontamination settings (data represented in Figure 4 exhibit minimum log reduction achieved by specific dosage as upon reaching the limit of detection (LOD)—real data are not quantifiable).

#### *7.1. Ultraviolet Germicidal Irradiation*

In a study, Ozog et al. [20] had demonstrated successful decontamination when an N95 mask was irradiated with 1.5 J/cm2 of UVC (254nm). It was concluded that the dose applied was sufficient. However, a concern on the disinfection of the strap arises due to its coverage by UVC on the strap surface. Rathnasinghe et al. [43] presented a simple UVC decontamination device without the mask's strap decontamination. Golovkine et al. [44], Smith et al. [45], Fischer et al. [46], and Geldert et al. [47] investigated and compared the efficiency of UVC-based decontamination systems for N95 respirators with other decontamination methods such as ethanol, heat, UVA, ethylene oxide, hydrogen peroxide plasma and vapor, MGS, bleach, and liquid hydrogen peroxide. Comparing across the studies, a UVC-based N95 disinfection treatment with a dosage of greater than 0.5 J/cm<sup>2</sup> can achieve the minimum pathogen load reduction required of three-log reduction against the SARS-CoV-2 virus. As Figure 3 illustrates, Geldert et al. [47] demonstrated notable disinfection of five-log reduction at a relatively low dosage of 0.5 J/cm2. Nevertheless, the reported sharp decline in the log reduction of SARS-CoV-2 [47] at lower UVC doses (0–0.5 J/cm2) must be addressed with caution.

#### *7.2. Hydrogen Peroxide Vapor (HPV)*

Smith et al. [45], Fischer et al. [46], Kumar et al. [58], Christie-Holmes et al. [59], and Oral et al. [60] have investigated the efficiency of HPV-based decontamination systems for N95 respirators against the SARS-CoV-2 virus. All the studies that reported HPV-based decontamination against the SARS-CoV-2 virus were designed using commercially available HPV generating machines. The comparison of the HPV-based N95 decontamination system efficiency across the studies is presented in Figure 4. The concentration of hydrogen peroxide exposed and the treatment time of a complete cycle comprised of four different processes are the variables that play a significant part in HPV-based decontamination systems to deliver the required decontamination efficiency. Notably, Kumar et al. [58] demonstrated a significant reduction in SARS-CoV-2 of six-log reduction while preserving the functional integrity of the N95 respirator post-treatment.

#### **8. Conclusions**

The COVID-19 pandemic shows the severity of the needed supply of PPE for healthcare workers to stay protected at all times. Decontamination of PPE could be an essential measure to mitigate the immediate risk of running out of PPE supply. UVGI- and HPVbased decontamination systems exhibit great potential as a good choice for N95 respirator decontamination. The study indicated that the UVGI and HPV methods could be used to deactivate the SARS-CoV-2 virus without affecting the integrity of the respirator. The excellent virucidal activity of UVGI- and HPV-based decontamination systems suggested that they are good candidates for N95 respirator decontamination.

**Author Contributions:** Conceptualization, M.R.M., N.S.M.N., N.A.H., J.J., and V.C.W.H.; methodology, M.R.M., T.G., and R.'A.M.Y.; software, R.'A.M.Y.; validation, M.R.M., N.S.M.N., N.A.H., J.J., and V.C.W.H.; formal analysis, T.G.; investigation, T.G.; resources, M.R.M., T.G., and R.'A.M.Y.; data curation, T.G.; writing—original draft preparation, T.G.; writing—review and editing, T.G. and R.'A.M.Y.; visualization, M.R.M. and T.G.; supervision, M.R.M.; project administration, M.R.M.; funding acquisition, M.R.M. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by AUN/SEED-Net, grant number UMSPRAC 2101.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data are contained within this article.

**Acknowledgments:** The authors would like to thank Mohd Fauzi Bakri Hashim for his assistance in this research project.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


## *Article* **Civilian-Military Collaboration before and during COVID-19 Pandemic—A Systematic Review and a Pilot Survey among Practitioners**

**Amir Khorram-Manesh 1,2,\*, Luc J. Mortelmans 3,4, Yohan Robinson 1,2, Frederick M. Burkle <sup>5</sup> and Krzysztof Goniewicz 6,\***


**Abstract:** Due to the similarity in skills and assets, Civilian-Military collaboration has emerged as one of the most reliable partnerships during the disaster and public health emergency management to address all necessary elements of surge capacity, i.e., staff, stuff, structure (space), and systems. This study aimed to evaluate this collaboration before and during the coronavirus 2019 pandemic. The outcomes of the systematic review revealed several published reports on successful civilian-military collaboration and proposed a need for further improvement. One hundred sixty-six individuals from 19 countries responded to nine questions, included in an online survey with the possibility to leave comments if necessary. The questionnaire referred to elements such as command and control, safety, communication, assessment, triage, treatment, and transport, as the crucial components of emergency management. The comprehensive examination of the survey results together with registered comments revealed a possible improvement in collaboration particularly on the strategic levels, i.e., meetings at the command-and-control level, safety, communication, and networking issues. While logistic collaboration seemed to be unchanged, the practical parts of the collaboration, i.e., clinical and non-clinical operational partnership (Triage and Treatment), mutual education, training, and operational understanding of each organization remained unchanged. In conclusion, although the current pandemic may have facilitated a more intense collaboration between civilian and military healthcare organizations, it lacks practical partnership and operative engagement, representing two crucial elements necessary for harmony and compatibility of both systems. Such collaboration may require a political will and perhaps a mutual civilian-military authority.

**Keywords:** civilian-military collaboration; interagency partnership; pandemic; public health

#### **1. Introduction**

Multiagency collaboration (MC) is universally accepted as an important part of the management of disasters and public health emergencies [1–5]. Among various factors influencing MC, leadership and communication are seemingly the most crucial [3–5]. While experience and transparent leaderships govern the functional relationships and the clarity of the roles between various partners, insufficient leaderships create significant challenges to MC and influence the trust, understanding, and mutual respect between the agencies [6–9]. Moreover, good inter-and intra-organizational communication facilitates a

**Citation:** Khorram-Manesh, A.; Mortelmans, L.J.; Robinson, Y.; Burkle, F.M.; Goniewicz, K. Civilian-Military Collaboration before and during COVID-19 Pandemic—A Systematic Review and a Pilot Survey among Practitioners. *Sustainability* **2022**, *14*, 624. https:// doi.org/10.3390/su14020624

Academic Editors: John Rennie Short and Marc A. Rosen

Received: 12 November 2021 Accepted: 5 January 2022 Published: 6 January 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

successful collaboration, although requiring adequate resourcing and financial assets to achieve the established goals [2,4,5].

Inter-organizational collaboration is a situation-dependent complex process with diversified scope, and structure, thus yielding different understandings, interpretations, and outcomes. Distinctions may arise when two organizations experience diverse responsibilities, autonomy, legacy, organizational framework, and authority structure [10,11]. Consequently, even well-formulated collaboration may fail to achieve its expected outcomes due to several factors, such as asymmetrical structures, and requires continuous supervision and cultivation [12–14].

#### *1.1. Theoretical Framework*

The collaboration aims at bringing two independent organizations together in a new structure, where they share the same commitments to conduct the same planning and mission to achieve the same outcomes, and ultimately produce or create something unique [11,13,14]. According to Vangen and Huxham [15], a successful collaboration consists of five perspectives:


These perspectives are necessary for a successful relationship between two organizations and influence their aims, working processes, communication, trust, and accountability [16–18]. Considering these perspectives in civilian-military relations, Shanks Kaurin [19], proposes that the outcome will create five diverse situations when civilian-military populations:


Within healthcare, one way to measure the outcomes of collaboration is to use the acronym CSCATTT used in MIMMS (Major Incident Medical Management and Support) courses [20]. CSCATTT stands for C: Command and control; S: Safety; C: Communication; A: Assessment; T: Triage; T: Treatment; and T: Transport. These factors illustrate medical and non-medical aspects of disaster and emergency management and should be included in healthcare planning and response and synchronized with other organizations, such as the military healthcare, in collaboration to achieve a fruitful outcome.

Applying the described theoretical framework to the elements of CSCATTT may facilitate a unique opportunity to measure and evaluate changes in disaster and emergency management processes over time.

#### *1.2. Civilian-Military Collaboration in Healthcare*

Historically, Civilian-Military Collaboration (CMC) has connected both agencies in various areas, e.g., the pyrotechnics industry and clinical practice, but a formalized collaboration started in educational sectors when military staff established academic carriers in civilian universities [20–24]. Nowadays, an increasing number of public health emergencies, armed conflicts, and disasters, together with the global financial awareness and healthcarerelated technological developments, have enforced new constraints on the healthcare sector, necessitating a new round of CMC collaboration in healthcare [5,25–27]. Lessons learned indicate that CMC should become compatible in both medical and non-medical aspects to achieve desired results [3,4,28–32]. Medical factors that influence the outcomes of CMC

include differences in triage systems, treatment and intervention alternatives, and logistics for patients' evacuation, while non-medical factors encompass differences in command and control and leadership, security, situation assessment, communication, information-sharing, and reporting systems [3,4,33,34]. These aspects are included in CSCATTT as essential elements of emergency management.

The coronavirus 2019 (COVID-19), as well as several others incidents, has resulted in several societal changes and revealed weaknesses and strengths of the current management system [27,35–37]. Political and economic-based decision-making has been one, affecting major public health decisions, preventing the crucial collaborative efforts in implementing public health strategies, and halting healthcare leaders from making unpopular but necessary public health decisions. The lack of proper communication has equally contributed to the failure in achieving the established goal, the inability of information sharing, and disrespect for medical decisions [36–43]. However, this pandemic has also provided a good opportunity to evaluate and compare the current collaboration with that reported in the past [11,40–47]. An evaluation is particularly crucial since disasters and public health emergencies are increasingly impacted by cross-border factors that place increasing demands on society to initiate a broader dialogue of partnership [27,45–48]. Moreover, it is important to not only review experts' publications but also the opinions of the operational populations to identify potential gaps in outcomes and comprehension.

This article attempts to identify the status of CMC before and after the COVID-19 pandemic, based on the aforementioned theoretical background and using CSCATTT, in two steps:


#### **2. Materials and Method**

#### *2.1. The Systematic Literature Review*

A systematic literature review was conducted, using the following search engines; Science Direct, Scopus, PubMed, Web of Science, and Gothenburg University's online library, according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to evaluate the status of CMC (Figure 1) [49]. The quality of each included study was assessed using Health Evidence Quality Assessment Tool (Appendix A) [50].

Inclusion criteria: Original research studies published in English (1995–2021).

Exclusion criteria: Conference papers, abstracts, reports, non-scientific publications.

Search string: Civilian Military Collaboration AND/OR Civilian Military Partnership, separately or in combination. The terms "collaboration" and "partnership" were chosen based on the definition provided by the Oxford dictionary, ensuring the words were synonymous [51].

Two authors assessed all abstracts and titles independently to agree on included and excluded articles. Whenever disagreed, the third author was consulted. After achieving a mutual consensus, the full texts of the studies were reviewed. The reasons for all excluded papers were documented. For included studies, data regarding the name of authors, article title, year of publication, the title of the journal, the used methods, main results, and conclusions were all collected in Appendix B.

**Figure 1.** The process of literature selection according to PRISMA flow chart for new systematic reviews [49].

#### *2.2. Questionnaire Preparation*

The research group with experience from both civilian and military healthcare systems, formulated a questionnaire, consisting of eight questions, representing eight different dimensions, inspired by CSCATTT, i.e., Command and control, Safety, Communication, Assessment, Triage, Treatment, and Transport [20]. The tool identifies the leading strategic, tactical and operational parts of collaboration during emergencies [2,3,11]. Command and control encompassed three parts: administrative, practical, and mutual activities such as issuing directives and recommendations. The remaining five questions concerned safety and security issues, situational assessment, communication, medical issues, and logistics. The final questionnaire included free space to allow participants to include their comments (question 9). All authors reviewed, evaluated, and approved the questionnaire in consensus and based on a combination of clarity, logic, relevance, eligibility, comprehension, and usability. Each defined question could be answered by using a Likert scale, which was marked as 1 to 5, where 1 was defined as a weak collaboration and 5 a strong collaboration, before and during the COVID-19 pandemic (Appendix C).

#### *2.3. Distribution of the Questionnaire*

The research subject was introduced as a discussion topic on ResearchGate (RG), from 1 January to the end of May 2021. RG represents a European social networking site, engaging over 19 million scientists and researchers, being, the largest European academic network in terms of active users. The members of RG actively share papers, ask and answer questions voluntarily. They also find research collaborators [52]. Individuals interested in the topic could, after reading the purpose of the study, click on a link to a google document page and answer questions voluntarily (see "Ethical Considerations"). The use of the RG represents a so-called "virtual snowball sampling," method, which has been used in numerous studies [22,53–55]. The method helps to identify individuals of interest for this research, thus, allowing an increase in the representativeness of the results. It can also, increase the number of responses, is inexpensive, and decreases the sampling time. However, the sample selection is biased toward the characteristics of the online population like gender, age, education level, and socioeconomic belonging [53–55].

#### *2.4. Ethical Considerations*

The survey was anonymous, and all respondents agreed to participate voluntarily by clicking on the available link, where information about their participation and the purpose of the study was provided additionally. No name, affiliation, or other searchable information was registered. Confidentiality was strictly respected during data collection and obtained information was stored in a secure and safe area. The study complied with the ethical guidelines stipulated by Swedish law (SFS 2008:192). In Sweden, ethical approval is necessary if the research includes data regarding participants' race, ethnic heritage, political views, religion, sexual habits, or if it uses health or physical interventions or methods that aim to affect the person physically or psychologically (SFS 2003:460). Since this study did not include any of these aspects and all individuals freely contributed with their views on an available scientific site, it was exempt from ethics approval [22,55].

#### *2.5. Statistics*

Data were transferred to a spreadsheet, where the scores for each question before and under the COVID-19 pandemic were inserted. The mean and SD for each question before and during the pandemic was calculated, and the means were compared by using a *t*-test for the entire study cohort and each involved nation to obtain statistically significant changes, using a GraphPad Prism *t*-test calculator.

#### **3. Results**

#### *3.1. Literature Review*

The term, "Civilian-Military Collaboration", returned over 80,000 hits for all search engines. Using "Civilian-Military Collaboration" AND OR "Civilian-Military Partnership" in combination or separately resulted in a manageable number of references. Of the total 602 publications identified, 306 papers remained after removing duplicates and ineligible publications. These papers were sifted by looking at the abstracts, methods, and aims, and the eligible papers (*n* = 104), were studied in detail (Figure 1). Abstracts and non-relevant papers with no association to the main search key and case studies were removed. The final 43 papers were included and reported in Appendix B.

#### *3.2. The Core Findings of the Review*

There was a lack of consistency in defining civilian-military relations. While words such as coordination, cooperation, and collaboration were used interchangeably, the publication did not necessarily deal with collaboration. Furthermore, the majority of papers did not illustrate the view of practitioners, i.e., individuals who are operationally active in the field and were reviewed or presented the views of strategic level, researchers, or experts in the field.

Although several authors reported the importance of CM cooperation in diverse fields like technological area [7], others reported that such cooperation results in a dual capacity building, which can enhance the integration between CM organizations, creating dual capacity and resource availability as a valuable advantage in prolonged disaster and emergency management [31]. However, besides a considerable cost, such integration causes primarily some confusion regarding the role of each organization in a specific event or activity, including pandemics [6,23,30,45]. It is well recognized that CM collaboration has resulted in advances in medical treatment of injuries, reduction of the number of deaths, and improvement of the Emergency Medical Systems. However, there seems to remain a need for new guidelines and directives to guarantee the benefits of such cooperation for both organizations and to eliminate or at least minimize some of the challenges between the two organizations, such as, in leadership, operative, and logistics partnership [3,4,9,24,46]. One way to make these organizations compatible seems to be mutual educational and training initiatives [25,30,56], which not only synchronize their activities but equally help clear the role and responsibility of each organization, joint operating mechanism, and treatment policies and reduce the organizational tensions that may exist between two populations [57].

The lack of trust has been reported as one of the core arguments for an insufficient engagement between two organizations [9,22,28]. Sharing information, planning, and developing a mutual administrative working activity, may however help to increase the trust between two organizations and enhance the development of a valuable partnership in all aspects of integration. Within healthcare, there are several contact points between civilian and military organizations [3,4,22,28]. Following the CSCATTT acronym, there is a need for synchronization between these two populations in leadership, safety, and communication issues to achieve mutual assessment of the situation. Such synchronization enables both organizations to achieve and obtain the advantages of dual capacity in staff, stuff, space, and system, i.e., all crucial elements of surge capacity, and in the outcomes of treatment and survival [3,4,32,35,58,59].

Although educational initiatives and training courses—besides other types of planning programs—are associated with a cost [60], they enable a multiagency collaboration that encompasses all agencies and not only, CM. These initiatives clarify the roles, increase the skills, and pave the way for achieving an established goal, individually and as a team [8,55,61]. Additionally, they may prompt agencies, especially CM, closer to creating one organization with responsibility for the development of all involved entities and in an all-risk scenario pattern [36,48,62], demonstrating practical, financial, and political advantages of such collaboration [44,63].

One significant advantage of CM collaboration is what both organizations can learn from each other. The collaboration aims at generating the same goal and such partnership requires opportunities that enhance learning of each other's limitations and capabilities [64,65], which also eases up and enables better resource and capacity sharing. The ideal collaboration should be developed through time [29,66]. Long-term development of such collaboration promotes and offers opportunities of creating one organization with responsibility for all development, education, and administration. Such an organization might be a necessity at the time of war and armed conflicts [28,55,59,67,68] to address all aspects of collaboration, socially and politically and in several levels: nationally and internationally. Consequently, increasing the trust needed for implementing delicate measures and making crucial decisions [41,42,69,70], without allowing one organization to be superior to the other [43,71].

A mutual organization may additionally provide other important elements of relief operations, considering cultural and linguistic understanding, human rights promotion, community-based needs assessment, besides role identification, team working and communication [28,47,55,59,72].

In summary, most publications emphasized the significance of the civilian-military partnership, prominently in how military support was incorporated in the national response, including support to national health systems, military repatriation and evacuation, and support to wider public systems. Additionally, the majority of studies suggest that collaborative educational initiatives in disaster medicine, public health and complex humanitarian emergencies, and international humanitarian law, along with advanced training in competency-based skill sets, should be included in the undergraduate education of health professionals. Finally, the most common CMC reported in the works of literature were in the fields of logistics and trauma. Other fields for CMC collaboration, e.g., infectious diseases, were poorly investigated [3,4,6–9,20–25,35,36,41–48,51–71,73,74].

#### *3.3. Survey Results*

A sum of 166 respondents answered the optimized questionnaires from the following 19 countries: Australia (2), Belgium (32), England (3), France (1), Germany (1), Greece (4), Iran (1), Italy (2), Israel (1), Mexico (1), Netherlands (3), Norway (3), Romania (8), Saudi Arabia (2), Sri Lanka (1), Sweden (11), Poland (80), Thailand (3), and the United Kingdom (3). Four respondents did not contribute their country of origin. All responses were sorted into four different groups for statistical analysis. Besides countries with over 10 participants (i.e., Belgium, Poland, and Sweden), all other nations, including responses with no country name, made up the fourth group called "others." Table 1 shows the age and gender distribution of all respondents. In total 128 respondents were physicians and 38 were other professionals, including nurses, psychologists, trainees, and strict military staff. The majority of participants were between 41–50 years of age, followed by 34 between 41–50 years of age, and 27 respondents with ages between 51–60 years. The number of male participants was twice that of females.

**Table 1.** Shows the gender and age distribution of respondents in this study.


The collected results were analyzed using qualitative research methods. After identifying the thematic contents, they were categorized into core contents. The representative statements were outlined at a point where no novel information was retrieved from the data [75].

#### *3.4. Changes in CSCATTT*

For all respondents, there was an increase in the mean number of all CSCATTT dimensions under the COVID-19 pandemic. However, these changes were not statistically significant for any of the dimensions (Figure 2). Looking at the individual countries, none of the countries with more than 10 participants demonstrated any statistically significant increase in collaboration before and after the COVID-19 pandemic. Nevertheless, the results obtained solely from Belgium displayed a possible tendency toward significance in dimensions 2 (practical interface in command and control), 4 (safety and security), and 8 (transport). In Poland, only dimension 5 (communication and information), and in Sweden only dimension 1 (administrative part of command and control) showed a tendency to a significant increase.

Finally, the group called others did not demonstrate any statistically significant increase, although some of them such as the UK showed a very high numeric increase (Figure 2).

**Figure 2.** Shows the differences in collaboration before and during COVID-19 for all countries, and some specific countries with a larger number of participants displayed as mean. It also shows the *p*-value for each significant change (CI 95%), where the light bars are the values before and darker (black bars) are values after the COVID-19 pandemic.

#### *3.5. Comments*

There were over 50 comments. All comments were grouped into four categories of No Collaboration (29/49 = 59.2%), Some Collaboration (12/49 = 24.5%), Full Collaboration (3/49 = 6%), and others (5/49 = 10.3%). In the latter group, two comments were technical and questionnaire-related (Table 2). Some of the comments were not about the subject but an expression of general dissatisfaction with slow progress in a specific country or the political interference with no results.

**Table 2.** Comments given by respondents were categorized into four groups.


**Table 2.** *Cont.*


that several in my (civilian) organization lacked this.


#### **4. Discussion**

Although the necessity of CMC in the management of evolving health crises has been reported and discussed in several publications [3,4,6,22–25,30–33,35,36], this study confirms the need and significance of CMC but fails to illustrate any significant improvement during the COVID-19 pandemic. The results from the survey may suggest possible improvements in some strategic areas, while the practical collaboration (e.g., training and operative engagement) remains missing or unchanged (e.g., logistics).

Theoretically, a successful CMC should encompass several perspectives, which are not completely visible during the current COVID-19 pandemic, globally, indicating that some nations may have a long way ahead to achieve an improved collaboration [16–18]. Although substantive outcomes and the more proficient use of resources represent a mutual target and may raise some awareness, there is still separate funding for both organizations and financial advantages might be a possible cause of collaboration. The current CMC may thus lack a political consensus and framework as cited by one of the participants;

*Participants 1: To my knowledge, there are vast differences in both organizational culture as well as planning and leadership methodology between civilian and military professions. To merge these two organizations to a certain degree, there should first be a framework grounded in political consensus, which is lacking at present time. As long as this first crucial step is not agreed upon, there will be no long-lasting collaborative structures being built. Then and only then agencies of the state may find cooperative areas to endorse. There may be two ways to perform this: either through time-consuming legislation or by the foundation of a new state agency being the major responsible actor in this process.*

There are different definitions of high productivity, probably due to diverse definitions of what collaboration is. The Oxford Dictionary [51] offers the following definitions: Collaboration is the act of working with another person or group of people to create or produce something. Cooperation is the fact of doing something together or of working together towards a shared aim. Finally, coordination is the act of making parts of something, groups of people, etc. work together in an efficient and organized way. While the literature seems to deal with reports of successful cooperation, few publications describe a unique production of CMC.

Emergent milestones are partly lacking. There are some joint events but practical collaboration with a mutual target, when both organizations may share a benefit barely exist. In most cases, military healthcare assists the civilian partner; there might be a different outcome if civilian healthcare is asked to assist the military partner in an armed conflict while confronting a constrained system with overloaded emergency departments [59,72–74]. Several participants in the pilot survey expressed their views;

*Participants 2: Communication has improved, but before COVID-19, we have some annual meetings. Now there is no coordination. They (military) used to "help" as a guard, which could be done by local guards' units*.

There has been broad recognition of CMC during the current pandemic, mainly from strategic sources, while a few operational participants, in this study, declared their sincere pride to highlight their successful collaboration in a compatible organizational culture. Thus, affecting communication, trust, accountability, and consequently the outcomes of collaboration. As mentioned by Shanks Kaurin [19], civilian-military populations may share the same values but have a conflicting understanding of a situation, and different priorities while sharing the process, as cited below;

*Participants 3: We have plans we have big dreams and nothing has changed.*

*Participants 4: In my humble opinion, the collaboration between civil and military medical services is severely lacking. A widely recognized and supported permanent collaboration platform would be ideal but, if it even exists, it is lacking the prominent position it would deserve.*

The differences between participating countries in survey data may indicate a lack of unified definition, diverse social and historical background, and nation's involvement in earlier conflicts [22,47,50,58,66,68]. The prominent changes in this study were chiefly within the administration of the command-and-control section, while the logistics cooperation was unchanged. These coordinating and cooperative activities aim at achieving collaboration but may not necessarily target similar goals and outcomes [19].

*Participants 5: Before COVID-19, we did not have much practical experience with CM collaboration. There wasn*´*t much during COVID-19 either, but there was a clear ambition and progress. I find that the military part had more understanding for the civilian, compared to the opposite. My own experience and CM collaboration made my collaboration good (importance of the network and understanding), but I found that several in my (civilian) organization lacked this.*

In some countries, e.g., Belgium, a mutual production of guidelines and instructions, safety and security considerations, communication and situational assessment, might indicate the first steps for a collaboration, however, defined by the Oxford Dictionary [51], achieving a collaboration, shared outcomes and goals, and establishing a control mechanism to ensure the operational outcome, are mandatory. Some countries with a few participants claimed a higher civilian-military collaboration level. The UK appears to be enjoying a fruitful and continuous interagency collaboration. Sri Lanka and Morocco also report prevailing collaboration between civilian and military healthcare systems. These successful collaborations may depend on previous involvement in international or national armed conflicts, which may necessitate such partnership or an apparent and continuous interest from the government. On the opposite, in countries such as Poland, there seems to be no trustful relationship between government and involved organizations, indicating the negative impacts of political interference in medical decision-making [27–29]. Nevertheless, there are not enough respondents from these countries to achieve a statistically significant result.

*Participants 6: Being engaged in 30 years of civil war, there has been a good CM collaboration as a necessity for wartime military injury burden. However, before COVID-19, after the war, there was no pressing demand for CM collaboration. During the COVID, the ministry of health used military assets effectively by formulation of joint operational*

#### *command comprising of both Director General Health Services and Commander to the Army.*

There are several essential factors for a successful partnership in disaster and emergency management. Factors such as relation-building focus on mutual learning and information sharing, bilateral and multilateral agreements, comprehending the concept of CMC, trust, and mutual practical exercises, were all crucial elements of such partnership [8,9,30,31,56–58,62]. These conditions seem to be met in countries, such as the UK, while lacking in other European countries, such as Poland. A recent literature review, targeting six European countries reported that the most prominent partnership in these countries during COVID-19 consisted of incorporation of military support into the national COVID-19 response, e.g., support to national health and broader public systems, and military repatriation and evacuation [71], confirming the supportive role of the military in CMC, but no real collaboration.

A fruitful CMC depends on organizations' mutual values, situational interpretations, priorities, processes and moral principles [2,22,63,74,76,77]. Since a fruitful and strong collaboration relies on a homogenous and synchronized relationship as well as compatible ethics, the goal in a collaboration should be having shared values and interpretation towards the same goal. The diverse responses from respondents included in this study regarding CMC dimensions indicate a difference between their perceptions compared to that of authorities, which also calls for the evaluation of ethical views in CMC. Firm leadership, collaboration, coordination, and decision-making are all crucial for planning, executing, and harmonizing all efforts needed for successful crisis management [3,4,64,77]. In opposition to previous studies, the current study may indicate that COVID-19 has offered new opportunities for a fruitful collaboration in command and control between military and civilian authorities [78–81]. Increasing administrative measures demand good communication to improve and enable situational awareness and assessment, resource distribution, technological development, practicing decision-making, and information sharing and provides new incentives for educational initiatives, and training [78,80,82]. An improved administrative meeting for mutual planning during the current pandemic inevitably has resulted in improved understanding of each other's abilities and shortcomings, issuing mutual documents and recommendations, and consequently an increase in partnership for smooth distribution of resources and logistics in some countries. While factors, such as a political will and unity, a trustful political-military-public relationship, transparency, and evidence-based approaches are necessary elements of any collaboration, collaboration should be practiced to allow all involved parties to realize their limitations and capabilities, practicing the crucial decision-making step in an environment where mistakes can be made with no harm [1,2,4,22,29,41,83].

#### **5. Limitations**

One limitation to this study is the small number of respondents in the survey, necessitating a larger population study to achieve greater statistically significant results. The overwhelming majority of respondents came from two specific countries and given the uniqueness of civil-military relations in each country; the generalizability of such results is very limited. Diversities and peculiarities in cultures, national health systems, and CMC attitudes and experiences should be deeply inquired, and taken into strong account when testing and explaining CMC in different countries with diverse institutions.

Another limitation of the study was the use of English in the questionnaire and the search of the literature, which may have created some misunderstandings among participants and limited our search results, respectively.

Furthermore, there might be some doubts about using snowball sampling. However, the method has been used in several studies and is scientifically accepted. CMC may have a greater impact in larger nations with large militaries or in countries, which have built-in CMC into their medical infrastructures. However, even small countries such as Sweden

without independent military healthcare seem to have a good collaboration, while larger countries such as Italy with different systems seem to have lost their routine partnership.

Finally, the use of coordination, cooperation, and collaboration in the literature to define the success and failure of CMC may have limited the results of the search. The use of a defined and united terminology is necessary for future publications.

#### **6. Recommendation**


#### **7. Conclusions**

The COVID-19 pandemic has been associated with several changes and has revealed weaknesses and strengths in the current disaster and public health emergency management system, highlighting the importance of multiagency collaboration, particularly CMC. Although COVID-19 seems to have resulted in some progress in communication, coordination, resource distribution, and information sharing, there is still a need for stronger leadership, organizational closeness, and educational and training initiatives to guarantee a synchronized and well-functioning CMC. These steps are necessary to safeguard the practical partnership, operative management, harmony, and compatibility of CMC and require a political will and perhaps a mutual civilian-military authority.

**Author Contributions:** A.K.-M. provided the main framework, identified primary materials, and collaborated on the writing of the paper. K.G. organized research materials, identified appropriate references, and collaborated on the writing of the paper. A.K.-M., L.J.M., Y.R., F.M.B., and K.G. collaborated on the writing and editing of the paper. F.M.B. edited the final version. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**


**Figure A1.** The protocol used for quality assessment of the included papers according to healthevidence.org, accessed on 14 June 2021.

