Next Article in Journal
Sustainable Education with Local-Wisdom Based Natural Reagent for Green Chemical Analysis with a Smart Device: Experiences in Thailand
Next Article in Special Issue
Experiences of Polish Tourists Traveling for Leisure Purposes during the COVID-19 Pandemic
Previous Article in Journal
A Theoretical Framework for Analysing Technology Transfer Processes Using Agent-Based Modelling: A Case Study on Massive Technology Adoption (AMTEC) Program on Rice Production
Previous Article in Special Issue
The Impact of COVID-19 on Bank Equity and Performance: The Case of Central Eastern South European Countries
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 13
Issue 20
10.3390/su132011146
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Struggling with COVID-19—A Framework for Assessing Health System Performance

by
Iwona Markowicz
and
Iga Rudawska
*
Institute of Economics and Finance, University of Szczecin, 71-101 Szczecin, Poland
*
Author to whom correspondence should be addressed.
Sustainability 2021, 13(20), 11146; https://doi.org/10.3390/su132011146
Submission received: 28 August 2021 / Revised: 28 September 2021 / Accepted: 6 October 2021 / Published: 9 October 2021
(This article belongs to the Special Issue Economic and Social Consequences of the COVID-19 Pandemic)
Browse Figures
Review Reports Versions Notes

Abstract

:
Currently, no guidelines exist on how to evaluate the performance of health systems fighting the SARS-CoV-2 pandemic. Therefore, this study seeks (1) to develop a conceptual framework that would be helpful in the given context, and (2) to test the feasibility of the proposed approach. The framework is conceptualized based on investigating critical dimensions and indicators for the successful design of a method for evaluating the performance of health systems. Subsequently, the taxonomic measure of development is used and a correlational analysis made in order to run a pilot test of the proposed concept. The finalized conceptual framework has five input dimensions (demographical burden, epidemiological burden, health-related quality of life, financial resources, and access) described by 18 indicators, and two output domains (outcomes, productivity) described by six indicators to monitor system performance under the COVID-19 pandemic. The pilot-test conducted in European Union countries and the United States proved the model to be useful and feasible. The proposed framework can be used to assess the performance of health systems fighting novel pathogens, such as SARS-CoV-2, worldwide. Our methodological approach can be used as a benchmark for international agencies such as the World Health Organization in developing their own frameworks. The paper presents the first research exploring the evaluation of a health system during the COVID-19 pandemic. It has the potential to help monitor a health system’s performance during the pandemic by accounting for specific country-related circumstances. In a broader sense, it can contribute to boosting health market competitiveness in terms of quality.

1. Introduction

On 30 December 2019 a cluster of pneumonia cases of an unknown origin in Wuhan City was reported to China’s National Health Commission. The next day, the World Health Organization (WHO) was alerted and on 7 January 2020 the Chinese authorities were able to identify a novel coronavirus as the cause of the pneumonia [1]. Four days later, the first death was reported. By the end of January there had been almost a thousand new cases reported in China (mostly in Hubei Province), accompanied by first diagnoses in Thailand and Japan. Ever since, the COVID-19 pandemic has affected almost all countries globally, putting at risk both their nations and the global economy.
The scope and the scale of the COVID-19 pandemic varies from country to country but, according to the WHO [2], the local epidemics share characteristics. Firstly, the virus demonstrates a very high human-to-human transmission rate and thus presents a capacity for an explosive spread. Because of this, the outbreak has continued to grow internationally at a rapid rate, with cases of secondary transmission confirmed around the world.
By 23 August 2021 over 211 million people had been infected globally, with over four million deaths [3]. The worst situation at the end of August 2021 was observed in the USA, India, Brazil, the Russian Federation, the United Kingdom, France, Turkey, Argentina and Columbia. Regions and countries characterized by high population density rates and high levels of deprivation and poverty have been seen to be hit the hardest (Table 1). The overall situation has stabilized in the Western Pacific and European regions, although intense transmission clusters are still being discovered in a number of countries, overwhelming their national health systems. The Americas have remained the global center of the pandemic since mid-April 2020.
Secondly, SARS-CoV-2 has a potentially detrimental effect on human health. The virus can cause severe illness (acute respiratory disease) and death, although approximately 80% of the cases appear to be mild [1]. The severe and critical cases are mostly observed among the older patients and the chronically ill, putting at risk the aging populations and those with multimorbidity.
Thirdly, the COVID-19 pandemic has caused a shock not only to human health and national health systems, but also to the economies of individual countries and the global economy. The widespread business shutdowns have impacted economic activity. Although the exact magnitude of their impact on GDP growth is very difficult to quantify, these measures have clearly led to sharp reductions in output, corporate investment, international trade and household spending [4]. The lockdown imposed on most national economies has mostly affected output in retail and wholesale trade and in professional and real estate services. The affected sectors account for between 30 and 40 percent of total output in most economies. According to OECD estimations [4], the overall direct initial hit to GDP has typically ranged between 20–25% in the G7 countries.
The aforementioned epidemic and socio-economic consequences of the COVID-19 pandemic call for a response strategy that will address public health issues and economic policy challenges. That strategy must involve ongoing performance measurement in order to make more informing choices and set policy priorities. Therefore, the purpose of this work is to develop a framework for evaluating the performance of health systems affected by SARS-CoV-2 and to examine the feasibility of the generated indicators through a pilot test. The paper is structured as follows: the first section highlights the rationale for performance evaluation of public health sectors; this is followed by a presentation of the original conceptual framework and a wide range of potential indicators; and, finally, the feasibility of the proposed approach is tested. The finalized framework is developed using the taxonomic measure of development (TMD) and a correlation analysis based on Pearson’s and Spearmen’s coefficients.

2. Theoretical Background

The idea of the need to evaluate the performance of public health sectors can be traced back to New Public Management (NPM). NPM was developed during the 1980s, and since then many governments and public sector entities have been adopting this approach in order to boost the effectiveness, efficiency and accountability of the public sector [5,6]. In this context, there is a growing recognition of the crucial role played by performance evaluation in health care [7,8]. Performance evaluation is thus considered as a central activity leading to improvements in health systems and upgraded rationality in decision-making. Consequently, it means that there is a need for reporting comparative information regarding national health systems based on meaningful and fair metrics. Given the complexity of health care, this task poses a substantial challenge [9]. The challenge is related to the fact that health care services are influenced directly and indirectly by a huge spectrum of elements, shaped by both professionals and organizations responsible for health care delivery [10]. If we additionally consider that the final output of care, i.e., population health, is strongly contextual in terms of demography and epidemiology, the evaluation process appears as a very complex and comprehensive one. Some studies have already pointed to such difficulties when developing a performance evaluation framework in public administration [11,12].
As regards the health care sector, many authors have focused on adopting comprehensive performance measurement frameworks such as the balanced scorecard and benchmarking at the organizational [13,14,15], regional [16], national [17,18] or international levels [19,20,21]. Other authors have looked beyond the tools already well-established in the management literature and proposed original conceptual frameworks and models to be used by public health care authorities [22,23,24]. At the same time, international organizations, such as the WHO [25,26] and the Organisation for Economic Co-operation and Development (OECD) [27], have proposed their own approaches in order to help countries in building effective tools for international comparisons.
The main idea behind the concept of performance refers to achieving the objectives of the health system, namely improving health, enhancing responsiveness and assuring fairness in financial contribution [28]. The first of these goals is usually defined in terms of outcomes, the second in terms of ethical issues and client orientation, and the third in terms of financial risk protection against incurring health care costs.
There are several dimensions that have been employed to describe system performance, among which the most common are effectiveness, efficiency, quality and equity [27,29]. The most popular assessment frameworks introduced by the WHO and by the OECD apply different configurations of these dimensions. Their contents and structures are discussed in detail in the literature [30]. The framework released by the WHO indicates five elements of performance, while adopting distribution and the average level as measures to achieve the first two objectives. It was redesigned in 2007 in response to global challenges. Ever since, the emphasis has been put on the functions of the health system, and the new model has been framed in six domains, namely (1) service delivery, (2) information, (3) health workforce, (4) medical products, vaccines and technologies, (5) stewardship, and 6) financing. [31] By comparison, the OECD model underlines the financial aspects of the health system by incorporating the level of health expenditure as a goal [30].
Since 2000, which was when the first WHO report was published, establishing standards for data has remained a major problem for performance evaluation of health systems. The unification of data and the establishment of common performance indicators would help to conduct cross-country comparisons and support reporting at the European Union (EU) level and worldwide. The second major challenge is that of coordinating methodologies. Most scholars agree that the main obstacle to an effective management of performance evaluation systems is posed by the complexity of selecting measures [32,33,34]. Overcoming these obstacles would definitively support health policy-makers and facilitate comprehensive analyses in the public sector.

3. Materials and Methods

The paper aims to (1) develop a conceptual framework that would be helpful to evaluate the performance of health systems struggling with the SARS-CoV-2 pandemic and (2) test the feasibility of the proposed approach.
We posed a research question: does a higher assessment of a country’s health and health system capacity profile give better results in combating the health crisis in that country?
The existing system performance evaluation models can be treated as references, but the novelty of the current epidemiological situation makes their permanent adoption a difficult task. Therefore, we designed a conceptual framework that would help to assess the performance of a health system during the COVID-19 pandemic. The idea behind the proposed framework is quite simple. We assume that there are several elements (variables) that determine the country’s health and health system capacity profile. Health variables pose a certain burden to the functioning of health systems, consequently affecting the size and structure of the demand for healthcare services. On the other hand, a health system has a certain capacity profile that probably affects the way the pandemic is being managed. Finally, both sub-profiles pose a challenge to a health system, resulting in the outcomes. These outcomes can be interpreted both from the epidemiological as well as from the technical point of view.
As the developments during the pandemic are strongly contextual, we decided to include several input variables. They were selected by examining health sector expert and stakeholder opinions and reviewing the relevant WHO recommendations [2]. Finally, the following 17 variables were included: (1) population density (number of people per sq. km); (2) median age; (3) aged 65 and older; (4) aged 70 and older; (5) GDP per capita; (6) cardiovascular disease death rate; (7) diabetes prevalence; (8) female smokers (percentage); (9) male smokers (percentage); (10) hospital beds per thousand population; (11)—life expectancy; (12) healthy life years at birth; (13) share of people with good or very good perceived health (aged 16 and older); (14) healthy life years at age 65; (15) total health care expenditure (in euros per capita); (16) practicing physicians (per 100,000 population); (17) nurses (number per 1000 population); and (18) current health expenditure (% of GDP).
The input variables pertain to the demographic, epidemiological, health-related quality of life, access, and financial dimensions (Table 2).
In the subsequent step, based on experts’ opinions we formulated six output variables which could be considered suitable for system performance evaluation during the COVID-19 pandemic: (1) observed case-fatality ratio (the number of deaths per 100 confirmed cases); (2) deaths per 100,000 population (this represents a country’s general population, with both confirmed cases and healthy people); (3) total cases per million population; (4) total tests per 1000 population; (5) daily tests per 1000 population (7-day smoothed); and (6) COVID-19 Case Recovery Rate (CRR—Recoveries/Confirmed Cases). They can be grouped in two performance domains and six corresponding performance indicators (Table 3).
The proposed model for evaluating health systems during the COVID-19 pandemic consists of five input dimensions and two output domains (Figure 1). Its main goal is to monitor the capacity of a health system and improve its response. In a broad sense, the idea behind the evaluation framework is to contribute to response strategy setting and the management of the global pandemic.
A pilot test was conducted to examine the feasibility of the proposed approach based on the data generated for the selected set of system and performance indicators shown above. We analyzed statistical data collected routinely by:
The data on the coronavirus pandemic were as of 28 June 2020. The analysis covered EU member states (EU-28) and the United States.
As the Open Data Repository is at the developing stage in Poland, we were not able to include the proper link to the database. The research was conducted in two stages. First, a “country health and health system capacity profile” (“the input”—Figure 1) for 29 countries was built. Next, its impact on the outcomes and productivity of the health systems during the COVID-19 pandemic was studied (a health system performance model—“the output”—Figure 1). A “country health and health system capacity profile” was designed based on available data characterizing the situation of the given country in terms of its demographic situation and the capacity of its health system (see Table 2).
In the first stage, the taxonomic measure of development was applied to assess and rank countries according to their health situation. The problem of the health profile is considered as a multivariate phenomenon with many characterizing aspects. The idea behind applying methods for ordering a linear set of objects is to rank them, i.e., to determine the order of objects according to a specific criterion. This particular measure is a synthetic measure of development that can serve as a tool for such ordering. The application of the taxonomic measure of development makes it possible to classify objects described by many features using one synthetic variable. There are multiple ways of constructing this indicator. Four basic scales are distinguished in the theory of measurement: nominal, ordinal, interval and ratio [35]. Systematics of scales refers to transformations which retain relations of the respective scale. These results are well-known and have been presented by various authors, for example [36]. The interval and ratio scales are high [37]. In the present study, the authors applied metric data (interval, ratio). In general, methods are divided into pattern and patternless. In our study the pattern formula was used [38]. Hellwig’s method [39] is one that is based on standardized values of the diagnostic features. The distance—the so-called Euclidean distance measure—between each test object and the established developmental pattern is determined—[40] in the following manner:
d i = [ ∑ k = 1 K ( z i k − z 0 k ) 2 ] 1 2
where:
  • i—the number of the object (country), i = 1, 2,..., N,
  • k—the number of variables (characteristics), k = 1, 2,..., K,
  • zik—the normalized value of the k variable for the i object, z i k = x i k − x ¯ k S k (𝑥ik—value of the k diagnostic variable for the i object; x ¯ k —arithmetic average of the diagnostic variable xk, Sk—the standard deviation from the diagnostic variable xk,),
  • z0k—the development pattern value for the k variable (it takes the maximum value of the variable for the tested objects if the variable is stimulant or its minimum value if it is a destimulant).
The distance di can be used to compare the level of development of the examined objects: the lower the value, the higher the level of development achieved by the given object. In order to normalize the indicator’s value, the relative taxonomic measure of development (TMD), also known as the Taxonomic Measure of Investment Attractiveness (TMAI) [41], can be applied:
z i = 1 − d i d 0
where: d 0 = d ¯ + a S d , d ¯ = 1 N ∑ i = 1 N d i (the arithmetic average of the designated distances from the development model), S d = [ 1 N ∑ i = 1 N ( d i − d ¯ ) 2 ] 1 2 (the standard deviation from the determined distances from the development model), the value of a is assumed as 2 or a maximum value of di (as assumed in the present study).
This measure is standardized and assumes values from the range: (0, 1). Higher values indicate a higher level of development of the examined objects. The measure replaces a description of the examined objects with a description of a single aggregate value. This method is often used to assess the level of economic development of countries and regions [42,43].
In the present study, 28 EU countries and the United States were examined for the 18 diagnostic variables listed in Table 2. Variables No. 5 and Nos. 10–18 were considered as stimulants, i.e., the higher their values, the better the situation. The other variables were deemed as destimulants, i.e., the lowest possible values were preferred.
In the second stage of the study, correlation coefficients were used to examine the impact of the “country health and health system capacity profile” (“the input”—Figure 1) on the performance of the given health system during the COVID-19 pandemic (“the output”—Figure 1):
  • Pearson’s coefficient—to measure the relationship between the values of the taxonomic development measure and the values of selected variables,
  • Spearman’s coefficient—to measure the relationship between the country ranking positions according to the “country health and health system capacity profile” and selected variables.

4. Results

A “country health and health system capacity profile” for the 29 countries was built using the TMD according to formula 2. There were 18 diagnostic variables (the I1–I18 indicators listed in Table 2) used. The main characteristics of these variables are shown in Table 4.
The countries were ranked from the highest to the lowest value (Figure 2). It turned out that, taking into account selected variables, the best health status of the population and the best health system capacity were primarily identified for Ireland, the United States, Sweden and Luxembourg.
Indicators (as shown in Table 3) characterizing the performance of a health system during the COVID-19 pandemic were adopted. The correlation coefficients were measured between two conceptualized parts of the model (the input and the output as in Figure 1), i.e., between the TMD and the variables O1–O6.
The values of Pearson’s and Spearman’s correlation coefficients were similar but not identical (Table 5). This was due to the unequal distances between the values of the synthetic measure of development for the individual countries (as shown in Figure 2).
The results (a positive relationship for the variables O1–O5) can be interpreted as follows—the better “the country health and health system capacity profile” of the country, the higher the values of the variables: observed case-fatality ratio—O1, deaths per 100,000 population—O2, total cases per million population—O3, total tests per 1000 population—O4 and daily tests per 1000 population (7-day smoothed)—O5.
It should be noted that the impact of the country health profile on the O1 variable was negligible. The number of deaths per 100 confirmed cases were actually independent of the national health situation. However, it turned out that the countries demonstrating a better (higher) health profile also performed more SARS-CoV-2 tests (Pearson’s coefficient for variables O4 and O5). Unfortunately, there were also proportionately more incidents and COVID-19 deaths (the variables O2 and O3) there. The relationship between the TMD and the variable O6 was low and negative, and statistically insignificant. Therefore, we cannot make the case recovery rate dependent on the level of the health profile.
The higher COVID-19 incidence and death rates of (variables O2 and O3) and the lower case recovery rates (variable O6) in the countries boasting better health profiles may have been related to higher numbers of tests performed (variables O4 and O5) and more people being hospitalized in them. This made the statistics more accurate (the number of patients, the causes of death). The same was indicated by our analysis of the correlations between the individual variables of the model input (I1–I18) and the variables of the model output characterizing health system performance (O1–O6). The highest values of Pearson’s coefficient (>0.7) pertained to the correlations between the variables I5–O3, I5–O5 and I15–O3. This means that the higher the GDP per capita and total health care expenditure, the more total cases per million population were diagnosed. Moreover, the higher the GDP per capita, the more daily tests per 1000 population were performed.

5. Discussion

Although COVID-19 has been around for some time now, there is a limited number of related research publications. The studies made so far have explored various aspects of the pandemic. For example, Kufel [44] indicates usefulness of the econometric model for predicting the dynamics of COVID-19 cases at the different stages of the pandemic, i.e., its development, the peak level of daily cases, and its decline. Other studies have analyzed the adverse effects the pandemic has had on the economy [45]. The impact of the COVID pandemic on a research agenda of health policies has also been studied [46], as well as country policy responses and citizens’ perceptions in the EU during COVD outbreak [47]. Other authors have also stressed the importance of drawing lessons from the situation experienced [48], emphasizing the need to test the responsiveness and resilience of health systems. The model framework that we proposed can be seen as such. Considering the existing health sector performance evaluation frameworks, none of them refer directly to the system performance indicators during Sars-CoV-2 pandemic. However, WHO has already established two valuable conceptual models in order to cope with a pandemic, i.e., JEE (Joint External Evaluation) [49] and (Pandemic Influenza Preparedness and Response Plan) [50]. Both documents were produced in 2005 and address the issues of country monitoring, evaluation and recommendations for national measures before and during pandemics.
The Joint External Evaluation—International Health Regulations entered into force in 2007. The JEE Tool focuses on the assessment of country capacity to prevent, detect, and rapidly respond to public health threats independently of whether they are naturally occurring, deliberate, or accidental [49]. Unlike our framework, this one is intended to provide health policy performance indicators within three key areas: (1) prevention (including national legislation, policy and financing), (2) detection (including national laboratory systems, disease reporting according to WHO requirements, and workforce development), and (3) responsiveness (including preparedness, emergency response operations, a national framework for transferring medical countermeasures, and risk communication). Thus, the JEE/IHR represents more of a health policy approach, whereas ours is more of a health system approach.
On the other hand, PIPP concentrates on recommended actions before, during and after a pandemic of influenza. These actions are organized into the five basic components, i.e.,: (1) planning and coordination, (2) situation monitoring and assessment, (3) reducing the spread of diseases, (4) continuity of health care provision, and (5) communications [50]. Comparing our evaluation framework to PIPP, one can see the similarities in the area of pandemic surveillance systems, which in both cases include a case fatality rate as a potential health indicator of severity. Our model complements the set of indicators describing the outcomes with additional ones, namely: deaths per 100,000 population due to COVID-19, total COVID-19 cases per million population and COVID-19 Case Recovery Rate.
Moreover, following the Global Health Security Agenda (GHSA) [51] launched in 2014 and refined in 2018 [52] it is worth highlighting that this approach emphasizes the need for strong multisectoral engagement, including human and animal health, security, agriculture, defense, law enforcement, foreign affairs, development assistance, research, and finance sectors, among others [52]. Comparing our approach to those mentioned above one can draw the conclusion that our framework basically seeks a health system approach with less attention to a disease-specific or all-hazard approach already addressed in GHSA and JEE/IHR.
The literature also highlights the problem of patient cost burden and the willingness to pay for health services. This issue might be of great importance while discussing health care coverage and delivery during the global health crisis (pandemic). Vuong Q.H. et al. evaluated the sensitivity of Vietnamese healthcare consumers against two groups of factors (demographic and socioeconomic-cognitive) regarding payment for periodic GHE (general health examinations) which are not covered by insurance [53]. A paper by Vuong Q.H. Ref. [54] represents the first research attempt to estimate the probabilities of Vietnamese patients falling into destitution due to financial burdens occurring during a curative hospital stay. Such studies tend to focus on Asia and developing countries. We did not address this issue in our research, which can be seen as a study limitation. The working and unemployed are insured in Poland within the system of universal social insurance (employed), and to some extend within state/regional budgets (unemployed). They receive health benefits in life-threatening situations as well as primary health care services.
The results obtained in our study could not be directly compared to the findings of other authors. The reason is that there is a lack of up-to-date studies that would cover exactly the same issues i.e., health system performance under the COVID-19 pandemic. Furthermore, existing frameworks proposed by WHO and the GHSA seem to be insufficient and incomplete in terms of the wide range of problems generated by pandemics (such as setting priorities in accessing and using care, mapping out the role of primary care physicians, cost burden reduction etc.). Our proposal is also not universal, but can be perceived as a first attempt to fit the model framework to the contextual conditions of care in a pandemic.

6. Conclusions

SARS-CoV-2 is a new pathogen, with the COVID-19 pandemic caused by the virus having affected regions around the world. National health systems are responding to this challenge by aiming to ensure safety and provide appropriate health care to their citizens. Regardless of the circumstances, public policy makers must be prepared to evaluate health system performance based on accurate and reliable data.
The present study is the first attempt to conceptualize an evaluation framework and test it. It contributes to the performance measurement methodology. The study has the potential to help monitor the performance of a health system during the pandemic by accounting for specific country-related circumstances.
The proposed framework can be used to assess the performance of health systems tackling SARS-CoV-2 worldwide. Our methodological approach can constitute a benchmark for international agencies such as the WHO in their efforts to develop their own models. Nevertheless, as the situation caused by the COVID-19 pandemic is a dynamic one, research needs to be revised and developed to account for new input and output variables.

Author Contributions

Conceptualization, I.R.; Methodology, I.M. and I.R.; Data Curation, I.M.; Analysis of model results, I.M. and I.R.; Validation, I.M.; Writing of the original draft, I.M. and I.R.; Writing revised versions & editing, I.M. and I.R. Both authors have read and agreed to the published version of the manuscript.

Funding

The work was supported by the Minister of Science and Higher Education’s program titled “The Regional Excellence Initiative” for the years 2019–2022, project number 001/RID/2018/19.

Data Availability Statement

Data are available upon request to authors.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. World Health Organization. 2019 Novel Coronavirus (2019-nCoV): Strategic Preparedness and Response Plan. Available online: https://www.who.int/docs/default-source/coronaviruse/srp04022020.pdf?-ua=1 (accessed on 5 June 2020).
  2. World Health Organization. WHO COVID-19 Preparedness and Response Progress Report (CH). Available online: https://www.who.int/publications/i/item/strategic-preparedness-and-response-plan-for-the-new-coronavirus (accessed on 1 June 2020).
  3. World Health Organization. WHO Coronavirus Disease (COVID-19) Dashboard. 2020. Available online: https://covid19.who.int (accessed on 21 October 2020).
  4. OECD. Evaluating the Initial Impact of COVID-19 Containment Measures on Economic Activity. 2020. Available online: https://read.oecdilibrary.org/view/?ref=126_126496evgsi2gmqj&title=Evaluating_the_initial_impact_of_COVID-19_containment_measures_on_economic_activity (accessed on 10 June 2020).
  5. Hood, C. Contemporary public management: A new global paradigm? Public Policy Adm. 1995, 10, 104–117. [Google Scholar] [CrossRef]
  6. Lapsley, I. Accounting and the New Public Management: Instruments of Substantive Efficiency or a Rationalising Modernity? Financ. Account. Manag. 1999, 15, 201–207. [Google Scholar] [CrossRef]
  7. Smith, P.C.; Mossialos, E.; Papanicolas, I.; Leatherman, S. Performance Measurement for Health System Improvement; Cambridge University Press: Cambridge, UK, 2010. [Google Scholar]
  8. Papanicolas, I.; Smith, P.C. Health System Performance Comparisons; Open University Press: London, UK, 2013. [Google Scholar]
  9. Smith, P.C. Performance Measurement in Health Care: History, Challenges and Prospects. Public Money Manag. 2005, 25, 213–220. [Google Scholar] [CrossRef]
  10. Levesque, J.-F.; Sutherland, K. Combining patient, clinical and system perspectives in assessing performance in healthcare: An integrated measurement framework. BMC Health Serv. Res. 2020, 20, 1–14. [Google Scholar] [CrossRef]
  11. Bouckaert, G.; Peters, B.G. Performance Measurement and Management: The Achilles’ Heel in Administrative Modernization. Public Perform. Manag. Rev. 2002, 25, 359–362. [Google Scholar] [CrossRef]
  12. Love, D.; Revere, L.; Black, K. A current look at the key performance measures considered critical by healthcare leaders. J. Health Care Financ. 2008, 34, 19–33. [Google Scholar]
  13. Yang, M.-C.; Tung, Y.-C. Using Path Analysis to Examine Causal Relationships Among Balanced Scorecard Performance Indicators for General Hospitals. The Case of a Public Hospital System in Taiwan. Health Care Manag. Rev. 2006, 31, 280–288. [Google Scholar] [CrossRef] [PubMed]
  14. Fottler, M.D.; Erickson, E.; Rivers, P.A. Bringing Human Resources To the Table. Utilization of an HR Balanced Scorecard at Mayo Clinic. Health Care Manag. Rev. 2006, 31, 64–72. [Google Scholar] [CrossRef] [PubMed]
  15. Ba-Abaad, K.M. Review of the literature of balanced scorecard and performance measurement: The case of healthcare organizations. Bus. eBull. 2009, 1, 33–47. [Google Scholar]
  16. Kloot, L.; Martin, J. Strategic performance management: A balanced approach to performance management issues in local government. Manag. Account. Res. 2000, 11, 231–251. [Google Scholar] [CrossRef]
  17. Chang, L.-C. The NHS performance assessment framework as a balanced scorecard approach. Int. J. Public Sect. Manag. 2007, 20, 101–117. [Google Scholar] [CrossRef]
  18. Guven-Uslu, P.; Conrad, L. Uses of Management Accounting Information for Benchmarking in NHS Trusts. Public Money Manag. 2008, 28, 239–246. [Google Scholar] [CrossRef]
  19. Smith, P.C. Measuring up. Improving Health Systems Performance in OECD Countries; OECD: Ottawa, ON, Canada, 2002. [Google Scholar]
  20. Wait, S.; Nolte, E. Benchmarking health systems: Trends, conceptual issues and future perspectives. Benchmarking: Int. J. 2005, 12, 436–448. [Google Scholar] [CrossRef]
  21. Arah, O.A.; Westert, G.P.; Hurst, J.P.; Klazinga, N.S. A conceptual framework for the OECD Health Care Quality Indicators Project. Int. J. Qual. Health Care 2006, 18 (Suppl. 1), 5–13. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  22. Hilarion, P.; Suñol, R.; Groene, O.; Vallejo, P.; Herrera, E.; Saura, R.M. Making performance indicators work: The experience of using consensus indicators for external assessment of health and social services at regional level in Spain. Health Policy 2009, 90, 94–103. [Google Scholar] [CrossRef]
  23. Giovanelli, L.; Marino, L.; Rotondo, F.; Fadda, N.; Ezza, A.; Amadori, M. Developing a performance evaluation system for the Italian public healthcare sector. Public Money Manag. 2015, 35, 297–302. [Google Scholar] [CrossRef]
  24. Kim, H.; Jeon, B. Developing a framework for performance assessment of the public long-term care system in Korea: Methodological and policy lessons. Health Res. Policy Syst. 2020, 18, 1–10. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. World Health Organization. The World Health Report: Health Systems: Improving Performance. 2000. Available online: https://www.who.int/whr/2000/en/whr00_en.pdf?ua=1 (accessed on 3 June 2020).
  26. World Health Organization. Health Systems Performance Assessment: Debates, Methods and Empiricism; World Health Organization: Geneva, Switzerland, 2003. [Google Scholar]
  27. Hurst, J.; Jee-Hughes, M. Performance Measurement and Performance Management in OECD Health Systems. In OECD Labour Market and Social Policy Occasional Papers; OECD: Ottawa, ON, Canada, 2001; Volume 47, pp. 1–69. [Google Scholar] [CrossRef]
  28. Murray, C.J.; Frenk, J. A framework for assessing the performance of health systems. Bull. World Health Organ. 2000, 78, 717–731. [Google Scholar]
  29. Bankauskaite, V.; Dargent, G. Health systems performance indicators: Methodological issues. Presup. Gasto Público 2007, 49, 125–137. Available online: https://www.ief.es/docs/destacados/publicaciones/revistas/pgp/49_Health.pdf (accessed on 3 June 2020).
  30. Caunic, R.E. Frameworks and measures for health systems performance assessment. SEA Pract. Appl. Sci. 2019, 7, 205–212. [Google Scholar]
  31. World Health Organization. Everybody’s Business: Strengthening Health Systems to Improve Health Outcomes. WHO’s Framework for Action. 2007. Available online: https://www.who.int/healthsystems/strategy/everybodys_business.pdf (accessed on 3 June 2020).
  32. Evans, S.M.; Lowinger, J.S.; Sprivulis, P.C.; Copnell, B.; Cameron, P.A. Prioritizing quality indicator development across the healthcare system: Identifying what to measure. Intern. Med. J. 2009, 39, 648–654. [Google Scholar] [CrossRef]
  33. Lizarondo, L.; Grimmer, K.; Kumar, S. Assisting allied health in performance evaluation: A systematic review. BMC Health Serv. Res. 2014, 14, 572. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Sun, D.; Ahn, H.; Lievens, T.; Zeng, W. Evaluation of the performance of national health systems in 2004-2011: An analysis of 173 countries. PLoS ONE 2017, 12, e0173346. [Google Scholar] [CrossRef] [Green Version]
  35. Stevens, S.S. Measurement, psychophysics and utility. In Measurement. Definitions and Theories; Churchman, C.W., Ratooch, P., Eds.; Wiley: New York, NY, USA, 1959. [Google Scholar]
  36. Jajuga, K.; Walesiak, M. Standardisation of data set under different measurement scales. In Classification and Information Processing at the Turn of the Millennium; Decker, R., Gaul, W., Eds.; Springer: Berlin/Heidelberg, Germany, 2000. [Google Scholar]
  37. Walesiak, M.; Dudek, A. Finding Groups in Ordinal Data: An Examination of Some Clustering Procedures. In Classification as a Tool for Research; Locarek-Junge, H., Weihs, C., Eds.; Springer: Berlin/Heidelberg, Germany, 2010. [Google Scholar]
  38. Bąk, I.; Markowicz, I. Taksonomiczne mierniki rozwoju w klasyfikacji obiektów (na przykładzie usług kulturalnych). Wiad. Stat. Pol. Stat. 2003, 4, 32–39. [Google Scholar]
  39. Hellwig, Z. Procedure of Evaluating High-Level Manpower Data and Typology of Countries by Means of the Taxonomic Method. In Towards a System of Human Resources Indicators for Less Developed Countries; Gostkowski, Z., Ed.; Papers Prepared for UNESCO Research Project; Ossolineum, The Polish Academy of Sciences Press: Wrocław, Poland, 1972; pp. 115–134. [Google Scholar]
  40. Walesiak, M.; Dudek, A. Selecting the Optimal Multidimensional Scaling Procedure for Metric Data with R Environment. Stat. Transit. New Ser. 2017, 18, 521–540. [Google Scholar] [CrossRef]
  41. Tarczyński, W.; Łuniewska, M. Stability of Selected Linear Ranking Methods—An Attempt of Evaluation for the Polish Stock Market. In Innovations in Classification, Data Science, and Information Systems. Studies in Classification, Data Analysis, and Knowledge Organization; Baier, D., Wernecke, K.D., Eds.; Springer: Berlin/Heidelberg, Germany, 2005. [Google Scholar]
  42. Pietrzak, M.B.; Balcerzak, A.P. Economic development of Polish voivodeships in the years 2010–2014. Application of taxonomic measure of development with entropy weights. Inst. Econ. Res. Work. Pap. 2017, 2, 1–11. [Google Scholar]
  43. Dehnel, G.; Walesiak, M.; Obrębalski, M. Comparative Analysis of the Ordering of Polish Provinces in Terms of Social Cohesion. Argum. Oecon. Crac. 2019, 1, 71–85. [Google Scholar] [CrossRef] [Green Version]
  44. Kufel, T. ARIMA-based forecasting of the dynamics of confirmed Covid-19 cases for selected European countries. Equilib. Q. J. Econ. Econ. Policy 2020, 15, 181–204. [Google Scholar] [CrossRef]
  45. Iacus, S.M.; Natale, F.; Satamaria, C.; Spyratos, S.; Vespe, M. Estimating and projecting air passenger traffic during the COVID-19 coronavirus outbreak and its socio-economic impact. Saf. Sci. 2020, 129, 104791. [Google Scholar] [CrossRef]
  46. Bal, R.; de Graaff, B.; van de Bovenkamp, H.M.; Wallenburg, I. Practicing Corona—Towards a research agenda of health policies. Health Policy 2020, 124, 671–673. [Google Scholar] [CrossRef]
  47. Sabat, I.; Neuman-Böhme, S.; Varghese, N.E.; Barros, P.; Brouwer, W.; van Exel, J.; Schreyögg, J.; Stargardt, T. United but divided: Policy responses and people’s perceptions in the EU during the COVID-19 outbreak. Health Policy 2020, 124, 909–918. [Google Scholar] [CrossRef] [PubMed]
  48. Forman, R.; Atun, R.; McKee, M.; Mossialos, E. 12 Lessons learned from the management of the coronavirus pandemic. Health Policy 2020, 124, 577–580. [Google Scholar] [CrossRef]
  49. World Health Organization. Joint External Evaluation Tool: International Health Regulations (2005). Monitoring and Evaluation Framework. Geneva, Switzerland. 2016. Available online: https://apps.who.int/iris/bitstream/handle/10665/204368/9789241510172_eng.pdf;jsessionid=7664826D3E6DC0F55092FFA810908414?sequence=1 (accessed on 27 July 2021).
  50. World Health Organization. Pandemic Influenza Preparedness and Response: A WHO Guidance Document. Global Influenza Programme. France. 2010. Available online: https://apps.who.int/iris/bitstream/handle/10665/44123/9789241547680_eng.pdf;jsessionid=D9C4B4292381A4A69FC36EF571DC0B19?sequence=1 (accessed on 27 July 2021).
  51. Centers for Disease Control and Prevention. Global Health Security Agenda: Action Packages. Available online: https://www.cdc.gov/globalhealth/healthprotection/ghs/pdf/ghsa-action-packages_24-september-2014.pdf (accessed on 27 July 2021).
  52. Global Health Security Agenda (GHSA). 2024 Framework November 2018. Available online: https://ghsagenda.org/wp-content/uploads/2020/06/ghsa2024-framework.pdf (accessed on 27 July 2021).
  53. Vuong, Q.-H.; Ho, T.; Nguyen, H.K.; Vuong, T.-T. Healthcare consumers’ sensitivity to costs: A reflection on behavioural economics from an emerging market. Palgrave Commun. 2018, 4, 70. [Google Scholar] [CrossRef]
  54. Vuong, Q.H. Be rich or don’t be sick: Estimating Vietnamese patients’ risk of falling into destitution. SpringerPlus 2015, 4, 529. [Google Scholar] [CrossRef] [PubMed]
Sustainability 13 11146 g001 550
Figure 1. The conceptual framework for system performance evaluation during the COVID-19 Pandemic. Source: author’s elaboration.
Figure 1. The conceptual framework for system performance evaluation during the COVID-19 Pandemic. Source: author’s elaboration.
Sustainability 13 11146 g001
Sustainability 13 11146 g002 550
Figure 2. Country health and health system capacity profile (“input”)—a ranking based on the TMD. Source: Authors’ elaboration.
Figure 2. Country health and health system capacity profile (“input”)—a ranking based on the TMD. Source: Authors’ elaboration.
Sustainability 13 11146 g002
Table
Table 1. Confirmed cases and deaths by WHO regions (as at 16 August 2021).
Table 1. Confirmed cases and deaths by WHO regions (as at 16 August 2021).
WHO RegionReported CasesReported Deaths
Americas81,824,7842,074,302
Europe63,790,9341,255,791
South-East Asia40,584,666629,604
Eastern Mediterranean14,112,751257,624
Western Pacific5,918,23482,578
Africa5,497,902130,785
Source: [3].
Table
Table 2. The set of system dimensions and indicators.
Table 2. The set of system dimensions and indicators.
System DimensionIndicatorKey Question Answered
Demographic burdenI1—population density (number of people per sq. km)
I2—median age
I3—population aged 65 and older (percentage)
I4—population aged 70 and older (percentage)		Is the population prone to COVID-19 due to demographic burden?
Epidemiological burdenI6—cardiovascular disease death rate
I7—diabetes prevalence
I8—female smokers (percentage)
I9—male smokers (percentage)		Is the population prone to COVID-19 due to epidemiological burden?
Health-related quality of life I11—life expectancy
I12—healthy life years at birth
I13—share of people with good or very good perceived health (aged 16 and older)
I14—healthy life years at age 65		Is the population resistant to COVID-19 due to good health-related quality of life?
Financial resourcesI5—GDP per capita
I15—total health care expenditure (in euros per capita)
I18—current health expenditure (% of GDP)		What is the system’s potential to manage SARS-CoV-2 in terms of financial resources?
Access (infrastructure and workforce)I10—hospital beds per 1000 population
I16—practicing physicians (per 100,000 population)I17—nurses (number per 1000 population)		What is the system potential to manage SARS-CoV-2 in terms of human and infrastructure resources?
Source: author’s elaboration.
Table
Table 3. The set of performance dimensions and indicators.
Table 3. The set of performance dimensions and indicators.
DomainIndicatorKey Question Answered
Outcomes O1—observed case-fatality ratio due to COVID-19
O2—deaths per 100,000 population due to COVID-19
O3—total COVID-19 cases per million population
O6—COVID-19 Case Recovery Rate (CRR—Recoveries/Confirmed Cases)		What is the dynamic of the SARS-CoV-2 pandemic?
ProductivityO4—total tests per 1000 population
O5—daily tests per 1000 population 		How is the pandemic managed?
Source: Authors’ elaboration.
Table
Table 4. Basic characteristics of the 18 variables for the 29 countries.
Table 4. Basic characteristics of the 18 variables for the 29 countries.
VariablesAverageStandard DeviationMinMax
I1176.99263.8518.141454.04
I242.832.4137.3047.90
I318.732.3113.4223.02
I412.481.978.5616.24
I537,859.1715,026.6518,563.3194,277.97
I6191.0590.5886.06424.69
I76.271.993.2810.79
I824.054.8216.3035.30
I933.188.8918.8052.70
I104.831.732.228.00
I1180.282.5375.0583.56
I1262.015.2152.3072.80
I1367.7710.1844.0087.40
I149.092.904.4015.70
I152708.231916.04493.788452.88
I16364.1475.84237.75550.00
I178.473.323.3717.37
I188.562.385.1617.06
Source: author’s elaboration.
Table
Table 5. Pearson’s and Spearmen’s coefficients for the TMD (the input) and the variables O1–O6 (the output).
Table 5. Pearson’s and Spearmen’s coefficients for the TMD (the input) and the variables O1–O6 (the output).
VariablesPearson’s CoefficientSpearmen’s Coefficient
O10.28450.3103
O20.5606 ***0.6601 ***
O30.7405 ***0.7606 ***
O40.3394 **0.2754
O50.4011 ***0.4842 **
O6−0.2723−0.1030
**—significance level 0.05, ***—significance level 0.01. Note: there was no statistical data available for the variable O4—Cyprus, France, Malta, and for the variable O5—Cyprus. Source: Authors’ elaboration.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Markowicz, I.; Rudawska, I. Struggling with COVID-19—A Framework for Assessing Health System Performance. Sustainability 2021, 13, 11146. https://doi.org/10.3390/su132011146

AMA Style

Markowicz I, Rudawska I. Struggling with COVID-19—A Framework for Assessing Health System Performance. Sustainability. 2021; 13(20):11146. https://doi.org/10.3390/su132011146

Chicago/Turabian Style

Markowicz, Iwona, and Iga Rudawska. 2021. "Struggling with COVID-19—A Framework for Assessing Health System Performance" Sustainability 13, no. 20: 11146. https://doi.org/10.3390/su132011146

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop