Next Article in Journal
Value-Based Pricing and Budget Impact Analysis for Multi-Indication Drugs: A Case Study of Immunotherapies
Previous Article in Journal
Incentive Mechanisms for Carbon Emission Abatement Considering Consumers’ Low-Carbon Awareness under Cap-and-Trade Regulation
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 19
Issue 7
10.3390/ijerph19074102
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Proposal for Handling of Medicine Shortages Based on a Comparison of Retrospective Risk Analysis

by
Bela Turbucz
1,*,
Martin Major
2,
Romana Zelko
1,* and
Balazs Hanko
1
1
University Pharmacy Department of Pharmacy Administration, Semmelweis University, 1085 Budapest, Hungary
2
Department of Oro-Maxillofacial Surgery and Stomatology, Faculty of Dentistry, Semmelweis University, 1085 Budapest, Hungary
*
Authors to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(7), 4102; https://doi.org/10.3390/ijerph19074102
Submission received: 16 February 2022 / Revised: 25 March 2022 / Accepted: 27 March 2022 / Published: 30 March 2022
Browse Figures
Versions Notes

Abstract

:
Introduction: We reviewed and compared current drug shortages and shortage management practices in six selected countries (Hungary, Belgium, Spain, Switzerland, Australia, United States) based on the most comprehensive national shortage databases for each country, for four Anatomical Therapeutic Chemical (ATC) groups, to analyze the criticality of drug shortages across countries and identify best practices in shortage management strategies. Materials and Methods: Countries were selected to cover a wide geographical range of high-income nations where a lack of economic power as a potential source of drug shortages is not observable. ATC groups were selected based on a pre-examination of the databases to analyze groups most often in shortage, and groups where the absence of which could have a severe negative impact on treatment outcomes. The bias originating from the different reporting systems had to be reduced to gain comprehensive and comparable information. The first bias-reducing mechanism was transforming the raw number of shortages into proportion per million people. Secondly, critical cases were classified, and thirdly, critical cases were compared with the Word Health Organization (WHO) Essential Medicine Lists. Results: The results indicate that every European country studied reports significantly higher total and critical shortages per population compared to the US and Australia. Within Europe, Hungary reports the highest number of cases both for total and critical shortages, while Spain has the lowest results in both aspects. While in the US and Australia critical shortages were observable in similar proportions across all ATC groups, in European countries ATC groups of anti-infectives for systemic use (J) and the nervous system (N) were found to account for a notably higher proportion of critical shortages. Current shortage management practices were examined in each country and classified into five groups to identify common best practices. Conclusions: Due to the different characterization of reporting systems, several bias-reducing mechanisms should be applied to compare and evaluate shortages. In addition, European harmonization should be initiated to create mutually acknowledged definitions and reporting systems, which could be the basis of good drug shortage handling practices in Europe.

1. Introduction

Drug shortages can be defined as when the available or calculated demand for drugs does not sufficiently meet the end-user level demand. The issue affects all stakeholders along the pharmaceutical supply chain, ranging from Marketing Authorization Holders (MAHs) through wholesalers to hospitals, community pharmacies, and patients, posing a serious threat to treatment outcomes [1]. Drug shortages are recognized as a global problem, to which local responses are necessary, taking into account regionally diverse factors [2]. However, international coordination and common best practices would be beneficial for more effective shortage management. As a starting point, internationally adopted guidelines and definitions would be useful to compare different regions, as currently these are lacking [3].
Competent authorities (CAs) have the most crucial role in handling shortages: they must ensure the continued volume of pharmaceutical products, and therefore they should always be aware of any interruptions in the supply of products and take actions to mitigate the impact of drug shortages [4]. Competent authorities worldwide took various actions over the past decades to collect, monitor, evaluate, and prevent drug shortages and reduce their impact on the healthcare system [5]. In Europe, the responsibility to avoid shortages is shared between authorities, MAHs, and distributors. According to Article 81 of Directive 2001/83, MAHs and the distributors shall ensure appropriate and continued supplies of the medicinal products to pharmacies and hospitals to address the needs of patients [6]. The European Association of Hospital Pharmacists (EAHP) started investigating drug shortages in 2012 due to increased reports from its members expressing difficulties in sourcing medicine [7]. In the United States (US), President Obama signed the Food and Drug Administration (FDA) Safety Innovation Act in July 2012 [8]. The Act mandated manufacturers to inform the FDA of impending shortages to enable early mitigation of the problem. In Australia, the Therapeutic Goods Administration (TGA) dedicated a website to monitoring medicine shortages in May 2014 [9]. However, drug shortages remain a global issue, also present in most developed countries, and competent authorities should further improve their shortage management practices, as the substitution or absence of safe and effective therapies can compromise medical procedures, cause errors, and have a serious adverse impact on patient care. This is especially important in the case of drugs included on the World Health Organization (WHO) essential medicine list, as their shortage is more likely to have severe negative impacts on treatment outcomes [10].
Drug shortages arise due to several root causes, which can be grouped into economic factors, manufacturing and quality factors, and logistic and regulatory factors [2]. Therefore, a diverse set of actions is necessary to effectively address them [11].
This study performs a comparative analysis of six selected countries to understand how total reported shortages and critical shortages compare internationally, and which ATC groups are mostly affected by critical shortages. The study also reviews shortage management practices already in place to reduce shortages and mitigate their impact, with the aim of identifying best practices.

2. Materials and Methods

To carry out the analysis, countries and respective databases have been selected according to pre-defined criteria. As a next step, databases have been processed focusing on selected therapeutic categories, the shortage of which would have the most severe impact on patient care. Finally, risk assessment has been carried out to identify critical shortages. In the analysis, both total and critical shortages have been compared across countries and ATC groups.

2.1. Criteria for Selecting Countries

Six countries have been selected for the analysis to cover a wide geographical range and allow for diversity. The selected countries are Australia from the Pacific region, the United States from North America, Belgium, Spain, and Hungary from the European Union (EU), and Switzerland, being a European country not part of the EU. As countries use diverse reporting systems and publish different information on shortages, the selection criteria has been developed to filter out countries where not enough details are public to carry out risk assessment and identify the ATC group for each shortage.
The following aspects were taken into account throughout the country selection method:
  • First world countries, where drug shortages are not a result of the lack of available financial resources.
  • Have a profound pharmaceutical industrial background.
  • Publicly available reporting system.
  • Reported shortages must be classifiable according to the Anatomical Therapeutic Chemical (ATC) classification system.
  • Information is public regarding available substitutes to allow for assessment of severity.
  • Discontinued presentations listed separately from current shortages.
The six selected countries satisfy all the above-mentioned criteria and therefore constitute a profound ground for analysis.

2.2. Processing the Databases

To process analogous and comparable information on shortages in the chosen countries, shortage databases of all countries were accessed in the one-week time period between 4 and 9 March 2021. As most databases do not offer the possibility to access detailed data on past shortages but only show current shortages, longitudinal analysis was not possible. In Hungary, Belgium, Spain, and Australia, one national database was available, managed by the competent authorities. In two countries, the United States and Switzerland, two or more databases were available, run by multiple authorities (US) or also by private bodies (Switzerland). In the US, The American Society of Health-System Pharmacists (ASHP) database has been chosen [12], as it presented more details regarding the available substitutes compared to the FDA reporting system. In Switzerland, the Martinelli database was chosen as the national competent authority’s database focuses only on a narrow scale of medicines considered essential by law [13].
Shortages reported because of discontinuation of production, stopped commercialization, or the interruption of commercialization were excluded from the analysis. This was necessary because there was not enough information regarding the reason and the duration of these shortages.

2.3. Therapeutic Categories

Shortages were classified according to the ATC Classification System rules, which classify the active ingredients of drugs according to the organ or system on which they act and their therapeutic, pharmacological, and chemical properties. The World Health Organization Collaborating Centre (WHO CC) controls it for Drug Statistics Methodology [14]. We have chosen the following particular ATC groups in shortage for further examination.
  • C: Cardiovascular system
  • L: Antineoplastic and immunomodulating agents
  • J: Anti-infectives for systemic use
  • N: Nervous system
These four ATC groups have been selected for analysis, as these categories of medications play a significant role in the therapeutic arsenal. The permanent or temporary absence of these medicines would cause a serious impact on patient care and the healthcare system [15,16,17,18]. The selection of these groups has been confirmed with the pre-examination of the data, which showed these ATC groups present are in the highest proportions among all reported shortages.

2.4. Allocation of Risk Assessment in Studied Countries

We have investigated all the posted products “currently in shortage” one-by-one to determine whether the risk is considered acceptable or critical. The different presentations of a particular pharmaceutical ingredient were counted as individual shortages. The aim was to assess the severity and separate critical and non-critical shortages. This is key, as the ultimate purpose of every reporting mechanism and countermeasure is to reduce critical shortages. Table 1 summarizes how critical shortages were identified for each country. In most of the studied countries, authorities did not publish a severity assessment, and therefore, criteria to distinguish critical cases had to be defined. The criterium was if no domestic alternatives were available, the shortage was considered critical, as in this case emergency imports would be necessary. It is general practice in every country that the competent authority suggests a domestic alternative for every drug that is in shortage. If no domestic alternatives are available, the country can perform an emergency import from abroad, but this is considered a high-risk and high-expense solution, which is to be avoided. In Belgium and Australia, the database already included some information regarding the severity, which has been augmented to match the criteria applied for all other countries.

2.5. Comparison of the Extent of Different Shortages

To avoid significant populational bias, instead of comparing nominal shortage numbers, the proportion of critical shortages to the country population was taken into consideration. The ratio was calculated as the number of shortages per million people [23].

2.6. Binomial Probability Tests of Proportion of Critical Shortages across ATC Groups

Data have been analyzed to obtain insights into critical shortages and understand whether the proportion of critical shortages differs across the four ATC groups studied (C, J, L, N). We have used the software “Stata” version 17.0 for the analysis (StataCorp LLC 4905 Lakeway Drive, College Station, TX, USA).
For the purpose of statistical analysis, the variable Critical_BI was defined, which is an indicator variable taking the value 1 for shortages that are critical, and 0 for the shortages that are non-critical. Our null hypothesis was that the proportion of critical shortages will not be significantly different across ATC groups.

3. Results

The number of shortages derived from the databases regarding the above-mentioned four ATC groups have been analyzed both quantitatively and qualitatively. Our aim regarding the quantitative analysis was to compare total and critical shortages per million people. The qualitative evaluation aimed to compare the proportion of critical shortages by ATC groups per million people. The number of total shortages for the four analyzed ATC groups were obtained for the six chosen countries. Figure 1 shows the shortages in the studied countries, including critical ones. Nevertheless, as Table 2 outlines, the characteristics of reporting systems differ significantly across countries, as they are built on different conceptions regarding the reporting obligation. To obtain comparable data and conduct a reliable analysis, three bias-reducing steps have been performed.
The bias-reducing steps were the following:
  • Transforming the data into population-proportionate figures.
  • Filtering out critical cases from all shortages (according to criteria in Table 1).
  • Comparing critical cases with the WHO Essential Medicine List.
These steps were all necessary so that despite the differing reporting systems, some comparative international oversight could still cover drug shortages. The lack of uniform definitions, reporting systems, and severity assessments along unified international criteria creates significant obstacles to any international comparison of shortages. The characteristics of the national reporting systems of the countries under analysis are highlighted in Table 2.
To perform the quantitative analysis of the data, the first two bias-reducing steps were carried out. Data have been transformed into population-proportionate figures for all countries, and critical cases have been filtered out from all shortages according to the predefined criteria described in the Materials and Methods Section of this paper. As a result, Table 3 was developed. Column (d) of Table 3 shows the calculated proportion of total shortages per million people for each country. The highest proportion was recorded in Hungary and the lowest in the US. The average of the European countries for total shortages per million people (d) is significantly higher than non-European figures. The Spanish figure is an outlier compared to other European countries, as shortages per million people (d) are at least 75% less than any other European country. However, comparing shortages per million people across countries still does not afford unbiased results, as the total number of shortages reported can largely differ across countries due to the reporting system in use. To obtain less biased and more relevant results, the proportion of critical shortages per million people has been calculated in column (e). These figures still reflect that the proportion of critical shortages per population is higher in every European country than the non-European figures. The percentage of critical shortages among all reported cases is shown in column (f). The percentage of critical shortages is on average two times higher in Europe than in the US or Australia. This observation is crucial, as the ultimate purpose of every reporting mechanism and countermeasure is to reduce the number of critical shortages in a given country.
Table 4 shows that the proportion of critical shortages is 16.87% across the whole sample. In contrast to our hypothesis, the binomial probability tests performed for each ATC group (ATC group is denoted by ATC_ID) indicated significant differences in the shortage proportions of certain ATC groups. Table 4 displays the result that the observed proportion (Observed p) is significantly different from the expected proportion (Expected p) for 3 out of 4 groups. In therapeutic group C, only 8.6% of total shortages are critical, while group J seems to be the most affected by critical cases (28.5% of observed shortages are critical). The proportion of critical shortages observed is 22.2% for group L and 17% for N, only slightly higher than the average across the sample.
Knowing that the proportion of critical shortages is significantly different across ATC groups, the next aim was to understand whether and how these differences vary across countries, taking into account the differences in population. Figure 2 outlines the proportion of critical shortages across the countries in specific ATC groups. A pattern can be observed regarding the relative distribution of shortages across the investigated ATC groups, except for a few outstanding data points. It is in line with our expectations based on the total number of critical shortages in Table 3 that the proportion of critical shortages per million people was significantly higher in Switzerland and Hungary in each ATC group, even compared to the European average. Similarly, the lowest proportion of critical cases for each ATC group was observed in the US. A pattern that can be observed is that the distribution of critical shortages across ATC groups for the US and Australia is relatively steady, and no ATC group shows significantly higher proportions of critical shortages. The opposite is true for Belgium, Hungary, and Switzerland, where the proportion of critical shortages is outstandingly higher in therapeutic group’s J and N, anti-infectives for systemic use and nervous system drugs. It is apparent that in European countries with high total numbers of shortages per population, these two groups are causing critical pharmaceutical gaps. Our results are in agreement with the surveys from the European Association of Hospital Pharmacists [7] and the American Society of Health-System Pharmacists [32], who state that in the group of anti-infectives, shortages are particularly an outstanding issue due to the high ratio of medication errors, the increasing antimicrobial resistance, the substandard patient outcome, and the lack of development of new antibiotics [33].
As the final step of the analysis, critical shortages were compared to the WHO Essential Medicine List. The results are presented in Table 5. The World Health Organization proposed the concept of essential medicines in 1977, which is a catalogue of every healthcare system’s minimum medicine needs. Essential medicines are those that satisfy the priority healthcare needs of the population. The basic concept is that high-priority drugs should be available as part of a functioning health system for all people, guiding physicians to evidence-based and rational prescribing [34]. Clinical evidence confirms that medicines included in the list can significantly improve patients’ outcomes and their shortage can have a severe negative impact on treatment quality [35]. Therefore, it should be a key priority to keep the shortage levels of medicines deemed essential by the WHO as low as possible.
Table 5 demonstrates what percentage of all critical shortages are included on the WHO Essential Medicine List by country. The goal of every healthcare system should be to keep this ratio as low as possible, as a WHO essential drug being in critical shortage would mean that the country has no domestic alternative to replace the medication and an emergency import is necessary. Table 5 shows that Switzerland and the US are performing best in this aspect, with 36.8% and 45% of critical shortages being WHO essential medicines, respectively. Belgium displays the worst result, with over 90% of its critical shortages being WHO essential. The results show no correlation between the volume of critical shortages per population and the percentage of WHO essential medicines among critical shortages. Countermeasures should be implemented to reduce both the number of critical shortages and specifically, the WHO essential drugs in critical shortage.

4. Discussion

4.1. Comparison of Shortage Management in Examined Countries

The analysis clearly shows that the proportion of critical shortages relative to population size is significantly higher in Europe than in other investigated regions of the world. This affects every ATC group under examination, but anti-infective (J) and nervous system (N) drugs show drastically higher critical shortage levels in Europe. Our findings are in agreement with the study from 2018 conducted by Videau in hospitals, where Switzerland was one of the most and Spain the least affected country by shortages in both studies, and anti-infective medications were the most affected therapeutic group [5]. The critical shortages of medicines on the WHO Essential Medicine List need to be reduced and prevented in particular. European nations need to adopt a wider range of transparent, unified shortage management countermeasures across countries to address drug shortages. This section of the paper reviews the various forms of countermeasures already in place in some of the examined countries. A selection of these could be unified and extended over the whole of Europe. The current forms of countermeasures by country are reported in Table 6 and can be grouped under the six following categories:
  • Compulsory stockpiling
  • Measures for essential medicines
  • Notification responsibility
  • Measures affecting wholesalers
  • Export bans
  • Emergency imports

4.2. Compulsory Stockpiling

In Australia, the National Medical Stockpile must maintain “key medicines” to avoid critical shortages of essential drugs [30]. A similar system exists in Switzerland, where the government defines medications that are subject to compulsory stockpiling in the appendix to Ordinance and the level of such stock that would satisfy average domestic consumption for three months without any import [40].

4.3. Measures for Essential Medicines

In Spain, the government can demand production and commercialization from MAHs to bridge supply gaps concerning essential products [52]. This seems to be an effective measure to cut back shortages, as critical shortages per million people in Spain were significantly the lowest among European countries. The US shortage management system works similarly. In response to the shortage of a medically necessary drug, the FDA can suggest and support establishing a new manufacturing site or even involving a new supplier. If a MAH decides on a voluntary recall, the FDA may conduct risk evaluation and encourage other MAHs to initiate, maintain, or increase production of the drug [46]. They can also facilitate the review of new generic applications that are potential alternatives [46]. The regulation also permits hospitals within the same health facility to repackage drugs into smaller units to alleviate drug shortages [8]. In Spain, MAHs that stop the distribution of a medicinal product without the authorities’ permission face heavy fines [53].

4.4. Notification Responsibility

In the US, authorities maintain an apparent oversight of shortages. They, therefore, can handle them very effectively—the quantitative analysis clearly reflected that the critical shortages per million people are the lowest in the US. The criteria for products reported in the central systems are strict and well-defined, so authorities can monitor developing shortages from an early stage and address them accordingly [12]. For example, MAHs are mandated to inform the FDA of impending shortages six months in advance when they plan to stop producing a single-source or medically necessary (life-supporting, life-sustaining, or intended for use in the prevention or treatment of a debilitating disease or condition, including emergency medical care or during surgery) drug [8]. This effectively helps to maintain a low percentage of shortages that turn into critical. The FDA may also oblige the MAHs to conduct periodic risk assessments to address vulnerabilities in its supply chain.

4.5. Measures Affecting Wholesalers

In Belgium and Spain, some countermeasures specifically target wholesalers, who are obliged to ensure continued and adequate supply. In Spain, all wholesalers are required to deliver within 24 working hours [38]. In Belgium, to reduce the cases where shortages arise due to “distribution problems”, distributors were assigned as full-line and regular wholesalers. MAHs are obliged to supply within a shorter period to full-line wholesalers. Full-line distributors are required to deliver emergency shipments in 24 h. They must have a range of specified medicines in stock to supply the needs of defined geographic areas. Moreover, full-line distributors are only allowed to supply strictly determined wholesalers, domestic pharmacies, and hospitals; thereby, parallel export is not permitted [36].

4.6. Export Bans

In Australia, export can only be performed by MAHs or designated distributors acting on behalf of the MAHs, not by most wholesalers like in many European countries [49]. This is highly important, as parallel export is usually a key factor in causing shortages, mostly affecting countries with low drug prices compared to international averages. In Spain, authorities (AEMPS) can restrict exportation only to medicinal products without therapeutic equivalents [26]. There are serious penalties and fines of up to 1 million euros for distributors who export medicines when this activity has been forbidden.

4.7. Emergency Imports

When there is no other possibility to resolve a critical shortage as no substitution is available domestically, emergency imports must be performed. The conditions of an emergency import are determined by national laws. In most countries, this is contingent on the approval of the competent authority, such as the FDA in the US, the OGYEI in Hungary, or the Spanish Agency of Medicinal Products and Medical Devices in Spain (see in Table 6). In Belgium, wholesalers may also perform emergency imports from the EU based on a doctor’s request, in the specific quantities necessary for the given treatment [39].

5. Limitations

The findings are limited by the high degree of incomparability of data on drug shortages across countries. As reporting criteria and reporting systems used significantly differ in every country examined, it has been hard to develop a comprehensive and comparable overview of shortages in each country (including the number, severity, causing factor, and possible solution of all shortages). The authors are aware that the analysis does not contain deep statistical research to support insights. The cause of this is that only publicly available databases could be recruited, which contain only limited and categorical data. Even the data fragments provided highly differ across countries. Publicly available, internationally uniformed, and frequently updated databases containing numerical data (duration of shortage, cost of alternative therapy, number of patients using medication, effectiveness of particular countermeasures) would be necessary to perform a more accurate statistical analysis on the topic. The criteria which define shortages to be reported towards authorities can make large differences in the data, e.g., in Spain, shortages where “quick solution is available” are not noted in the reporting system, which can significantly alter the results of the cross-country comparison. In Australia, the lower proportion of critical shortages could be explained with the definition complexity as it considers both the number of alternatives and the potential impacts on patients. In Switzerland and the US, multiple databases of shortages are available offering different details, maintained by various authorities (FDA and ASHP) or organizations (Federal Office of Public Health, Swissmedic, Federal Office for National Economic Supply, Martinelli Consulting). This is an issue because large discrepancies among the various databases and varying definitions limit transparent international comparison.

6. Conclusions

The following recommendations should be considered to handle the issue of drug shortages more effectively in Europe. Since drug shortages do not respect borders and cross-country collaboration would be beneficial for more effective shortage management, a unified European definition and reporting criteria of shortages would be necessary to assure internationally consistent monitoring, reporting, comparisons, responses, and solutions. National authorities across Europe should be aware of shortages through coordinated systems, increasing cross-country transparency, and facilitating solutions in every country. It was a significant step forward in the European Union that in July 2019, the EMA published the “Guidance on detection and notification of shortages of medicinal products for Marketing Authorisation Holders in the Union” [54]. The document contains the effort to facilitate the more uniform reporting and communication of drug shortages and create a harmonization “drug shortage” definition. The document “Good practice guidance for communication to the public on medicines’ availability issues” [55] contains communication guidelines for the national authorities and the EMA for patients and healthcare professionals.
The ideal unified definition for critical shortages should positively discriminate products included on the WHO Essential Medicine List, should take into account the number of alternatives, and should determine the exact start and end dates of a shortage. International harmonization should be initiated to create mutually acknowledged definitions and reporting systems, which could be the basis of good drug shortage handling practices in Europe.

Author Contributions

Conceptualization, B.T. and R.Z.; methodology, B.T. and R.Z.; formal analysis, B.T. and M.M.; investigation, B.T.; data curation, B.T. and M.M.; writing the draft, B.T.; review of the draft and supervision, B.H. and R.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. De Weerdt, E.; Simoens, S.; Casteels, M.; Huys, I. Toward a European definition for a drug shortage: A qualitative study. Front. Pharmacol. 2015, 6, 253. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Gray, A.; Manasse, H.R., Jr. Shortages of medicines: A complex global challenge. Bull. World Health Organ. 2012, 90, 158. [Google Scholar] [CrossRef] [PubMed]
  3. Turbucz, B.; Hankó, B. Overview of the causes and management of drug shortages in the United States and in Hungary. Acta Pharm. Hung. 2020, 90, 170–184. [Google Scholar] [CrossRef]
  4. Acosta, A.; Vanegas, E.P.; Rovira, J.; Godman, B.; Bochenek, T. Medicine Shortages: Gaps Between Countries and Global Perspectives. Front. Pharmacol. 2019, 10, 763. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Videau, M.; Chemali, L.; Stucki, C.; Saavedra-Mitjans, M.; Largana, S.; Guerin, A.; Bonnabry, P.; Delhauteur, B.; VAN Hees, T.; Lebel, D.; et al. Drug Shortages in Canada and Selected European Countries: A Cross-Sectional, Institution-Level Comparison. Can. J. Hosp. Pharm. 2019, 72, 7–15. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. European Parliament, Council. Directive 2001/83/EC of the European Parliament and of the Council on the Community Code Relating to Medicinal Products for Human Use 2001/83/Ec. 2021. Available online: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32001L0083&from=en (accessed on 14 March 2021).
  7. Miljković, N.; Gibbons, N.; Batista, A.; Fitzpatrick, R.W.; Underhill, J.; Horák, P. Results of EAHP’s 2018 Survey on Medicines Shortages. Eur. J. Hosp. Pharm. 2019, 26, 60–65. [Google Scholar] [CrossRef] [PubMed]
  8. Susan, T.; Erin, B.; Corby-Edwards, A.K.; Glassgold, J.M.; Johnson, J.A.; Lister, S.A.; Sarata, A.K. The Food and Drug Administration Safety and Innovation Act (FDASIA, P.L. 112-144), Report, February 4 2013; Library of Congress: Washington, DC, USA, 2013; Available online: https://digital.library.unt.edu/ark:/67531/metadc808297/ (accessed on 24 June 2021).
  9. Tan, Y.X.; Moles, R.J.; Chaar, B.B. Medicine shortages in Australia: Causes, impact and management strategies in the community setting. Int. J. Clin. Pharm. 2016, 38, 1133–1141. [Google Scholar] [CrossRef] [PubMed]
  10. Fox, E.R.; Birt, A.; James, K.B.; Kokko, H.; Salverson, S.; Soflin, D.L.; Pharm, B.S. ASHP Guidelines on Managing Drug Product Shortages in Hospitals and Health Systems. Am. J. Health-Syst. Pharm. 2009, 66, 1399–1406. [Google Scholar] [CrossRef] [PubMed]
  11. Ventola, C.L. The drug shortage crisis in the United States: Causes, impact, and management strategies. Pharm. Ther. Peer-Rev. J. Formul. Manag. 2011, 36, 740–757. [Google Scholar]
  12. American Society of Health-System Pharmacists. Drug Shortages: Current Shortages. Database: ASHP. 2021. Available online: https://www.ashp.org/Drug-Shortages/Current-Shortages (accessed on 9 March 2021).
  13. Martinelli Consulting. Database: Martinelli. 2021. Available online: https://www.drugshortage.ch/ (accessed on 6 March 2021).
  14. WHO Collaborating Centre for Drug Statistics Methodology. Guidelines for ATC Classification and DDD Assignment 2013. Oslo, 2012. 2021. Available online: https://www.whocc.no/filearchive/publications/1_2013guidelines.pdf (accessed on 23 May 2021).
  15. Reed, B.N.; Fox, E.R.; Konig, M.; Jackevicius, C.A.; Masoudi, F.A.; Rabinstein, A.A.; Page, R.L. The impact of drug shortages on patients with cardiovascular disease: Causes, consequences, and a call to action. Am. Heart J. 2016, 175, 130–141. [Google Scholar] [CrossRef] [PubMed]
  16. The Economist Intelligence Unit (EIU) Report, Supported by the European Society for Medical Oncology (ESMO) Country Profile: Cancer Medicines Shortages. 2019, pp. 1–8. Available online: https://www.esmo.org/content/download/197324/3552968/1/ESMO-Country-profile-Belgium.pdf (accessed on 18 April 2021).
  17. Griffith, M.M.; Gross, A.E.; Sutton, S.H.; Bolon, M.K.; Esterly, J.S.; Patel, J.A.; Postelnick, M.J.; Zembower, T.R.; Scheetz, M.H. The Impact of Anti-infective Drug Shortages on Hospitals in the United States: Trends and Causes. Clin. Infect. Dis. 2012, 54, 684–691. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Danon, J.J.; Reekie, T.A.; Kassiou, M. Challenges and Opportunities in Central Nervous System Drug Discovery. Trends Chem. 2019, 1, 612–624. [Google Scholar] [CrossRef]
  19. Spanish Agency of Medicines and Health Products. Database: AEMPS. Available online: https://cima.aemps.es/cima/publico/home.html (accessed on 12 March 2021).
  20. Országos Gyógyszerészeti és Élelmezés-Egészségügyi Intézet. Database: OGYEI. Available online: https://ogyei.gov.hu/gyogyszeradatbazis (accessed on 9 March 2021).
  21. Federal Agency for Medicines and Health Products, Unavailable Medicines. Database: FAMHP. 2021. Available online: https://www.famhp.be/en/items-HOME/unavailability_of_medicinal_products (accessed on 4 March 2021).
  22. Therapeutic Goods Administration, Medicine Shortages Information Initiative. Database: TGA. 2021. Available online: https://apps.tga.gov.au/prod/MSI/search (accessed on 8 March 2021).
  23. World Population Dashboard, United Nations Population Fund. Database: UNFPA. 2021. Available online: https://www.unfpa.org/data/world-population-dashboard (accessed on 18 May 2021).
  24. Wet Betreffende de Verplichte Verzekering Voor Geneeskundige Verzorging en Uitkeringen Gecoördineerd Op. Available online: http://www.ejustice.just.fgov.be/cgi_loi/change_lg.pl?language=nl&la=N&cn=1994071438&table_name=wet (accessed on 5 March 2021).
  25. Bochenek, T.; Abilova, V.; Alkan, A.; Asanin, B.; Beriain, I.D.M.; Besovic, Z.; Bonanno, P.V.; Bucsics, A.; Davidescu, M.; De Weerdt, E.; et al. Systemic Measures and Legislative and Organizational Frameworks Aimed at Preventing or Mitigating Drug Shortages in 28 European and Western Asian Countries. Front. Pharmacol. 2018, 8, 942. [Google Scholar] [CrossRef] [PubMed]
  26. Agencia Española de Medicamentos y Productos Sanitarios AEMPS. How Are Medicines and Medical Devices Regulated in Spain. Available online: https://www.aemps.gob.es/publicaciones/publica/regulacion_med-PS/v2/docs/reg_med-PS-v2-light-ing.pdf?x80916. (accessed on 12 April 2021).
  27. Therapeutic Goods Act 1989 (TG Act). Available online: https://www.legislation.gov.au/Details/C2017C00226 (accessed on 2 May 2021).
  28. Cook, J. Therapeutic Goods (Reportable Medicines) Determination December 2018. 2021. Available online: https://www.legislation.gov.au/Details/F2018L01678 (accessed on 3 May 2021).
  29. Australian Government Department of Health. Management and Communication of Medicine Shortages in Australia, A New Protocol. Available online: https://www.tga.gov.au/sites/default/files/consultation-management-and-communication-of-medicine-shortages-in-australia-a-new-protocol.pdf (accessed on 12 April 2021).
  30. Cook, J. Therapeutic Goods (Medicines Watch List) Determination December 2018. Available online: https://www.legislation.gov.au/Details/F2018L01679 (accessed on 5 May 2021).
  31. Act XCV of 2005 on Medicinal Products for Human Use and on the Amendment of Other Regulations Related to Medicinal Products. 2020, pp. 1–76. Available online: https://net.jogtar.hu/getpdf?docid=a0500095.tv&targetdate=&printTitle=Act+XCV+of+2005&dbnum=62&getdoc=1#:~:text=1%20Promulgated%20on%2015%20July,and%20international%20regulations%20and%20recommendations (accessed on 9 April 2021).
  32. American Society of Health-System Pharmacists. Drug Shortages Statistics. Database: ASHP. 2021. Available online: https://www.ashp.org/Drug-Shortages/Shortage-Resources/Drug-Shortages-Statistics (accessed on 10 May 2021).
  33. Miljković, N.; Godman, B.; van Overbeeke, E.; Kovačević, M.; Tsiakitzis, K.; Apatsidou, A.; Nikopoulou, A.; Yubero, C.G.; Horcajada, L.P.; Stemer, G.; et al. Risks in Antibiotic Substitution Following Medicine Shortage: A Health-Care Failure Mode and Effect Analysis of Six European Hospitals. Front. Med. 2020, 7, 157. [Google Scholar] [CrossRef] [PubMed]
  34. World Health Organization. World Health Organization Model List of Essential Medicines: 21st List 2019. Database: WHO. 2021. Available online: https://apps.who.int/iris/handle/10665/325771 (accessed on 11 May 2021).
  35. Hogerzeil, H.V. The concept of essential medicines: Lessons for rich countries. BMJ 2004, 329, 1169–1172. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. MARS 1964. Loi sur les Médicaments. Available online: https://www.ejustice.just.fgov.be/cgi_loi/change_lg.pl?language=fr&la=F&cn=1964032530&table_name=loi (accessed on 12 March 2021).
  37. Belgisch Moniteur Staatsblad Belge. Available online: http://www.ejustice.just.fgov.be/cgi/welcome.pl (accessed on 26 March 2021).
  38. European Commission Health and Food Safety Directorate-General Summary of Responses to the Questionnaire on the Measures Implemented in the Member States Territories in the Context of Article 81 of Directive 2001/83/EC. 2017. Available online: https://ec.europa.eu/health/sites/default/files/files/committee/ev_20180525_summary_en.pdf (accessed on 12 April 2021).
  39. Service Public Federal Sante Publique Sdlcaee. 25 April 2014. Royal Decree Amending the Royal Decree of 14 December 2006 Relating to Medicinal Products for Human and Veterinary Use. 2021, pp. 1–10. Available online: https://www.ejustice.just.fgov.be/cgi_loi/change_lg.pl?language=fr&la=F&cn=2006121431&table_name=loi (accessed on 12 March 2021).
  40. SR 531.215.32. Verordnung Vom 12. August 2015 Über Die Meldestelle für Lebenswichtige Humanarzneimittel. 2013, pp. 1–24. Available online: https://www.fedlex.admin.ch/eli/cc/2015/544/de (accessed on 13 May 2021).
  41. Bart, T.N. Parallel Trade of Pharmaceuticals: A Review of Legal, Economic, and Political Aspects. Value Health 2008, 11, 996–1005. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  42. Swiss Agency for Therapeutic Products. Database Swissmedic. 2021. Available online: https://www.swissmedic.ch/swissmedic/en/home/humanarzneimittel/marketsurveillance/out-of-stock.html (accessed on 12 April 2021).
  43. Royal Decree 1345/2007 of 11 October, Which Regulates the Procedure for Authorisation, Registration and Dispensation of Industrially Manufactured Medicaments for Human Use Conditions. 2021, pp. 1–91, Database. Available online: https://www.global-regulation.com/translation/spain/1445110/royal-decree-1345---2007-of-11-october%252c-which-regulates-the-procedure-for-authorisation%252c-registration-and-dispensation-of-industrially-manufacture.html (accessed on 12 April 2021).
  44. Agencia, D.C.; Sanitarios, A.P.; Decreto, R.; Espa, A.; Sanitarios, P.; Decreto, R. Circular no 3/2011. 2011, pp. 1–7. Available online: https://www.legisweb.com.br/legislacao/?id=392406 (accessed on 24 April 2021).
  45. Law 29/2006 of 26 July, on Guarantees and Rational Use of Medicines and Medical Devices. Available online: https://www.global-regulation.com/translation/spain/1446301/law-29-2006-of-26-july%252c-on-guarantees-and-rational-use-of-medicines-and-medical-devices.html (accessed on 26 April 2021).
  46. Jensen, V.; Kimzey, L.M.; Goldberger, M.J. FDA’s role in responding to drug shortages. Am. J. Health Pharm. 2002, 59, 1423–1425. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  47. US Food and Drug Administration. Database FDA. Available online: https://www.fda.gov/news-events/press-announcements/fda-publishes-list-essential-medicines-medical-countermeasures-critical-inputs-required-executive (accessed on 28 March 2021).
  48. National Medical Stockpile the National Medical Stockpile Is a Strategic Reserve of Drugs, Vaccines, Antidotes and Personal Protective Equipment for Use in National Health Emergencies. We Emergency and Able to Meet High Levels of Demand. 2021; 19–21 April. Database TGA. Available online: https://www.health.gov.au/initiatives-and-programs/national-medical-stockpile (accessed on 8 May 2021).
  49. Therapeutic Goods Administration, Exporting Therapeutic Goods. Available online: https://www.tga.gov.au/exporting-therapeutic-goods-0 (accessed on 25 May 2021).
  50. 44/2004 of the Minister for Health, Labour Affairs and Family on the Prescription and Dispensing of Medicinal Products for Human Use. 2020, pp. 1–30. Available online: https://net.jogtar.hu/jogszabaly?docid=a0400044.esc (accessed on 8 May 2021).
  51. Government Regulation 448/2017. On Authorisation of the Prescription and Individual Use of Medicinal Products for Human Use. Paragraph 5(1). 2020, pp. 1–8. Available online: https://net.jogtar.hu/jogszabaly?docid=A1700448.KOR (accessed on 11 March 2021).
  52. King of Spain. Royal Legislative Decree 1/2015, of July 24, Approving the Consolidated Text of the Law on Guarantees and Rational Use of Medicines and Medical Devices. 2021, pp. 1–163. Available online: https://www.global-regulation.com/translation/spain/615883/royal-legislative-decree-1-2015%252c-of-july-24%252c-approving-the-revised-text-of-the-law-on-guarantees-and-rational-use-of-medicines-and-medical-devices.html (accessed on 14 May 2021).
  53. Royal Decree 177/2014 of 21 March, the Reference Price System and Homogeneous Groups of Medicines in the National Health System Is Regulated, and Certain Information Systems on Financing and Price. 2021, pp. 1–17. Available online: https://www.global-regulation.com/translation/spain/1457557/royal-decree-177-2014-of-21-march%252c-the-reference-price-system-and-homogeneous-groups-of-medicines-in-the-national-health-system-is-regulated%252c-and-.html (accessed on 12 March 2021).
  54. Guidance on Detection and Notification of Shortages of Medicinal Products for Marketing Authorisation Holders (MAHs) in the Union (EEA). Available online: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/guidance-detection-notification-shortages-medicinal-products-marketing-authorisation-holders-mahs_en.pdf (accessed on 15 March 2022).
  55. Good Practice Guidance for Communication to the Public on Medicines’ Availability Issues Recommendations for EU National Competent Authorities and EMA to Ensure Adequate Public Information. Available online: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/good-practice-guidance-communication-public-medicines-availability-issues_en.pdf (accessed on 15 March 2022).
Ijerph 19 04102 g001 550
Figure 1. Number of shortages according to ATC groups (including both critical and non-critical). B: Belgium; CH: Switzerland; S: Spain; US: United States; A: Australia; H: Hungary. ATC groups are shown in brackets (C, J, L, N).
Figure 1. Number of shortages according to ATC groups (including both critical and non-critical). B: Belgium; CH: Switzerland; S: Spain; US: United States; A: Australia; H: Hungary. ATC groups are shown in brackets (C, J, L, N).
Ijerph 19 04102 g001
Ijerph 19 04102 g002 550
Figure 2. Comparison of critical shortages of various countries by ATC groups.
Figure 2. Comparison of critical shortages of various countries by ATC groups.
Ijerph 19 04102 g002
Table
Table 1. Allocation of critical shortages.
Table 1. Allocation of critical shortages.
CountryAllocation of Critical Shortages
SwitzerlandIf there is no domestic alternative [13].
SpainIf there is no domestic alternative [19].
HungaryIf there is no domestic alternative [20].
USThe “shortage risk index” was defined with the ratio of unavailable and available presentations. If this number is higher than 5, the shortage is considered critical. If the database states: “there are no presentations available” or “there is insufficient supply for usual ordering”, despite that the index will be lower than 5, the shortage had been considered critical.
BelgiumAccording to the FAMHP decision tree and in case it is necessary to import from abroad [21].
AustraliaAccording to the TGA definition, if the medicine is included on the Medicines Watch List (MWL) or if the shortage has the potential impact to have a life-threatening or serious impact on patients and there is not likely to be a sufficient supply of potential substitutes automatically considered as critical [22].
Table
Table 2. Characteristics of notification systems.
Table 2. Characteristics of notification systems.
CountryManager of DatabaseDefinition of ShortageSource of InformationScope of ReportCountermeasures
BelgiumFederal Agency for Medicines and Health Products (FAMHP)Unable to deliver for an uninterrupted period of four days [24].MAH should notify the FAMHP within 7 days after the start of the unavailability when a drug will be unavailable for longer than 14 days [21].Concerns about a certain presentation. The entire range of medicine is not unavailable.Emergency import
Greater responsibility to full-line distributors
MAH should give an exact reason and compensate costs
SwitzerlandMartinelli Consulting GmbHSupplies not satisfying demand and orders [13].The website is based on voluntary reports from companies and users [13].Drugs officially approved in Switzerland are listed in Martinelli database.Emergency import
Strategic stockpiling
Define essentiality
SpainSpanish Agency of Medicines and Medical Products (AEMPS)The number of available units is below the level of national or local consumption needs [25].AEMPS database lists current or anticipated supply problems for different presentations. If a quick solution is expected, not included in the list [19].Concerns about a certain presentation. The entire range of medicine is not unavailable [26].Define essentiality
Emergency import
Maintain MA and production
Delivery in 24 working hours
Export ban
USAmerican Society of Health-System Pharmacists (ASHP)Supply issue that affects how a pharmacy prepares, dispenses a drug, or influences patient care when prescribers must use an alternative agent [10].Voluntary reports from practitioners, patients, and others [12].ASHP lists every drug shortage reported through the online report form as soon as it is investigated and confirmed, usually within 24–72 h [12].Define essentiality
Emergency import
Reevaluates voluntary recalls
Expedite changes
Maintain MA and production
Drugs into smaller units
ASHP management practice
AustraliaTherapeutic Goods Administration (TGA)The supply of medicine will not (or may not) meet the demand for the medicine at the subsequent six months, including all patients who take (may need to take) [27].MAHs are required to report all registered medicines in 2–10 days upon severity [22].The medicines are set out in Therapeutic Goods Determination [28].Emergency import
Export registration
National Medical Stockpile [29]
Medicine Watch List (MWL) [30].
HungaryNational Institute of Pharmacy and Nutrition (OGYEI)If the MAH is unable to maintain adequate and
continuous supplies of specific medicinal products, or
unwilling to supply the preparation temporarily or permanently [31].		Before the final delivery to the wholesaler, but in a maximum of two months [31].Concerns about a certain presentation. The entire range of medicine is not unavailable.Recommending alternatives
Shortage declaration by CA
Emergency import
Export ban
Table
Table 3. Number of shortages per million people in the studied countries, including critical cases.
Table 3. Number of shortages per million people in the studied countries, including critical cases.
CountryDateNo. of Shortages (a)Considered as Critical
(b)		Population in Millions
(c)		Shortage per Million People
(d = a/c)		Critical Shortage per Million People
(e = b/c)		Percentage of Critical/All Shortages
(f = e/d)
Belgium4 Mar 20212274111.619.573.5318.1
Spain6 Mar 20212675146.75.721.0919.1
Hungary9 Mar 2021250639.626.046.5625.2
Switzerland6 Mar 2021173388.719.894.3722.0
Average of examined European countries-229.2548.25-17.805 *3.8921.1
US9 Mar 202168393332.92.050.2813.6
Australia8 Mar 20212442525.89.460.9710.2
* The European average values were calculated from the data of individual European countries examined.
Table
Table 4. Proportion estimation of critical shortages over the whole dataset. Binomial probability tests of proportion of critical shortages across ATC groups.
Table 4. Proportion estimation of critical shortages over the whole dataset. Binomial probability tests of proportion of critical shortages across ATC groups.
Critical_BIProportionStd. Err.Logit
(95% Conf. Interval)
00.83134490.00871990.81354580.8477629
10.16865510.00871990.15223710.1864542
ATCNumber of All Shortages (N)Observed (k)Expected (k)Expected (p)Observed (p)
C63955107.77060890.168660.08607
J36910562.23373190.168660.28455
L1763929.68329760.168660.22159
N660112111.3123660.168660.16970
Table
Table 5. Summary of critical shortages of the examined countries according to the WHO Essential Medicine List.
Table 5. Summary of critical shortages of the examined countries according to the WHO Essential Medicine List.
BelgiumSwitzerlandSpainHungaryAverage of Examined European CountriesUSAustralia
No. of critical shortages/million people3.534.371.096.563.890.280.97
No. of critical shortages on WHO Essential List/million people3.191.610.753.962.380.280.74
WHO Essential/all critical shortages (%)90.3636.8268.7660.3464.0745.0675.92
Table
Table 6. Countermeasures introduced by countries. * Not applicable means that we did not obtain relevant literature on this topic.
Table 6. Countermeasures introduced by countries. * Not applicable means that we did not obtain relevant literature on this topic.
Compulsory StockpilingMeasures for Essential Medicines—Legal DefinitionsNotification Responsibilitiy—Legal BackgroundMeasures Affecting WholesalersExport BansEmergency Imports
BEFull-line wholesalers required to have a range of products in stock for the needs of the given geographic territory/ies [36].Not defined.Not providing the exact reasons is considered a clear legal violation [37].Full-line wholesalers are assigned besides regular ones with special responsibilities and privileges [36,38].Temporarily to medicinal products for which a shortage is notified [37].Based on a doctor’s request wholesalers may temporarily import medicine from the EU, if no substitutes are available in BE, in specific quantities requested by the doctor [39].
CHDelegated to companies at a defined level of stocks in Ordinance [40].Compounds for which there are no or only limited substitutes have been affected by a supply shortage over the previous three years [40].Determined in 531.215.32 Ordinance [40].Managing the strategic level of inventory stock delegated by the federal government in local law [40].Parallel export does not become significant due to high prices [41]. Upon application for temporary import submission [42].
ESCA may demand commercialization to grant the suspension, or the revocation of the product [43].Essential if the pharmaceutical gap cannot fully cover or have a high economic impact. Critical if it has no available therapeutic alternatives and has a complex manufacturing process, and/or has only one supplier [38].Spanish Medicine and Products Devices Agency Circular No. 3/2011 [44].All wholesalers are required to deliver within 24 working hours [38].If lack of medicinal products causes a pharmaceutical gap [45].The Spanish Agency of Medicinal Products and Medical Devices can also approve the import of medicines labeled in other languages or with an expiry date shorter than 6 months [38].
USFDA supports MAHs to maintain production [46].If used to treat or prevent a serious disease or condition, and there is no other adequate available source [47].Safety and Innovation Act in 2012 [8].* Not applicable.* Not applicable.FDA may allow emergency importation [46].
ANational Medical Stockpile maintains the strategic reserve of products [48].Included on Medicines Watch List, has a potential life-threatening or serious impact, or has no potential substitutes [22].Therapeutic Goods (Reportable Medicines) Determination 2018 [28].Wholesalers also have a duty to notify authorities about the expected duration of a discontinuation [29].Only for MAH or designated entity [49].Therapeutic Goods Act has been amended to assist import [27].
HUProducts decreed by the minister should be available in the quantity defined therein [31].Not defined.Act XCV of 2005 on Medicinal Products for Human Use [31].Authorized wholesale distributors shall be required to procure and supply the medicinal products as their authorization for wholesale distribution pertains [31].The active substances decreed by the minister for a period not exceeding one year [31].On the wholesaler’s request “contingent-approval” [50] or the physician statement “individual approval of OGYEI” [51].
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Turbucz, B.; Major, M.; Zelko, R.; Hanko, B. Proposal for Handling of Medicine Shortages Based on a Comparison of Retrospective Risk Analysis. Int. J. Environ. Res. Public Health 2022, 19, 4102. https://doi.org/10.3390/ijerph19074102

AMA Style

Turbucz B, Major M, Zelko R, Hanko B. Proposal for Handling of Medicine Shortages Based on a Comparison of Retrospective Risk Analysis. International Journal of Environmental Research and Public Health. 2022; 19(7):4102. https://doi.org/10.3390/ijerph19074102

Chicago/Turabian Style

Turbucz, Bela, Martin Major, Romana Zelko, and Balazs Hanko. 2022. "Proposal for Handling of Medicine Shortages Based on a Comparison of Retrospective Risk Analysis" International Journal of Environmental Research and Public Health 19, no. 7: 4102. https://doi.org/10.3390/ijerph19074102

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop