Next Article in Journal
A New Insight on the Radioprotective Potential of Epsilon-Aminocaproic Acid
Previous Article in Journal
Ayurgenomics and Modern Medicine
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Medicina
Volume 56
Issue 12
10.3390/medicina56120662
medicina-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Temporal Change in Alert Override Rate with a Minimally Interruptive Clinical Decision Support on a Next-Generation Electronic Medical Record

by
Won Chul Cha
1,2,3,
Weon Jung
1,
Jaeyong Yu
1,
Junsang Yoo
4 and
Jinwook Choi
2,*
1
Department of Digital Health, SAIHST, Sungkyunkwan University, Seoul 06355, Korea
2
Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul 03080, Korea
3
Department of Emergency Medicine, Samsung Medical Center, Seoul 06355, Korea
4
Department of Nursing, Sahmyook University, School of Nursing, Institution of Healthcare Resource, Seoul 01795, Korea
*
Author to whom correspondence should be addressed.
Medicina 2020, 56(12), 662; https://doi.org/10.3390/medicina56120662
Submission received: 11 November 2020 / Revised: 25 November 2020 / Accepted: 29 November 2020 / Published: 30 November 2020
Browse Figures
Versions Notes

Abstract

:
Background and objectives: The aim of this study is to describe the temporal change in alert override with a minimally interruptive clinical decision support (CDS) on a Next-Generation electronic medical record (EMR) and analyze factors associated with the change. Materials and Methods: The minimally interruptive CDS used in this study was implemented in the hospital in 2016, which was a part of the new next-generation EMR, Data Analytics and Research Window for Integrated kNowledge (DARWIN), which does not generate modals, ‘pop-ups’ but show messages as in-line information. The prescription (medication order) and alerts data from July 2016 to December 2017 were extracted. Piece-wise regression analysis and linear regression analysis was performed to determine the temporal change and factors associated with it. Results: Overall, 2,706,395 alerts and 993 doctors were included in the study. Among doctors, 37.2% were faculty (professors), 17.2% were fellows, and 45.6% trainees (interns and residents). The overall override rate was 61.9%. There was a significant change in an increasing trend at month 12 (p < 0.001). We found doctors’ positions and specialties, along with the number of alerts and medication variability, were significantly associated with the change. Conclusions: In this study, we found a significant temporal change of alert override. We also found factors associated with the change, which had statistical significance.

1. Introduction

Clinical decision support (CDS) plays a critical role in everyday clinical practice. Combined with advanced electronic medical records (EMRs), CDS is identified as a key for quality medicine, improving the safety and effectiveness of healthcare [1,2,3]. The use of CDS is associated with better adherence to clinical guidelines, a reduction in morbidity, and a reduction in medical errors [4,5,6,7]. These merits of CDS are influenced by various factors because the system is often integrated into complicated clinical settings [8,9]. A CDS to EMR is like an application to a smartphone, which is essentially intertwined with users and the work process.
The limitations of CDS, however, have been highlighted with an increase in CDS implementation. Alert fatigue is the most frequently addressed problem of CDS [10]. It is derived from alarm fatigue, which was observed during a study of aviation and nuclear fields where the false alarm rate was high [11]. Similar to these critical fields, healthcare providers tend to override alerts, which can lead to errors. Paradoxically, alert fatigue caused by safety stipulations established by the CDS can result in patient safety issues [12]. The effect of CDS on clinical outcomes seems as ambivalent.
Redesigning the CDS has been suggested to lower alert fatigue and alert override rates. Modals, i.e., pop-ups, are the least preferred way to increase the alert acceptance rate, whereas tailoring CDS for users is the best-suggested method [13]. Tiering alerts by severity or designing the alerts to be less interruptive seem to be a useful approach to increase the acceptance rate by users [14,15]. However, reports on minimally interruptive CDSs are extremely rare, and few studies have analyzed the various factors that affect the alert-override on the CDS based on a temporal trend [16]. Up to today, the effect of reducing interruptions on the acceptance of CDSs remains unanswered.
The aim of this study is to describe the temporal trend of an alert-override with a minimally interruptive CDS and analyze the factors associated with this change.

2. Methods

This is a retrospective study using data from an EMR. The study was approved by the institutional review board of the study site (IRB: 2019-05-038).

2.1. Study Setting

The study was conducted in an academic tertiary hospital in Seoul, which serves approximately two million out-patient (OPD) visits annually and provides an in-hospital service for 2000 beds. Approximately 1000 doctors and 6000 nurses work in such institutes. According to the governmental policy for tertiary hospitals, hospitals focus on care for critical and severe patients such as cancers and cardiovascular diseases.

2.2. Next-Generation Electronic Medical Record System and Medication Order (Prescription) System

The next-generation EMR used in this study was a home-grown system, which was launched in 2016 after almost three years of development. The system was a result of the “next-generation EMR project,” which was later named Data Analytics and Research Window for Integrated kNowledge (DARWIN). DARWIN is a comprehensive system that includes computerized order entry of the physicians as well as nursing, pharmacy, billing, research support, and even patient portal and web services. The conceptual architecture of DARWIN is described in Figure 1 [17].
Within DARWIN, as well as other EMR systems, the medication process is carried out as a sequence of action triggered by physicians. Once the process is initiated, a patient is selected, the diagnosis is confirmed, and orders for tests and medications are followed. When ordering a specific medicine, the medicine is searched for, and specifics such as the routes, doses, and duration are entered. This sequence is usually repeated multiple times before the order is finalized. The CDS in this study is made up of activated sequences between drug selection and finalization. The process is also described in Figure 2.

2.3. Minimally Interruptive Clinical Decision Support Design

During the development of DARWIN, a minimally interruptive medication CDS was integrated into the system. A minimally interruptive CDS is similar to a non-interruptive CDS or is not likely to intervene in the work process of physicians. Although this type of alert may reduce alert fatigue, it may also result in decreased effectiveness of the CDS [18,19]. The CDS at the order entry is mainly for physicians and only covers medication alerts.
The rule-based database for the CDS is supplied by Medi-Span (Wolters Kluwer Health, Philadelphia, PA, USA), with monthly updates. The types of alerts (domain) were age, allergy, disease, duplication, gender, lactation, pregnancy, dose, drug-drug interactions, and route. A screenshot of alerts along with the medication order is presented in Figure 3. During the study period, there was no significant change in the EMR and CDS. Only minor revisions and database updates had been made.

2.4. Study Participants

This study included patients from July 2016 to December 2017. Patients were included if they visited the out-patient department (OPD), emergency department (ED), or were admitted to inpatient wards (IW) during this 18-month period. Cases were selected if the ordering physicians had given orders throughout the study phase. If there had been any month during which no medication order was given by a doctor, the doctor was excluded from the dataset.

2.5. Data Extraction and Preparation

Medication order data were extracted from the clinical data warehouse (CDW) of the study site. Included data were the visit date, place of visit (OPD, ED, and IW), and medication (name, dose, and route). The specifics of the physicians were also collected: specialty department, position (trainee versus faculty (board-certified physicians)). CDS alert data were extracted from the DARWIN repository. Alert information contained the diagnosis code, reasons and types of alerts, and the level of caution.

2.6. Definition of Alert Override

Because the CDS is minimally interruptive, it does not mandate users to input response to the alert. Users may simply ignore the alert and continue the process. The definitions of an alert-override were as follows:
  • If neither medication was deleted from a drug-drug interaction (DDI), the DDI alert is overridden.
  • If a physician does not adjust the dose after a dose alert, the dose alert is overridden.
  • If a physician does not delete the medication order after an alert for age, allergy, disease, duplication, gender, lactation, pregnancy, or route, the alert is overridden.
  • This definition was validated in a previous study from the study site [16].

2.7. Data Analysis

The basic characteristics of the patients and alerts were described as simple statistics. A temporal change was demonstrated using time-series analysis. Piece-wise regression was conducted to determine the temporal change in the alert override rate. Finally, multivariate analysis was applied to determine if the change was statistically significant after adjusting for covariates. The data were analyzed using the statistical software R (v4.0.3) with the segmented (v1.3) package for piece-wise regression and the “lm” (linear model) function for linear regression.
For piece-wise regression, the alert override rate was analyzed using an iterative algorithm that utilizes maximum likelihood estimation of the observed data to identify unknown change point(s) in piece-wise regression models. The merit of using this method is that this method does not require prior knowledge of a changepoint. Changepoint estimate and 95% confidence intervals in the piece-wise regression model were calculated using this method.

3. Results

3.1. Basic Characteristics of Subjects

During the study period, 993 doctors wrote medication orders. Among the doctors, 37.2% were faculty (professors), 17.2% were fellows, and 45.6% were trainees (interns and residents). Doctors from every department were included; the most common specialty was general surgery (99% or 10.0%), general internal medicine (87% or 8.8%), and anesthesiology and pain medicine (85% or 8.6%), respectively. General internal medicine is a label for trainees in the internal medicine department, and its use is the reason why there were only one fellow and one faculty member in the department (Table 1) (Figure 4).

3.2. CDS Alerts and Override Information

During the study period, 2,706,395 alerts were fired to the subjects. Alerts were fired based on eight types of medication CDSs. Among these types, the dose type was the most common (1,551,876, or 57.3%), followed by age type (460,793, or 17.0%), and pregnancy type (292,266, or 10.8%). In terms of the location of the alert, OPD had a much larger number of alerts than IW or ED; it is also widely known that the patient and order volume was incomparably higher in OPD than in the other departments (Table 2).
Regarding the medication types, there were 1240 types of medications that caused an alert. The most common medication was ketorolac tromethamine (analgesics) (n = 106,851), which is one of the widely used non-steroidal anti-inflammatory drugs. The second and third most common drugs with an alerts were Propacetamol HCl (analgesics–nonnarcotic) (n = 78,432), and chlorpheniramine (antihistamines) (n = 75,361).
The types of medications ordered by doctors varied widely, whereas the median number was 84, with an interquartile range of 38 to 129. Moreover, the number of alerts per doctor also varied greatly. For the number of alerts per week, the median was 37.5, whereas the interquartile range was 15.1 to 74.7 during the 18-month period. The distribution of these numbers is demonstrated in Figure 5.

3.3. Temporal Change of Alerts and Alert Overrides

To determine the change in trend regarding the alert overrides, a time-series analysis was conducted. The analysis was based on the place of order (OPD, IW, and ED) and position (faculty, fellow, and trainee) (Figure 6a,b). The trend of the alert overrides was also demonstrated based on the CDS type (Figure 6c). This analysis revealed that an increase in overrides was observed at an institution-wide level, not just by a few groups of doctors on a certain type. Piece-wise regression was used to determine the trend change, which confirmed that there was a significant change at month 12. (difference of slope: 0.017 Confidence Interval (0.010, 0.063)) (p < 0.001). The degree of change varied among the doctors (Figure 6d). The mean difference in the alert overrides between the first 12 months (before) and the last 6 months (after) was 4.9% (SD = 8.3%). The interquartile range was 1.5% to 9.3%.

3.4. Factors Associated with Increased Alert Overrides

Because the increase in alert overrides varied among doctors, a multivariate analysis was conducted to reveal the factors associated with an increase in alert overrides. The adjusted co-efficiency showed an increased override rate in the fellow and trainee groups compared to the faculty group. Surgeons showed less increase in override rates; other specialty doctors showed higher increases. When grouped into numbers of medication (how many types of drugs are prescribed by a doctor, i.e., drug variability), the higher the number was, the lower the alert overrides observed in the third quartile group. The frequency of received alerts showed an incremental effect on the override rate; groups with a higher number of alerts were associated with a greater change (Table 3).

4. Discussion

In this study, alert and alert-override patterns over an 18-month period were described. The study included all patients and providers across a tertiary hospital and found significant changes in alert override during the study period. We also found factors that were related to an increase in alert overrides. To the best of the authors’ knowledge, this is the first study to include inpatient, emergency department, and out-patient orders, which is important because doctors in tertiary centers tend to work for these sites simultaneously.
We also found a low rate of alert override, i.e., 61.9%, compared to previous reports of 49% to 94% [12,20,21]. This is even more impressive, considering that it has been widely accepted that CDS with a hard stop, which is a highly interruptive approach, is more effective [22]. The low rate of alert override may have resulted from the design of the CDS in which the relevant message is displayed at the bottom of the order line while the order is being typed in. The modification over message delivery, timing, and content adjustment are key components of a successful CDS [23,24].
The method used to determine the change in the trend of the override rate was piece-wise regression, also known as interrupted time-series analysis. The method is becoming increasingly used because of its robustness to controlling secular trends and ease of interpretation [25,26]. However, there remains a risk of producing misleading or incorrect outcomes if the statistical method is not properly used [27,28].
This study does not include the appropriateness of the alert overrides. Although there are many reasons for an alert override, it is difficult to determine the accurate reasons because many doctors do not always correctly reflect the reason for the override [29]. Overrides are often the result of the clinical context, which in turn is considered appropriate by chart reviews [30,31]. It is reasonable to consider that many of the alerts are appropriate, and the CDS can always be improved in terms of specificity, which means reducing the false-alert rate.
In this study, we found a statistically significant association between an override increase and factors specific to doctors. These factors were consistent with those of previous studies, which also included these factors. The specialty department, along with the number of medications and number of alerts, were reported as major causes of an alert override. Other studies have also suggested reasons for alert overrides, i.e., the clinical condition (severity) of the patients, [32] appropriateness of the medication, doctors’ habits, and admitting conditions of specialty departments [33,34].
Factors included as the place of order, types of alerts, and position of doctors who made the prescription orders could be considered as a static factor. Variables that could change over time, which include the degree of experience, the policies of the institution, or change of disease epidemiology, could be considered as temporal factors. These temporal factors’ association with alert override requires further studies since they were not covered in the scope of this study. A qualitative method could be appropriate since it could reveal situations that users had faced at that time.
The results of this study imply that even without intervention, minimally interruptive CDS alert overrides may increase after about one year of usage. This one year of usage may have made users of EMR familiar with the system; at the same time, it may have caused sufficient fatigue to increase the overrides. To prevent an override and make the system more effective, caution must be exercised by the high-risk group of doctors who can be defined by the results of this study. Medical doctors prescribing many different types of drugs, and receiving a large volume of alerts could be potential targets.

Limitation

This study has certain limitations. First, the study was conducted in a single center. Because a CDS and an EMR, along with clinical norms, differ widely among institutions, the result of this study should be applied to other centers with caution. Second, this study did not cover all confounders of alerts in the study site. Alert fatigue may be driven by multiple sources, and other confounding factors could have influenced the outcome with varying degrees of interaction. Though there is no evidence, a significant change in the CDS rule engine could have influenced the outcome. Third, owing to the design of minimally interruptive CDSs, specific reasons for the overrides were unavailable. Without confirmed reasons for an alert override, it is difficult to determine the appropriateness.

5. Conclusions

In this study, we described the temporal trend of an alert-override with a minimally interruptive CDS and analyzed the factors associated with the change. The analysis revealed variables such as doctor’s specialty, frequency of prescription on specific drugs, and the number of alerts received as significant factors associated with the increase in the trend.

Author Contributions

J.C. conceived and designed the experiments; W.J. performed the experiments; J.Y. (Jaeyon Yu) and W.J. analyzed the data; J.Y. (Junsang Yoo) contributed reagents/materials/analysis tools; W.C.C. wrote the paper. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Saini, V.; Garcia-Armesto, S.; Klemperer, D.; Paris, V.; Elshaug, A.G.; Brownlee, S.; AIoannidis, J.P.; Fisher, E.S. Drivers of poor medical care. Lancet 2017, 390, 178–190. [Google Scholar] [CrossRef]
  2. Elshaug, A.G.; Rosenthal, M.B.; Lavis, J.N.; Brownlee, S.; Schmidt, H.; Nagpal, S.; Littlejohns, P.; Srivastava, D.; Tunis, S.; Saini, V. Levers for addressing medical underuse and overuse: Achieving high-value health care. Lancet 2017, 390, 191–202. [Google Scholar] [CrossRef]
  3. Institute of Medicine Roundtable on Evidence-Based, M. The National Academies Collection: Reports funded by National Institutes of Health. In The Learning Healthcare System: Workshop Summary; Olsen, L., Aisner, D., McGinnis, J.M., Eds.; National Academies Press (US): Washington, DC, USA, 2007. [Google Scholar]
  4. Fretheim, A.; Flottorp, S.; Oxman, A. Effect of Interventions for Implementing Clinical Practice Guidelines; Norwegian Knowledge Centre for the Health Services: Oslo, Norway, 2015. [Google Scholar]
  5. Moja, L.; Kwag, K.H.; Lytras, T.; Bertizzolo, L.; Brandt, L.; Pecoraro, V.; Rigon, G.; Vaona, A.; Ruggiero, F.; Mangia, M.; et al. Effectiveness of computerized decision support systems linked to electronic health records: A systematic review and meta-analysis. Am. J. Public Health 2014, 104, e12–e22. [Google Scholar] [CrossRef] [PubMed]
  6. Bonnabry, P.; Despont-Gros, C.; Grauser, D.; Casez, P.; Despond, M.; Pugin, D.; Rivara-Mangeat, C.; Koch, M.; Vial, M.; Iten, A.; et al. A risk analysis method to evaluate the impact of a computerized provider order entry system on patient safety. J. Am. Med. Inform. Assoc. 2008, 15, 453–460. [Google Scholar] [CrossRef] [PubMed]
  7. Kaushal, R.; Shojania, K.G.; Bates, D.W. Effects of computerized physician order entry and clinical decision support systems on medication safety: A systematic review. Arch. Intern. Med. 2003, 163, 1409–1416. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Owens, D.K.; Bravata, D.M. Computer-based decision support: Wishing on a star? Eff. Clin. Pract. 2001, 4, 34–38. [Google Scholar]
  9. Bonetti, D.; Johnston, M.; Pitts, N.B.; Deery, C.; Ricketts, I.; Tilley, C.; Clarkson, J.E. Knowledge may not be the best target for strategies to influence evidence-based practice: Using psychological models to understand RCT effects. Int. J. Behav. Med. 2009, 16, 287–293. [Google Scholar] [CrossRef]
  10. Kesselheim, A.S.; Cresswell, K.; Phansalkar, S.; Bates, D.W.; Sheikh, A. Clinical decision support systems could be modified to reduce ‘alert fatigue’ while still minimizing the risk of litigation. Health Aff. 2011, 30, 2310–2317. [Google Scholar] [CrossRef]
  11. Wickens, C.D.; Hollands, J.G.; Banbury, S.; Parasuraman, R. Engineering Psychology and Human Performance; Psychology Press: New York, NY, USA, 2013. [Google Scholar]
  12. van der Sijs, H.; Aarts, J.; Vulto, A.; Berg, M. Overriding of drug safety alerts in computerized physician order entry. J. Am. Med. Inform. Assoc. 2006, 13, 138–147. [Google Scholar] [CrossRef]
  13. Hussain, M.I.; Reynolds, T.L.; Zheng, K. Medication safety alert fatigue may be reduced via interaction design and clinical role tailoring: A systematic review. J. Am. Med. Inform. Assoc. 2019, 26, 1141–1149. [Google Scholar] [CrossRef] [Green Version]
  14. Paterno, M.D.; Maviglia, S.M.; Gorman, P.N.; Seger, D.L.; Yoshida, E.; Seger, A.C.; Bates, D.W.; Gandhi, T.K. Tiering drug–drug interaction alerts by severity increases compliance rates. J. Am. Med. Inform. Assoc. 2009, 16, 40–46. [Google Scholar] [CrossRef] [Green Version]
  15. Green, L.A.; Nease, D.; Klinkman, M.S. Clinical reminders designed and implemented using cognitive and organizational science principles decrease reminder fatigue. J. Am. Board Fam. Med. 2015, 28, 351–359. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Yoo, J.; Lee, J.; Rhee, P.; Chang, D.K.; Kang, M.; Choi, J.S.; Bates, D.W.; Cha, W.C. Alert Override Patterns with a Medication Clinical Decision Support System in an Academic Emergency Department: Retrospective Descriptive Study. JMIR Med. Inform. 2020, 8, e23351. [Google Scholar] [CrossRef] [PubMed]
  17. Jung, K.Y.; Kim, T.; Jung, J.; Lee, J.; Choi, J.S.; Mira, K.; Chang, D.K.; Cha, W.C. The Effectiveness of Near-Field Communication Integrated with a Mobile Electronic Medical Record System: Emergency Department Simulation Study. JMIR Mhealth Uhealth 2018, 6, e11187. [Google Scholar] [CrossRef] [PubMed]
  18. Kubben, P.; Dumontier, M.; Dekker, A. Fundamentals of Clinical Data Science; Springer: Berlin/Heidelberg, Germany, 2018. [Google Scholar]
  19. Scheepers-Hoeks, A.M.; Grouls, R.J.; Neef, C.; Ackerman, E.W.; Korsten, E.H. Physicians’ responses to clinical decision support on an intensive care unit--comparison of four different alerting methods. Artif. Intell. Med. 2013, 59, 33–38. [Google Scholar] [CrossRef] [PubMed]
  20. Castillo-Páramo, A.; Clavería, A.; Verdejo González, A.; Rey Gómez-Serranillos, I.; Fernández-Merino, M.C.; Figueiras, A. Inappropriate prescribing according to the STOPP/START criteria in older people from a primary care setting. Eur. J. Gen. Pract. 2014, 20, 281–289. [Google Scholar] [CrossRef] [Green Version]
  21. van der Sijs, H.; Mulder, A.; van Gelder, T.; Aarts, J.; Berg, M.; Vulto, A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol. Drug Saf. 2009, 18, 941–947. [Google Scholar] [CrossRef]
  22. Roshanov, P.S.; Fernandes, N.; Wilczynski, J.M.; Hemens, B.J.; You, J.J.; Handler, S.M.; Nieuwlaat, R.; Souza, N.M.; Beyene, J.; Van Spall, H.G.C.; et al. Features of effective computerised clinical decision support systems: Meta-regression of 162 randomised trials. BMJ Br. Med. J. 2013, 346, f657. [Google Scholar] [CrossRef] [Green Version]
  23. Kawamoto, K.; Houlihan, C.A.; Balas, E.A.; Lobach, D.F. Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. BMJ 2005, 330, 765. [Google Scholar] [CrossRef] [Green Version]
  24. Wendt, T.; Knaup-Gregori, P.; Winter, A. Decision support in medicine: A survey of problems of user acceptance. Stud. Health Technol. Inform. 2000, 77, 852–856. [Google Scholar]
  25. Penfold, R.B.; Zhang, F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013, 13, S38–S44. [Google Scholar] [CrossRef] [PubMed]
  26. Fretheim, A.; Tomic, O. Statistical process control and interrupted time series: A golden opportunity for impact evaluation in quality improvement. BMJ Qual. Saf. 2015, 24, 748–752. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  27. Ewusie, J. Improved Methods for Interrupted Time Series Analysis Useful When Outcomes Are Aggregated: Accounting for Heterogeneity across Patients and Healthcare Settings. Ph.D. Thesis, MacMaster University, Hamilton, Canada, 2019. [Google Scholar]
  28. Ramsay, C.R.; Matowe, L.; Grilli, R.; Grimshaw, J.M.; Thomas, R.E. Interrupted time series designs in health technology assessment: Lessons from two systematic reviews of behavior change strategies. Int. J. Technol. Assess. Health Care 2003, 19, 613–623. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  29. Dekarske, B.M.; Zimmerman, C.R.; Chang, R.; Grant, P.J.; Chaffee, B.W. Increased appropriateness of customized alert acknowledgement reasons for overridden medication alerts in a computerized provider order entry system. Int. J. Med. Inform. 2015, 84, 1085–1093. [Google Scholar] [CrossRef] [PubMed]
  30. Taylor, L.K.; Tamblyn, R. Reasons for physician non-adherence to electronic drug alerts. Stud. Health Technol. Inform. 2004, 107 Pt 2, 1101–1105. [Google Scholar]
  31. Nanji, K.C.; Seger, D.L.; Slight, S.P.; Amato, M.G.; Beeler, P.E.; Her, Q.L.; Dalleur, O.; Eguale, T.; Wong, A.; Silvers, E.R.; et al. Medication-related clinical decision support alert overrides in inpatients. J. Am. Med. Inform. Assoc. 2018, 25, 476–481. [Google Scholar] [CrossRef]
  32. Kennell, T.I., Jr.; Cimino, J.J. A Potential Answer to the Alert Override Riddle: Using Patient Attributes to Predict False Positive Alerts. AMIA Annu. Symp. Proc. 2020, 2019, 532–541. [Google Scholar]
  33. Ahn, E.K.; Cho, S.Y.; Shin, D.; Jang, C.; Park, R.W. Differences of Reasons for Alert Overrides on Contraindicated Co-prescriptions by Admitting Department. Healthc Inform. Res. 2014, 20, 280–287. [Google Scholar] [CrossRef] [Green Version]
  34. Ahearn, M.D.; Kerr, S.J. General practitioners’ perceptions of the pharmaceutical decision-support tools in their prescribing software. Med. J. Aust. 2003, 179, 34–37. [Google Scholar] [CrossRef]
Medicina 56 00662 g001 550
Figure 1. Overview of the electronic medical record (EMR) system of the study site. In addition to essential computerized order entry (CPOE), the EMR consists of various support systems such as patient portals and administrative functions.
Figure 1. Overview of the electronic medical record (EMR) system of the study site. In addition to essential computerized order entry (CPOE), the EMR consists of various support systems such as patient portals and administrative functions.
Medicina 56 00662 g001
Medicina 56 00662 g002 550
Figure 2. Medication order (prescription) sequence in electronic medical records. The process is carried out in a sequence, and the clinical decision support (CDS) in the study site intervened between certain action points. The points at which the CDS intervened are marked as thick gray arrows.
Figure 2. Medication order (prescription) sequence in electronic medical records. The process is carried out in a sequence, and the clinical decision support (CDS) in the study site intervened between certain action points. The points at which the CDS intervened are marked as thick gray arrows.
Medicina 56 00662 g002
Medicina 56 00662 g003 550
Figure 3. Description of clinical decision support (CDS) of the study. The system is designed as minimally interruptive; it does not block the user sites or stop the working process. Alert messages appear under the prescription line simultaneously while the order details are being inputted. The alert message disappears when the prescription details are revised according to the recommendation. The color represents the seriousness of the alerts: red, high; orange, medium; and blue, low.
Figure 3. Description of clinical decision support (CDS) of the study. The system is designed as minimally interruptive; it does not block the user sites or stop the working process. Alert messages appear under the prescription line simultaneously while the order details are being inputted. The alert message disappears when the prescription details are revised according to the recommendation. The color represents the seriousness of the alerts: red, high; orange, medium; and blue, low.
Medicina 56 00662 g003
Medicina 56 00662 g004 550
Figure 4. The alerts selection process of this study. All counts were based on alerts. Alerts were excluded if they were minor types or if alerts were fired to doctors whose experience on alerts was minimal (less than 60 during the study period). Alerts to doctors whose alert experience did not last 18 months were also excluded.
Figure 4. The alerts selection process of this study. All counts were based on alerts. Alerts were excluded if they were minor types or if alerts were fired to doctors whose experience on alerts was minimal (less than 60 during the study period). Alerts to doctors whose alert experience did not last 18 months were also excluded.
Medicina 56 00662 g004
Medicina 56 00662 g005a 550Medicina 56 00662 g005b 550
Figure 5. Number of medications used and alerts in the view of doctors. (a) The number of medication types by doctors; (b) The weekly number of alerts per doctor.
Figure 5. Number of medications used and alerts in the view of doctors. (a) The number of medication types by doctors; (b) The weekly number of alerts per doctor.
Medicina 56 00662 g005aMedicina 56 00662 g005b
Medicina 56 00662 g006a 550Medicina 56 00662 g006b 550
Figure 6. Trend of Alert firing rate and alert override rate by physicians. (a) Alert firing trend by place, (b) Alert firing by position; (c) Alert firing by CDS type; (d) The overall alert firing with piece-wise regression. These findings consistently indicate that there was a significant increase in the alert firing rate and override rate after 1-year of implementation.
Figure 6. Trend of Alert firing rate and alert override rate by physicians. (a) Alert firing trend by place, (b) Alert firing by position; (c) Alert firing by CDS type; (d) The overall alert firing with piece-wise regression. These findings consistently indicate that there was a significant increase in the alert firing rate and override rate after 1-year of implementation.
Medicina 56 00662 g006aMedicina 56 00662 g006b
Table
Table 1. Composition of specialties among study subjects. General internal medicine is a specialty given to trainees in internal medicine, which resulted in only one staff member or fellow belonging to this category.
Table 1. Composition of specialties among study subjects. General internal medicine is a specialty given to trainees in internal medicine, which resulted in only one staff member or fellow belonging to this category.
DepartmentFaculty
n (%)		Fellow
n (%)		Trainee
n (%)		Total
n (%)
General Surgery35 (3.5)9 (0.9)55 (5.5)99 (10.0)
General Internal Medicine1 (0.1)1 (0.1)85 (8.6)87 (8.8)
Anesthesiology and Pain Medicine36 (3.6)9 (0.9)40 (4.0)85 (8.6)
Pediatrics19 (1.9)16 (1.6)32 (3.2)67 (6.7)
Gynecology and Obstetrics12 (1.2)8 (0.8)31 (3.1)51 (5.1)
Neurology21 (2.1)11 (1.1)12 (1.2)44 (4.4)
Thoracic surgery16 (1.6)6 (0.6)20 (2.0)42 (4.2)
Gastroenterology24 (2.4)16 (1.6)0 (0.0)40 (4.0)
Orthopedic Surgery13 (1.3)6 (0.6)20 (2.0)39 (3.9)
Cardiology24 (2.4)9 (0.9)0 (0.0)33 (3.3)
Emergency Department5 (0.5)5 (0.5)23 (2.3)33 (3.3)
Hematology and Oncology26 (2.6)6 (0.6)0 (0.0)32 (3.2)
Ophthalmology10 (1.0)10 (1.0)12 (1.2)32 (3.2)
Neurosurgery14 (1.4)4 (0.4)13 (1.3)31 (3.1)
Otolaryngology11 (1.1)4 (0.4)14 (1.4)29 (2.9)
Psychiatry8 (0.8)5 (0.5)16 (1.6)29 (2.9)
Urology12 (1.2)5 (0.5)12 (1.2)29 (2.9)
Family Medicine4 (0.4)2 (0.2)21 (2.1)27 (2.7)
Table
Table 2. Type of alerts and place of orders during the study period.
Table 2. Type of alerts and place of orders during the study period.
Alert Type Place of Order
OPD
n(%)		 IW
n (%)		 ED
n (%)		 Total
n (%)
Dose584,414 (50.0%)900,000 (62.8%)67,462 (64.3%)1,551,876 (57.3%)
Age193,991 (16.6%)248,050 (17.3%)18,752 (17.9%)460,793 (17.0%)
Pregnancy164,644 (14.1%)118,139 (8.2%)9483 (9.0%)292,266 (10.8%)
Disease53,998 (4.6%)74,781 (5.2%)4171 (4.0%)132,950 (4.9%)
Duplication88,793 (7.6%)15,096 (1.1%)1540 (1.5%)105,429 (3.9%)
Drug-Drug Interaction39,801 (3.4%)34,337 (2.4%)1388 (1.3%)75,526 (2.8%)
Gender33,144 (2.8%)22,614 (1.6%)429 (0.4%)56,187 (2.1%)
Allergy10,411 (0.9%)19,238 (1.3%)1719 (1.6%)31,368 (1.2%)
OPD, Outpatient Department; IW, Inpatient Ward; ED, Emergency Department.
Table
Table 3. Factors associated with increased alert override by doctors. The outcome was defined as an incremental rate of alerts (delta) from the prior 12 months to the rest of the study phase.
Table 3. Factors associated with increased alert override by doctors. The outcome was defined as an incremental rate of alerts (delta) from the prior 12 months to the rest of the study phase.
VariablesUnivariate AnalysisMultivariate Analysis
Coefficient95% CIp-ValueCoefficient95% CIp-Value
Doctor Position <0.001
StaffReference
Fellow0.14(0.08, 0.19)<0.0010.53(0.47, 0.58)<0.001
Trainee−0.92(−0.94, −0.90)<0.0010.19(0.17, 0.22)<0.001
Doctor Specialty <0.001
Internal MedicineReference
Surgical Department−2.10(−2.12, −2.07)<0.001−1.72(−1.75, −1.70)<0.001
Others1.59(1.57, 1.62)<0.0011.56(1.53, 1.58)<0.001
Order Drug Variability <0.001
1st quartile (low)Reference
2nd quartile−2.59(−2.61, −2.56)<0.001−2.37(−2.40, −2.35)<0.001
3nd quartile−3.32(−3.35, −3.30)<0.001−3.62(−3.65, −3.60)<0.001
4th quartile (high)−1.39(−1.42, −1.36)<0.001−2.31(−2.35, −2.27)<0.001
Volume of Alerts <0.001
1st quartile (low)Reference
2nd quartile 1.73(1.70, 1.75)<0.0012.60(2.58, 2.63)<0.001
3nd quartile1.40(1.38, 1.43)<0.0012.91(2.88, 2.94)<0.001
4th quartile (high)2.47(2.44, 2.50)<0.0013.29(3.26, 3.33)<0.001
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Cha, W.C.; Jung, W.; Yu, J.; Yoo, J.; Choi, J. Temporal Change in Alert Override Rate with a Minimally Interruptive Clinical Decision Support on a Next-Generation Electronic Medical Record. Medicina 2020, 56, 662. https://doi.org/10.3390/medicina56120662

AMA Style

Cha WC, Jung W, Yu J, Yoo J, Choi J. Temporal Change in Alert Override Rate with a Minimally Interruptive Clinical Decision Support on a Next-Generation Electronic Medical Record. Medicina. 2020; 56(12):662. https://doi.org/10.3390/medicina56120662

Chicago/Turabian Style

Cha, Won Chul, Weon Jung, Jaeyong Yu, Junsang Yoo, and Jinwook Choi. 2020. "Temporal Change in Alert Override Rate with a Minimally Interruptive Clinical Decision Support on a Next-Generation Electronic Medical Record" Medicina 56, no. 12: 662. https://doi.org/10.3390/medicina56120662

Article Metrics

Back to TopTop