1. Introduction
Computerized clinical decision support systems (CCDSS) aim to support healthcare professionals in decision-making in clinical practice. As electronic health records (EHR) have largely replaced paper health records, sufficient reliable clinical data registries from multiple data sources are now available for CCDSS to support clinicians in their daily practice. In the present study, we analyze whether a CCDSS can be used for calculating risk scores such as in this case the CHA2DS2-VASc score.
Checklists are one of the oldest decision support tools. Over time checklists have evaluated from static paper lists in 1940’s to intelligent dynamic checklists [
1,
2]. The modality of a checklist serves as an ideal method for CCDSS to communicate the output of computer algorithms to advise clinicians in real-time at the bedside. Recently, we demonstrated that intelligent dynamic checklists in an intensive care (ICU) environment improved the adherence to best practice by increasing the percentage of checked items from 75% using traditional paper checklists to a 100% score [
3,
4]. Importantly, when the physicians used the dynamic intelligent checklist, there were no aspects overlooked during the daily ICU rounds.
In this manuscript, we used a CCDSS to automatically calculate the CHA2DS2-VASc score (further mentioned as automated clinical decision support system (aCDSS)). Just like an ICU round, there are a lot of items that needs to be checked during a preoperative evaluation. These checks cost time, manpower and are prone for human errors or omitted checks. The transition towards EHR facilitates the application of aCDSS that can perform the calculations of risk score more accurately and efficiently.
In this manuscript we compare a manual calculation of CHA2DS2-VASc scores (manual clinical decision support system (mCDSS)) in patients with atrial rhythm disturbances with an automatic calculation performed by a transparent intelligent algorithm from a aCDSS during a perioperative screening outpatient clinic visit. We hypothesize that the manual calculations contain more erroneous scores than aCDSS due to human errors, while in the meantime the automated calculation is also more efficient.
2. Materials and Methods
2.1. Patient Selection
Patients with atrial rhythm disturbances were included in this analysis when they visited the pre-operative outpatient clinics at the Catharina hospital Eindhoven from October 2017 until January 2018.
Written informed consent was given by all subjects.
Ethical committee approved this study nWMO-2017.81 (local) and W17.144 (national: MEC-U), received 30 November 2017.
2.1.1. Automated CHA2DS2-VASc Score Calculation
Clinical rules to calculate the CHA
2DS
2-VASc were constructed using an aCDSS [
5] designed in the Netherlands (by Gaston Medical, Eindhoven, The Netherlands). Gaston is an aCDSS that was applied in the Catharina hospital Eindhoven as a Pharmalogical support system to analyse errors in prescribing medication in EZIS (Elektronische Ziekenhuis Informatie Systemen; a EHR) created by Chipsoft Holding B.V., Amsterdam, The Netherlands) [
3,
6]. In case of an error, Gaston notifies pharmacists or even give pop-ups in the EHR to alert physicians. It can also screen the EHR and other medical data sources for targeted extraction and analysis of medical data to evaluate each of the CHA
2DS
2-VASc criteria. For this study we included data like the sex, age, hypertension (including measurement in vital parameters > three times systolic blood pressure above 140 mmHg and/or diastolic blood pressure > 90 mmHg), diabetes (including laboratory results fasting glucose > 6 mmol/L and/or normal glucose > 8 mmol/L), thromboembolic disorders (cerebrovascular accident, transient ischemic attack, long embolus and deep venous thrombosis), heart failure, symptomatic arteriosclerosis in the legs, aorta and symptomatic coronary disease (
Figure 1). Besides structured data GASTON can extract information from free-text electronic hospital health record notes such as a medical condition or complications [
7]. Gaston uses a rule engine with various clinical rules consisting of manually build transparent algorithms that are comparable with a decision tree (
Figure 2). The whole system was designed to create or modify the rules easily. The rule engine can manually or automatically be activated to run the algorithm to screen, extract and analyze all relevant data [
8]. Then Gaston calculates the CHA
2DS
2-VASc score and report the result in the EHR, as an item in an intelligent dynamic checklist, or alert the physician with a pop-up. A detailed description of this process for this study is described in
Supplementary S1.
2.1.2. Manual CHA2DS2-VASc Score Calculation
Three physicians were asked to manually calculate the CHA2DS2-VASc scores by evaluating the EHR and other available data sources. They reported the time they needed to analyze one single patient, each separate item of the CHA2DS2-VASc score, and the total CHA2DS2-VASc score.
2.2. Outcomes
The primary outcome of this study is to evaluate if the automatically calculated CHA2DS2-VASc score is as accurate as a manually calculated score which is the current medical standard of practice. Secondary outcome measurements are the time to calculate the CHA2DS2-VASc score automatically and manually, and a comparison between the CCDSS and the manual evaluation to determine the degree of correlation for the following CHA2DS2-VASc score criteria: hypertension, diabetes, thromboembolic disorders (cerebrovascular accident, transient ischemic attack, long embolus and deep venous thrombosis), heart failure, symptomatic arteriosclerosis in the legs and symptomatic coronary disease.
2.3. Statistical Analysis
For statistical analysis SPSS (IBM version 25) was used. Agreement between total CHA
2DS
2-VASc score of each patient, comparing our aCDSS with mCDSS, was assessed by calculating intraclass correlations. We categorized the total CHA
2DS
2-VASc score in three groups: 0 (no anticoagulants necessary), 1–7 (coagulants necessary, but no need to bridge when temporarily stopped) and >7 (coagulants necessary, but indication to bridge with LMWH (low molecular weight heparins) when temporarily stopped). A Bland-Altman plot and regression analysis were used to visualize the agreement between the two different interventions (aCDSS vs. mCDSS). The one-sample t tests from the Bland-Altman plot are shown in
Table 1 and
Table 2. The regression coefficients analysis was performed determine the level of agreement between the total CHA
2DS
2-VASc scores of both interventions. We also calculated the interclass correlation of our aCDSS and mCDSS regarding CHA
2DS
2-VASc score criteria. The investigators performed a comprehensible evaluation of the EHR by unblinding them to Gaston’s output to re-calculate the CHA
2DS
2-VASc score in the patients where the results of the calculations differed between the two interventions. These reviewed and re-calculated scores were then considered by mutual agreement between the investigators to be the true CHA
2DS
2-VASc score to clarify if the differences were due to a human or an algorithmic error.
3. Results
In total, 224 patients were selected for analysis. The mean age was 72, of whom 67% was male and the ASA (Physical Status Classification System) ranged from 2 to 4. The three physicians divided the 224 patients equally for their manual evaluations.
The interclass correlation of aCDSS and mCDSS of the total CHA
2DS
2-VASc score, was 0.859 (95%-CI: 0.611–0.931) (
Table 3).
Bland-Altman analysis indicated a bias of −0.79 with limit of agreement (95%-CI) between −2.95 and 1.37 between the two interventions. Linear regression analysis indicated no significant relation (R = 0.477 (95%-CI: 0.435–0.519),
p > 0.05) (
Figure 3).
The Cohen’s kappa score was 0.42 for the total CHA
2DS
2-VASc score which indicates a moderate level of agreement between the two interventions (
Table 1 and
Table 2). This result derives from the low correlation between the two interventions for the CHA
2DS
2-VASc score criteria: hypertension, diabetes and heart failure (
Table 3). A good correlation was found for the CHA
2DS
2-VASc score criteria: thromboembolic disorders, symptomatic arteriosclerosis in the legs, and symptomatic coronary disease. A retrospective evaluation performed by the investigators showed that 28% of the calculations had a 100% agreement, 72% did not.
Figure 4 shows that mCDSS missed mainly items in heart failure, hypertension and diabetes compared to aCDSS.
The aCDSS took 40 min to calculate all CHA2DS2-VASc scores of the 224 patients. This is on average 11 s per patient. The three physicians took 180 min to calculate these scores with an average of 48 s per patient. The aCDSS calculated the total CHA2DS2-VASc score 4.5 times faster compared to physicians.
4. Discussion
This study shows a moderate agreement between CHA2DS2-VASc scores derived mCDSS and aCDSS. The differences between the scores were in 88% of the patients a result of a human error. Besides being more accurate was the calculation with the aCDSS also more time efficient. These findings imply that the application of such an aCDSS can support clinicians during preoperative screening outpatient clinic visit.
Our results are in line with previous studies, which showed that decision support systems can help decrease decision conflicts and increase knowledge of patients with atrial rhythm disturbances about risks atrial rhythm disturbances and treatment options [
9,
10,
11,
12,
13,
14,
15]. Others provided evidence that it also helps to improve adherence to guideline in the treatment of atrial fibrillation [
11]. In the study of Wang et al. [
13] and Silbernagel et al. [
13] it was shown that physicians prescribed more anticoagulants due to the clinical decision support system. Moreover, incidence of major bleeding tended to be lower when clinical support systems where used [
9]. In our study we also conclude that our aCDSS helps the physician make an accurate decision according to the guideline regarding the use of anticoagulants in perioperative care for patients with atrial fibrillation.
Our results show a moderate agreement between the aCDSS and mCDSS calculated CHA2DS2-VASc score due to human errors which is a result of the low level of agreement for several criteria of the CHA2DS2-VASc score. Further analysis showed that the aCDSS included the correct data on different items of the CHA2DS2-VASc score, whereas during mCDSS, items such as diabetes and heart failure were sometimes overlooked. Therefore, the aCDSS seems to be a more accurate method for calculating the CHA2DS2-VASc score than mCDSS of the patient health records, while saving time for the clinician.
This study has various limitations. First, this was a retrospective analysis while manual calculations can be performed with the help of a patient during preoperative out-patient clinic visits. Another limitation is that three different physicians divided the 224 patients for mCDSS. Differences of the performance between these physicians might have influenced the results. However, differences between clinicians are common in practice and the constant method of calculations favors the aCDSS. In addition, it will be easier to change practice with the aCDSS if the CHA2DS2-VASc score change in the future, because with the aCDSS the algorithm can be updated easily while human training takes more time and effort. Bias might have occurred since the physicians were not blinded for the purpose of this study. They may have paid extra attention to retrieve all criteria for calculating the CHA2DS2-VASc score which may not reflect real practice. The level of agreement might therefore be even smaller. Finally, we do not know whether it is possible that the agreement between the two interventions on individual items in the CHA2DS2-VASc score is based on chance, but our results after retrospective analysis indicate otherwise.
For the future, an automated calculation of the HAS-BLED score might be added to the algorithm to provide the clinicians a well-balanced advice. Furthermore, a clinical study is needed that evaluates the accuracy, the patient’s and clinician’s benefits, and the usability of a real-time automated CHA2DS2-VASc score calculations.
5. Conclusions
The agreement of a mCDSS CHA2DS2-VASc score as current standard of care and calculation by an aCDSS is low. This result implies that calculations performed by an aCDSS might be more accurate and more time efficient than a manual calculation. Implementation of this aCDSS in clinical practice can therefore support physicians during their pre-operative outpatient clinics.
Author Contributions
Conceptualization, C.v.G., R.A.B. and H.H.M.K.; methodology, C.v.G. and R.A.B.; software, R.A.B. and H.H.M.K.; validation, C.v.G.; formal analysis, C.v.G., R.A.B. and A.J.R.D.B.D.; investigation, C.v.G., R.A.B., A.J.R.D.B.D. and E.H.J.M.; writing—original draft preparation, C.v.G.; writing—review and editing, C.v.G., R.A.B. and A.J.R.D.B.D.; visualization, C.v.G. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Ethical committee approved this study nWMO-2017.8.1 (local) and W17.144 (national: MEC-U).
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement
Not applicable.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Webster, C.S. Checklists, cognitive aids, and the future of patient safety. Br. J. Anaesth. 2017, 119, 178–181. [Google Scholar] [CrossRef]
- Singh, S.K. (Ed.) Database Systems: Concepts, Design and Applications, 3rd ed.; Pearson Education India: Noida, India, 2009. [Google Scholar]
- De Bie, A.J.R.; Nan, S.; Vermeulen, L.R.E.; Van Gorp, P.M.E.; Bouwman, R.A.; Bindels, A.J.G.H.; Korsten, H.H.M. Intelligent dynamic clinical checklists improved checklist compliance in the intensive care unit. Br. J. Anaesth. 2017, 119, 231–238. [Google Scholar] [CrossRef] [PubMed]
- De Bie, A.J.R.; Mestrom, E.; Compagner, W.; Nan, S.; van Genugten, L.; Dellimore, K.; Eerden, J.; van Leeuwen, S.; van de Pol, H.; Schuling, F.; et al. Intelligent checklists improve checklist compliance in the intensive care unit: A prospective before-and-after mixed-method study. Br. J. Anaesth. 2021, 126, 404–414. [Google Scholar] [CrossRef] [PubMed]
- van Berkel, M.; Wasylewicz, A.; Kreeftenberg, H.G.; Boer, A.-K. GASTON, de elektronische ondersteuning voor zorgprofessionals. Ned. Tijdschr. Klin. Chem. Labgeneesk. 2015, 40, 26–29. [Google Scholar]
- Nan, S.; Van Gorp, P.; Lu, X.; Kaymak, U.; Korsten, H.; Vdovjak, R.; Duan, H. A meta-model for computer executable dynamic clinical safety checklists. BMC Med. Inform. Decis. Mak. 2017, 17, 170. [Google Scholar] [CrossRef] [PubMed]
- Wasylewicz, A.; van de Burgt, B.; Weterings, A.; Jessurun, N.; Korsten, E.; Egberts, T.; Bouwman, A.; Kerskes, M.; Grouls, R.; van der Linden, C. Identifying adverse drug reactions from free-tekst electronic hospitaal health record notes. Br. J. Clin. Pharmacol. 2022, 88, 1235–1245. [Google Scholar] [CrossRef] [PubMed]
- Scheepers-Hoeks, A.M.; Klijn, F.; van der Linden, C.; Grouls, R.; Ackerman, E.; Minderman, N.; Bergmans, J.; Korsten, E. Chapter 15: Clinical decision support systems for “Making in easy to do it right”. In Data Mining: Concepts, Methodologies, Tools, and Applications; IGI Global: Hershey, PA, USA, 2012. [Google Scholar]
- Sakurai, R.; Ohe, K. Effects of Computerized Guideline-Oriented Clinical Decision Support System on Antithrombotic Therapy in Patients with Atrial Fibrillation: A Systematic Review and Meta-Analysis. Stud. Health Technol. Inform. 2019, 264, 768–772. [Google Scholar] [CrossRef] [PubMed]
- Sheibani, R.; Nabovati, E.; Sheibani, M.; Abu-Hanna, A.; Heidari-Bakavoli, A.; Eslami, S. Effects of Computerized Decision Support Systems on Management of Atrial Fibrillation: A Scoping Review. J. Atr. Fibrillation 2017, 10, 1579. [Google Scholar] [CrossRef] [PubMed]
- Sheibani, R.; Sheibani, M.; Heidari-Bakavoli, A.; Abu-Hanna, A.; Eslami, S. The Effect of a Clinical Decision Support System on Improving Adherence to Guideline in the Treatment of Atrial Fibrillation: An Interrupted Time Series Study. J. Med. Syst. 2018, 42, 26. [Google Scholar] [CrossRef] [PubMed]
- Van Doorn, S.; Rutten, F.; O’Flynn, C.; Oudega, R.; Hoes, A.; Moons, K.; Geersing, G. Effectiveness of CHA2DS2-VASc based decision support on stroke prevention in atrial fibrillation: A cluster randomised trial in general practice. Int. J. Cardiol. 2018, 273, 123–129. [Google Scholar] [CrossRef] [PubMed]
- Wang, Y.; Bajorek, B. Pilot of a Computerised Antithrombotic Risk Assessment Tool Version 2 (CARATV2.0) for stroke prevention in atrial fibrillation. Cardiol. J. 2017, 24, 176–187. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Silbernagel, G.; Spirk, D.; Hager, A.; Baumgartner, I.; Kucher, N. Electronic Alert System for Improving Stroke Prevention Among Hospitalized Oral-Anticoagulation-Naïve Patients with Atrial Fibrillation: A Randomized Trial. J. Am. Heart Assoc. 2016, 5, e003776. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Abidi, S.R.; Cox, J.; Abusharekh, A.; Hashemian, N.; Abidi, S.S.R. A Digital Health System to Assist Family Physicians to Safely Prescribe NOAC Medications; European Federation for Medical Informatics: Athens, Greece; IOS Press: Amsterdam, The Netherlands, 2016. [Google Scholar]
| Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).