Next Article in Journal
High-Dose Supplementation of Folic Acid in Infertile Men Improves IVF-ICSI Outcomes: A Randomized Controlled Trial (FOLFIV Trial)
Next Article in Special Issue
Characteristics of Critically Ill Patients with COVID-19 Compared to Patients with Influenza—A Single Center Experience
Previous Article in Journal
A New Outlook on the Ability to Accumulate an Iodine Contrast Agent in Solid Lung Tumors Based on Virtual Monochromatic Images in Dual Energy Computed Tomography (DECT): Analysis in Two Phases of Contrast Enhancement
Previous Article in Special Issue
Vasoactive-Inotropic Score as an Early Predictor of Mortality in Adult Patients with Sepsis
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JCM
Volume 10
Issue 9
10.3390/jcm10091875
jcm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Using Machine Learning to Predict Invasive Bacterial Infections in Young Febrile Infants Visiting the Emergency Department

by
I-Min Chiu
1,2,
Chi-Yung Cheng
1,2,
Wun-Huei Zeng
2,
Ying-Hsien Huang
3 and
Chun-Hung Richard Lin
2,*
1
Department of Emergency Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung 833, Taiwan
2
Department of Computer Science and Engineering, National Sun Yat-Sen University, Kaohsiung 804, Taiwan
3
Department of Pediatrics, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung 833, Taiwan
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2021, 10(9), 1875; https://doi.org/10.3390/jcm10091875
Submission received: 27 March 2021 / Revised: 19 April 2021 / Accepted: 23 April 2021 / Published: 26 April 2021
(This article belongs to the Special Issue Management of Sepsis and Septic Shock)
Browse Figure
Versions Notes

Abstract

:
Background: The aim of this study was to develop and evaluate a machine learning (ML) model to predict invasive bacterial infections (IBIs) in young febrile infants visiting the emergency department (ED). Methods: This retrospective study was conducted in the EDs of three medical centers across Taiwan from 2011 to 2018. We included patients age in 0–60 days who were visiting the ED with clinical symptoms of fever. We developed three different ML algorithms, including logistic regression (LR), supportive vector machine (SVM), and extreme gradient boosting (XGboost), comparing their performance at predicting IBIs to a previous validated score system (IBI score). Results: During the study period, 4211 patients were included, where 126 (3.1%) had IBI. A total of eight, five, and seven features were used in the LR, SVM, and XGboost through the feature selection process, respectively. The ML models can achieve a better AUROC value when predicting IBIs in young infants compared with the IBI score (LR: 0.85 vs. SVM: 0.84 vs. XGBoost: 0.85 vs. IBI score: 0.70, p-value < 0.001). Using a cost sensitive learning algorithm, all ML models showed better specificity in predicting IBIs at a 90% sensitivity level compared to an IBI score > 2 (LR: 0.59 vs. SVM: 0.60 vs. XGBoost: 0.57 vs. IBI score >2: 0.43, p-value < 0.001). Conclusions: All ML models developed in this study outperformed the traditional scoring system in stratifying low-risk febrile infants after the standardized sensitivity level.

1. Introduction

Febrile infants ≤ 60 days of age are prone to contracting serious bacterial infections (SBIs). However, no reliable physical examination findings or specialized routine laboratory investigations exist that can aid a clinician in differentiating an SBI from benign viral infections. Although several criteria (Boston, Philadelphia, and Rochester) proposed to stratify low-risk patients [1,2,3] are regarded as commonly used assessment tools, the development of novel diagnostic tools and changes in epidemiology [4] have led to a decrease in the adherence to these guidelines by clinical physicians [5,6,7]. This issue urged the formulation of a new management guideline to effectuate improvements on the several protocols proposed earlier [8,9].
Urinary tract infections (UTIs) constitute the majority of SBIs in febrile infants [10,11,12]. Urinalysis is a highly noninvasive and sensitive method that facilitates diagnosis for UTI and is performed in an emergency department (ED) [13]. Therefore, focusing on the specific type of infection was more important to ED physicians than the generalized concept of SBIs, in terms of evaluation and management. Consequently, the term “invasive bacterial infection (IBI)”, including bacteremia and bacterial meningitis, which were more likely to result in adverse outcomes [14], gained more popularity [15,16,17].
Several studies focusing on developing prediction models for identifying young infants at low-risk for IBIs have been reported over the past few years [18,19]. In 2014, Mintegi et al. adopted various parameters, including clinical appearance, age, urinalysis results, and laboratory examinations such as white blood cell (WBC) count, c-reactive protein (CRP), and procalcitonin tests, to achieve better prediction performance than that of existing protocols [19]. As procalcitonin and CRP levels are not routinely obtained in general practice, Aronson et al. further proposed a prediction model called “IBI score” that employed only four characteristics (age, temperature, urinalysis results, and absolute neutrophil count) to enhance the generalizability of the model [20].
Machine learning (ML), as an application of artificial intelligence using computer-based algorithms that can directly learn and identify trends from a dataset, has exhibited promising results on the diagnosis of sepsis and predicted clinical outcomes in adult patients in the ED [21,22,23,24]. However, to the best of our knowledge, a study on the prediction of IBIs in young infants based on ML algorithms has not been conducted. Thus, the objective of this study was to develop and validate an ML model to predict the possibility of IBIs among young febrile infants based on certain clinical parameters.

2. Materials and Methods

2.1. Study Design

This paper details a retrospective study conducted in the EDs of three medical centers across Taiwan, from 1 January 2011 to 31 December 2018. All three hospitals are branches of the same healthcare system and are geographically well dispersed. The three hospitals were located in northern, middle east, and southern Taiwan with annual pediatric ED visit of 30,000, 10,000, and 20,000, respectively. The study was approved by the institutional review board of the Chang Gung Medical Foundation (IRB number: 202001949B0, Date of Approval: 24 November 2020). The patient and physician records and information were anonymized and deidentified prior to the analysis.

2.2. Patient Population and Data Collection

All patients aged 0–60 days visiting the ED with clinical symptoms of fever in the period mentioned above were included as subjects in this study. Patients with subjectively described fever or elevated body temperature noted at ED admission were both included. Patients with missing laboratory data, uncertain test results (due to the inability to obtain blood or cerebral spinal fluid (CSF) culture reports), or complex chronic conditions were excluded from the study population, along with transfers from other hospitals and premature infants [25]. The parameters collected were age, sex, vital signs during the ED visit, and laboratory test results. In this study, the growth of a pathogen observed in the blood and CSF cultures was considered to be an IBI [16].

2.3. Feature Selection

Previous statistical analyses focused on the association between features and outcomes, in which p-values were used as a measure of association [26,27]. In contrast, we determined the order of importance among the features, using a forward stepwise method to control overfitting, and included the most suitable subsets of these features. This method used a sequence of steps so that only one feature can enter at a time. In most cases, the process converged to a subset of features that helped the ML model yield the best prediction performance, which typically used the value of area under the receiver-operating characteristic curve [AUROC] [28] as an evaluation tool in an imbalanced dataset. The best feature subset of the AUROC value was used for ML training.

2.4. Machine Learning Models

We developed and trained three ML algorithms in this study, i.e., logistic regression (LR), support vector machine (SVM), and extreme gradient boosting (XGBoost), using the TensorFlow 2.3 package. LR is a predictive analysis algorithm based on the concept of probability, which uses the sigmoid function to derive the predicted output for classification. SVM is another ML model commonly used for solving classification problems. The SVM constructs a set of hyperplanes in a higher-dimensional space to obtain the largest distance to the nearest training data point of any class. The larger the distance of this margin achieved, the lower the generalization error [29]. This method usually performed better than LR. In contrast, XGBoost is an ensemble learning algorithm proposed for sparse data and weighted quantile sketch. XGBoost was developed to solve real-world imbalanced data problems using the stacking of decision trees [30].
As the probability of IBI was low in the general population visiting the ED, IBI prediction was considered as an imbalanced class problem. In this study, we replaced the loss function of the cross entropy, typically used in ML classification problem, with the cost sensitive matrix. The essence of the cost sensitive method helps the ML model make the optimal decision while accounting for the cost of the prediction error during training to avoid deviations in the prediction performance to the non-IBI group [31].

2.5. Model Evaluation and Statistical Analysis

All included patients were randomly assigned into five groups which contained 20% of patients, and the developed ML models were assessed using 5-fold cross validation, which means one group was regarded as a validation set and the other four were regarded as training sets each time. The result of model performance was then compared to the IBI score devised by Aronson et al. [20] to predict IBIs in young febrile infants ≤ 60 days old. The IBI score were developed statistically and ranged from 0–10, the higher scores indicated higher risk of IBIs (Table 1). Other statistically derived scoring systems in previous studies, such as the step-by-step method [19] and lab-score [18], were not included for comparison, as the procalcitonin test was not conducted for most of the patients in this study.
We estimated the AUROC curve with sensitivity and specificity as the performance parameters. For each model, we adjusted the class weight on the outcome prediction to prioritize sensitivity at a level of 90% and compared the predicted results with the IBI score at the same level (IBI score ≥ 2). The above statistical analyses were performed using Python 3.8 with the Scikit-learn 0.22.2 package [32].

3. Results

In the study period, 4211 patients were included for analysis. Patients’ distribution and demographics in three hospitals were shown in Table 2. A total of 126 (3.1%) patients had IBI; among them, 117 (2.8%) and 18 (0.4%) patients had bacteremia and bacterial meningitis, respectively. The clinical characteristics of the patients with and without IBI are demonstrated in Table 3. The factors significantly associated with IBI were age (31 (20–43) vs. 36 (23–50) days, p = 0.001), temperature at triage (38.4 (37.9–38.9) vs. 37.7 (37.1–38.3) °C, p < 0.001), heart rate (177 (161–189) vs. 159 (143–175) per minute, p < 0.001), and laboratory tests, namely hemoglobin, platelet, C-reactive protein, differential count of WBC, including neutrophil, band, eosinophil, lymphocyte, and abnormal urine tests. The IBI score was significantly higher in patients with IBI than that of those without IBI (4 (2–6) vs. 2 (0–4), p < 0.001).
The results of the forward stepwise feature selection method performed for each of the ML models are described in Table 4. Among the three models, SVM used the least number of features to achieve the highest AUROC value (0.84 ± 0.03). Among these, the common shared features were CRP and band. The neutrophil count and heart rate were also adopted in two out of three models.
Analysis of variance (ANOVA) was performed to compare the performance of the ML models with the IBI score (Table 5). All the ML models outperformed the IBI score in the prediction of IBI in young febrile infants based on the AUROC level. The receiver operating characteristic curves of all the evaluated models are illustrated in Figure 1. There was no significant difference in the AUROC value among the selected ML models in the post-hoc test. After class-weight adjustment, the ML models demonstrated no statistical difference in sensitivity of IBI prediction compared with using an IBI score of ≥2 as a predictor. In contrast, the ML models displayed better specificity levels than those of the IBI score. Among the three models, the specificity for IBI prediction was slightly better in LR and SVM than XGBoost, with significant differences in post-hoc analysis.

4. Discussion

Ramgopal et al. [33] developed various ML models, including the LR, SVM, RF, and single-layer neural networks, to predict SBIs in young febrile infants. With a 9.3% incidence of SBIs in the included patients, the ML techniques were able to produce better results than those of previous scoring systems. In this study, we extended the above-mentioned study to predict IBIs in young febrile infants visiting the ED, as bacteremia and bacterial meningitis are difficult to diagnose in EDs, and UTIs can usually be diagnosed with a urinalysis. This finding showed that urinalysis is the most important feature for prediction of SBI in that study, while UTIs account for 88.4% in all SBI patients. Our study focused on predicting IBIs and showed that ML was still able to achieve better AUROC values when predicting IBIs compared to the IBI score.
Another difference between our study and Ramgopal et al. is that we did not include procalcitonin as a feature in developing ML models. We also did not exclude patients of critical appearance. The major reason for the design is we tend to keep as many patients as possible to avoid model bias on clinical application. The difference can be noticed as only 20.0% patients encountered were included in Ramgopal’s study, and most of the patients encountered were included in our study.
In this study, we used a cost sensitive technique to manage the imbalanced data problem. Cost-sensitive algorithms are a subfield of ML and are commonly adopted during the training of ML models as imbalanced data is frequently observed in real-world scenarios. In basic supervised ML models, the algorithm tends to lean toward the majority of the population, which affects the application of the model. For example, if disease X occurred in only 1% of the study population, the model can easily predict that no one in the population has disease X as this would be true in 99% of the cases. Cost-sensitive methods were hence developed to solve this problem by assigning more cost when the model predicted trends in the minority population inaccurately during training [34,35].
With aid from practical ML applications, the result of this study can reduce unnecessary admissions and prolonged antibiotic use in low-risk patients. In clinical practices, physicians can prioritize high sensitivity to differentiate IBIs from benign viral infections in young febrile infants. With the low prevalence of IBIs among young infants, previous models can only achieve a 46.9–52.0% specificity in the prediction of IBIs at a 90% sensitivity level [9,20]. Using the cost sensitive algorithm, we were able to set the sensitivity of the prediction models to 90% and evaluate their performance on the specificity at a similar standard. The results showed that all three models outperformed the IBI score at stratifying low-risk patients (Table 5).
A total of eight, five, and seven features were used in the LR, SVM, and XGboost, respectively, to achieve optimal AUROC levels during training through the stepwise feature selection method (Table 4). The common features extracted in the ML models based on the adopted parameters were of reasonable relevance, as CRP, neutrophil, and band counts were reported to have a positive correlation with bacterial infections in young infants [17,36]. Compared to the IBI score, which consisted of four different parameters to predict IBI, commonly shared parameters in ML models were ANC and neutrophil. Age and temperature were only adopted in LR. SVM and XGBoost did not acquire these two features and were still able to achieve similar AUC values as LR. Increase of temperature was one of the vital signs that correlated with pediatric sepsis. However, a recent study showed that although IBI was more likely with higher temperatures, degree of fever should not be used in the risk stratification of febrile infants [37]. On the other hand, bacteremia was associated with UTIs in 4–10% of febrile infants [38]. It was reasonable to include abnormal urinalysis in ML models to predict IBIs in febrile infants. One possible cause that all of the ML models did not use abnormal urinalysis as a ML feature is that we adopted CRP as one of the features, which was proved to be able to stratify risk in febrile infants with leukocyturia in multiple studies [39,40].
A multicenter study conducted in 2017 discovered that none of the parameters in a complete blood cell count were able to efficiently predict IBIs in young infants [16]. In this study, we used the stepwise feature-selection method not only to demonstrate the real-world applicability of the extracted features, but also to attempt to obtain the highest possible accuracy for each of the N-feature choices among all of the feature sets. The idea behind deploying a forward stepwise regression for feature selection, instead of a regression analysis, was to achieve better performance from the ML models using feature combinations, rather than combining statistically significant features. Consequently, we were also able to enumerate the selected features in an orderly manner based on their level of importance to the model.
Although ML models can directly learn from the dataset and improve the diagnosis and outcome prediction in several areas across medicine, the black-box nature of each network can present challenges when applying or extending the proposed models to clinical practice. In this study, we attempted to overcome this problem using the stepwise feature-selection method and selecting features with better generalizability, such as age, sex, vital signs, and laboratory test results. We speculate that the results of our study can provide a reference for the application of ML models across various healthcare systems. However, we observed a limitation. As this is a retrospective study, data could only be collected based on past medical records. More recent laboratory tests that may have direct links to pediatric sepsis, such as procalcitonin or lactate [41,42], could not be included as a large amount of data was not obtained during the ED visit. This situation limited the comparison of the developed ML models with other recently reported prediction models.

5. Conclusions

In this study, we developed and evaluated the performance of three ML models, including LR, SVM, and XGBoost, to predict IBIs in young febrile infants in the ED. All the ML models outperformed traditional scoring systems in stratifying low-risk febrile infants after standardized sensitivity level. Among the three, the SVM algorithm exploited the least number of features to achieve the optimal result.

Author Contributions

Conceptualization, I.-M.C. and C.-H.R.L.; methodology, W.-H.Z.; software, W.-H.Z.; validation, C.-H.R.L. and Y.-H.H.; formal analysis, I.-M.C.; data curation, C.-Y.C. and Y.-H.H.; writing—original draft preparation, I.-M.C.; writing—review and editing, Y.-H.H. and C.-H.R.L. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of Chang Gung Medical Foundation (protocol code 202001949B0 and date of approval 24 November 2020).

Informed Consent Statement

Not applicable.

Data Availability Statement

Data was obtained from Chang Gung Medical Foundation and are available from corresponding author with the permission of Chang Gung Medical Foundation.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Baskin, M.N.; O’Rourke, E.J.; Fleisher, G.R. Outpatient treatment of febrile infants 28 to 89 days of age with intramuscular administration of ceftriaxone. J. Pediatr. 1992, 120, 22–27. [Google Scholar] [CrossRef]
  2. Baker, M.D.; Bell, L.M.; Avner, J.R. Outpatient management without antibiotics of fever in selected infants. N. Engl. J. Med. 1993, 329, 1437–1441. [Google Scholar] [CrossRef]
  3. Jaskiewicz, J.A.; McCarthy, C.A.; Richardson, A.C.; White, K.C.; Fisher, D.J.; Dagan, R.; Powell, K.R. Febrile infants at low risk for serious bacterial infection—An appraisal of the Rochester criteria and implications for management. Febrile Infant Collaborative Study Group. Pediatrics 1994, 94, 390–396. [Google Scholar]
  4. Woll, C.; Neuman, M.I.; Pruitt, C.M.; Wang, M.E.; Shapiro, E.D.; Shah, S.S.; McCulloh, R.J.; Nigrovic, L.E.; Desai, S.; DePorre, A.G.; et al. Epidemiology and Etiology of Invasive Bacterial Infection in Infants ≤ 60 Days Old Treated in Emergency Departments. J. Pediatr. 2018, 200, 210–217. [Google Scholar] [CrossRef] [PubMed]
  5. Jain, S.; Cheng, J.; Alpern, E.R.; Thurm, C.; Schroeder, L.; Black, K.; Ellison, A.M.; Stone, K.; Alessandrini, E.A. Management of febrile neonates in US pediatric emergency departments. Pediatrics 2014, 133, 187–195. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Meehan, W.P.; Fleegler, E.; Bachur, R.G. Adherence to guidelines for managing the well-appearing febrile infant: Assessment using a case-based, interactive survey. Pediatr. Emerg. Care 2010, 26, 875–880. [Google Scholar] [CrossRef] [PubMed]
  7. Klarenbeek, N.N.; Keuning, M.; Hol, J.; Pajkrt, D.; Plötz, F.B. Fever Without an Apparent Source in Young Infants: A Multicenter Retrospective Evaluation of Adherence to the Dutch Guidelines. Pediatr. Infect. Dis. J. 2020, 39, 1075–1080. [Google Scholar] [CrossRef] [PubMed]
  8. Bonadio, W. In Search of an Ideal Protocol to Distinguish Risk For Serious Bacterial Infection in Febrile Young Infants. J. Pediatr. 2021, 231, 32–34. [Google Scholar] [CrossRef] [PubMed]
  9. Gomez, B.; Mintegi, S.; Bressan, S.; Da Dalt, L.; Gervaix, A.; Lacroix, L. Validation of the “Step-by-Step” Approach in the Management of Young Febrile Infants. Pediatrics 2016, 138, e20154381. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  10. Byington, C.L.; Reynolds, C.C.; Korgenski, K.; Sheng, X.; Valentine, K.J.; Nelson, R.E.; Daly, J.A.; Osguthorpe, R.J.; James, B.; Savitz, L.; et al. Costs and infant outcomes after implementation of a care process model for febrile infants. Pediatrics 2012, 130, e16–e24. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Garra, G.; Cunningham, S.J.; Crain, E.F. Reappraisal of criteria used to predict serious bacterial illness in febrile infants less than 8 weeks of age. Acad. Emerg. Med. 2005, 12, 921–925. [Google Scholar] [CrossRef]
  12. Hui, C.; Neto, G.; Tsertsvadze, A.; Yazdi, F.; Tricco, A.C.; Tsouros, S.; Skidmore, B.; Daniel, R. Diagnosis and management of febrile infants (0–3 months). Evid. Rep. Technol. Assess. 2012, 205, 1–297. [Google Scholar]
  13. Shah, A.P.; Cobb, B.T.; Lower, D.R.; Shaikh, N.; Rasmussen, J.; Hoberman, A.; Wald, E.R.; Rosendorff, A.; Hickey, R.W. Enhanced versus automated urinalysis for screening of urinary tract infections in children in the emergency department. Pediatr. Infect. Dis. J. 2014, 33, 272–275. [Google Scholar] [CrossRef]
  14. Pruitt, C.M.; Neuman, M.I.; Shah, S.S.; Shabanova, V.; Woll, C.; Wang, M.E.; Alpern, E.R.; Williams, D.J.; Sartori, L.; Desai, S. Factors associated with adverse outcomes among febrile young infants with invasive bacterial infections. J. Pediatr. 2019, 204, 177–182.e171. [Google Scholar] [CrossRef]
  15. Talbert, A.W.; Mwaniki, M.; Mwarumba, S.; Newton, C.R.; Berkley, J.A. Invasive bacterial infections in neonates and young infants born outside hospital admitted to a rural hospital in Kenya. Pediatr. Infect. Dis. J. 2010, 29, 945. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Cruz, A.T.; Mahajan, P.; Bonsu, B.K.; Bennett, J.E.; Levine, D.A.; Alpern, E.R.; Nigrovic, L.E.; Atabaki, S.M.; Cohen, D.M.; VanBuren, J.M. Accuracy of complete blood cell counts to identify febrile infants 60 days or younger with invasive bacterial infections. JAMA Pediatr. 2017, 171, e172927. [Google Scholar] [CrossRef]
  17. Chiu, I.-M.; Huang, L.-C.; Chen, I.-L.; Tang, K.-S.; Huang, Y.-H. Diagnostic values of C-reactive protein and complete blood cell to identify invasive bacterial infection in young febrile infants. Pediatr. Neonatol. 2019, 60, 197–200. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  18. Bressan, S.; Gomez, B.; Mintegi, S.; Da Dalt, L.; Blazquez, D.; Olaciregui, I.; de la Torre, M.; Palacios, M.; Berlese, P.; Ruano, A. Diagnostic performance of the lab-score in predicting severe and invasive bacterial infections in well-appearing young febrile infants. Pediatr. Infect. Dis. J. 2012, 31, 1239–1244. [Google Scholar] [CrossRef]
  19. Mintegi, S.; Bressan, S.; Gomez, B.; Da Dalt, L.; Blázquez, D.; Olaciregui, I.; de la Torre, M.; Palacios, M.; Berlese, P.; Benito, J. Accuracy of a sequential approach to identify young febrile infants at low risk for invasive bacterial infection. Emerg. Med. J. 2014, 31, e19–e24. [Google Scholar] [CrossRef] [Green Version]
  20. Aronson, P.L.; Shabanova, V.; Shapiro, E.D.; Wang, M.E.; Nigrovic, L.E.; Pruitt, C.M.; DePorre, A.G.; Leazer, R.C.; Desai, S.; Sartori, L.F.; et al. A Prediction Model to Identify Febrile Infants ≤ 60 Days at Low Risk of Invasive Bacterial Infection. Pediatrics 2019, 144, e20183604. [Google Scholar] [CrossRef]
  21. Delahanty, R.J.; Alvarez, J.; Flynn, L.M.; Sherwin, R.L.; Jones, S.S. Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis. Ann. Emerg. Med. 2019, 73, 334–344. [Google Scholar] [CrossRef]
  22. Shafaf, N.; Malek, H. Applications of machine learning approaches in emergency medicine; a review article. Arch. Acad. Emerg. Med. 2019, 7, 34. [Google Scholar] [PubMed]
  23. Taylor, R.A.; Pare, J.R.; Venkatesh, A.K.; Mowafi, H.; Melnick, E.R.; Fleischman, W.; Hall, M.K. Prediction of in-hospital mortality in emergency department patients with sepsis: A local big data–driven, machine learning approach. Acad. Emerg. Med. 2016, 23, 269–278. [Google Scholar] [CrossRef] [Green Version]
  24. Horng, S.; Sontag, D.A.; Halpern, Y.; Jernite, Y.; Shapiro, N.I.; Nathanson, L.A. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS ONE 2017, 12, e0174708. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Feudtner, C.; Feinstein, J.A.; Zhong, W.; Hall, M.; Dai, D. Pediatric complex chronic conditions classification system version 2: Updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatr. 2014, 14, 199. [Google Scholar] [CrossRef] [Green Version]
  26. Malhi, A.; Gao, R.X. PCA-based feature selection scheme for machine defect classification. IEEE Trans. Instrum. Meas. 2004, 53, 1517–1525. [Google Scholar] [CrossRef]
  27. Song, F.; Guo, Z.; Mei, D. Feature selection using principal component analysis. In Proceedings of the 2010 International Conference on System Science, Engineering Design and Manufacturing Informatization, Yichang, China, 12–14 November 2010; pp. 27–30. [Google Scholar]
  28. Tanaka, K.; Kurita, T.; Meyer, F.; Berthouze, L.; Kawabe, T. Stepwise feature selection by cross validation for EEG-based Brain Computer Interface. In Proceedings of the 2006 IEEE International Joint Conference on Neural Network Proceedings, Vancouver, BC, Canada, 16–21 July 2006; pp. 4672–4677. [Google Scholar]
  29. Bastanlar, Y.; Ozuysal, M. Introduction to machine learning. Methods Mol. Biol. 2014, 1107, 105–128. [Google Scholar] [CrossRef] [Green Version]
  30. Chen, T.; Guestrin, C. XGBoost: A Scalable Tree Boosting System. arXiv 2016, arXiv:1603.02754. [Google Scholar]
  31. Elkan, C. The foundations of cost-sensitive learning. In Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 13–19 July 2018; pp. 973–978. [Google Scholar]
  32. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
  33. Ramgopal, S.; Horvat, C.M.; Yanamala, N.; Alpern, E.R. Machine learning to predict serious bacterial infections in young febrile infants. Pediatrics 2020, 146, e20194096. [Google Scholar] [CrossRef] [PubMed]
  34. Cao, P.; Zhao, D.; Zaiane, O. An optimized cost-sensitive SVM for imbalanced data learning. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Singapore, 11–14 May 2020; pp. 280–292. [Google Scholar]
  35. Xia, Y.; Liu, C.; Liu, N. Cost-sensitive boosted tree for loan evaluation in peer-to-peer lending. Electron. Commer. Res. Appl. 2017, 24, 30–49. [Google Scholar] [CrossRef]
  36. Yo, C.-H.; Hsieh, P.-S.; Lee, S.-H.; Wu, J.-Y.; Chang, S.-S.; Tasi, K.-C.; Lee, C.-C. Comparison of the test characteristics of procalcitonin to C-reactive protein and leukocytosis for the detection of serious bacterial infections in children presenting with fever without source: A systematic review and meta-analysis. Ann. Emerg. Med. 2012, 60, 591–600. [Google Scholar] [CrossRef] [PubMed]
  37. Michelson, K.A.; Neuman, M.I.; Pruitt, C.M.; Desai, S.; Wang, M.E.; DePorre, A.G.; Leazer, R.C.; Sartori, L.F.; Marble, R.D.; Rooholamini, S.N.; et al. Height of fever and invasive bacterial infection. Arch. Dis. Child. 2020, Aug, 20. [Google Scholar] [CrossRef] [PubMed]
  38. Schroeder, A.R.; Shen, M.W.; Biondi, E.A.; Bendel-Stenzel, M.; Chen, C.N.; French, J.; Lee, V.; Evans, R.C.; Jerardi, K.E.; Mischler, M. Bacteraemic urinary tract infection: Management and outcomes in young infants. Arch. Dis. Child. 2016, 101, 125–130. [Google Scholar] [CrossRef] [PubMed]
  39. Schnadower, D.; Kuppermann, N.; Macias, C.G.; Freedman, S.B.; Baskin, M.N.; Ishimine, P.; Scribner, C.; Okada, P.; Beach, H.; Bulloch, B.; et al. Febrile infants with urinary tract infections at very low risk for adverse events and bacteremia. Pediatrics 2010, 126, 1074–1083. [Google Scholar] [CrossRef] [PubMed]
  40. Velasco, R.; Benito, H.; Mozún, R.; Trujillo, J.E.; Merino, P.A.; Group for the Study of Febrile Infant of the RISeuP-SPERG Network. Febrile young infants with altered urinalysis at low risk for invasive bacterial infection. a Spanish Pediatric Emergency Research Network’s Study. Pediatr. Infect. Dis. J. 2015, 34, 17–21. [Google Scholar] [CrossRef]
  41. Standage, S.W.; Wong, H.R. Biomarkers for pediatric sepsis and septic shock. Expert Rev. Anti-Infect. Ther. 2011, 9, 71–79. [Google Scholar] [CrossRef] [Green Version]
  42. Scott, H.F.; Donoghue, A.J.; Gaieski, D.F.; Marchese, R.F.; Mistry, R.D. The utility of early lactate testing in undifferentiated pediatric systemic inflammatory response syndrome. Acad. Emerg. Med. 2012, 19, 1276–1280. [Google Scholar] [CrossRef]
Jcm 10 01875 g001 550
Figure 1. Receiver-operating characteristic (ROC) curves of the IBI score and the developed machine learning models on the validation set; SVM = Support vector machine; XGBoost = Extreme gradient boosting.
Figure 1. Receiver-operating characteristic (ROC) curves of the IBI score and the developed machine learning models on the validation set; SVM = Support vector machine; XGBoost = Extreme gradient boosting.
Jcm 10 01875 g001
Table
Table 1. Invasive bacterial infection (IBI) score.
Table 1. Invasive bacterial infection (IBI) score.
PredictorPoints a
Age < 21 days old1
Highest temperature in the ED 38.0–38.4 °C2
Highest temperature in the ED ≥ 38.4 °C4
Abnormal urinalysis result b3
ANC ≥ 5185 cells per µL2
a Total possible scores ranged from 0 to 10. b Urine dipstick with positive leukocyte esterase or positive nitrites or urine microscopy with >5 WBCs per high-power field or >5 WBCs per mm3 on enhanced urinalysis.
Table
Table 2. Distribution and demographics of patients in three examined hospitals.
Table 2. Distribution and demographics of patients in three examined hospitals.
Northern Hospital
Mean (SD)/n(%)		Middle West Hospital
Mean (SD)/n(%)		Southern Hospital
Mean (SD)/n(%)		p-Value
Total number of patients26531681390
Age, days-old32 (18.0)31 (17.5)32 (18.1)0.504
Male1552 (58.5)91 (54.2)819 (58.9)0.497
IBI82 (3.1)3 (1.8)41 (2.9)0.625
Bacteremia76 (2.9)2 (1.2)39 (2.8)0.439
Bacterial Meningitis14 (0.5)1 (0.6)3 (0.2)0.333
Table
Table 3. Demographic and clinical characteristic comparison between patients with and without invasive bacterial infections.
Table 3. Demographic and clinical characteristic comparison between patients with and without invasive bacterial infections.
With IBI (n = 126)Without IBI (n = 4085)p-Value
Age, d, median (IQR)31 (20–43)36 (23–50)0.001
Male sex, n (%)78 (61.9)2384 (58.4)0.463
Vital signs
Triage temperature, median (IQR)38.4 (37.9–38.9)37.7 (37.1–38.3)<0.001
Highest ED temperature, median (IQR)38.6 (38.0–39.1)37.8 (37.2–38.4)<0.001
Triage HR, median (IQR)177 (161–189)159 (143–175)<0.001
Laboratory test
WBC, median (IQR)10.9 (6.4–14.1)11.2 (8.2–13.4)0.462
Hb, median (IQR)11.6 (9.8–12.8)12.3 (10.2–14.2)<0.001
Platelet, median (IQR)365 (285–441)389 (302–458)0.027
Neutrophil, median (IQR)55.7 (43.7–69.1)37.1 (24–49)<0.001
Band, mean0 (0–1.5)0 (0)<0.001
Eosinophil, median (IQR)0.8 (0–2.0)2 (1–4)<0.001
Lymphocyte, median (IQR)33.9 (22.8–45.1)47.5 (35.2–60.0)<0.001
ANC, median (IQR)6397 (2828–8795)4368 (2112–5546)<0.001
CRP, median (IQR)35.2 (2.4–47.6)0.8 (0–5.7)<0.001
Abnormal urine test, n (%)61 (48.6)740 (18.2)<0.001
IBI score, median (IQR)4 (2–6)2 (0–4)<0.001
IBI ≥ 2, n (%)110 (87.3)2345 (57.4)<0.001
Table
Table 4. Adopted features in the developed machine learning (ML) models using the stepwise feature selection method.
Table 4. Adopted features in the developed machine learning (ML) models using the stepwise feature selection method.
Logistic RegressionSVMXGBoost
1stNeutrophilCRPEosinophil
2ndCRPHeart RateBand
3rdLymphocyteNeutrophilWBC
4thBasophilBasophilCRP
5thBandBandHeart Rate
6thPlatelet ANC
7thAge Monocyte
8thTemperature
Table
Table 5. ANOVA analysis of outcome prediction using the IBI score and developed ML models.
Table 5. ANOVA analysis of outcome prediction using the IBI score and developed ML models.
Outcome, Mean (SD)IBI ScoreLRSVMXGBoostp-Value
AUROC0.70 (0.03) *0.85 (0.04)0.84 (0.03)0.84 (0.03)<0.001
IBI score ≥ 2
Sensitivity0.85 (0.06)0.90 (0.07)0.91 (0.07)0.90 (0.08)0.219
Specificity0.43(0.01) *0.59 (0.02) **0.60 (0.03) **0.57 (0.02)<0.001
* significantly lower than the outcomes of other models in the post-hoc test. ** significantly higher than the outcomes of other models in the post-hoc test.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Chiu, I.-M.; Cheng, C.-Y.; Zeng, W.-H.; Huang, Y.-H.; Lin, C.-H.R. Using Machine Learning to Predict Invasive Bacterial Infections in Young Febrile Infants Visiting the Emergency Department. J. Clin. Med. 2021, 10, 1875. https://doi.org/10.3390/jcm10091875

AMA Style

Chiu I-M, Cheng C-Y, Zeng W-H, Huang Y-H, Lin C-HR. Using Machine Learning to Predict Invasive Bacterial Infections in Young Febrile Infants Visiting the Emergency Department. Journal of Clinical Medicine. 2021; 10(9):1875. https://doi.org/10.3390/jcm10091875

Chicago/Turabian Style

Chiu, I-Min, Chi-Yung Cheng, Wun-Huei Zeng, Ying-Hsien Huang, and Chun-Hung Richard Lin. 2021. "Using Machine Learning to Predict Invasive Bacterial Infections in Young Febrile Infants Visiting the Emergency Department" Journal of Clinical Medicine 10, no. 9: 1875. https://doi.org/10.3390/jcm10091875

APA Style

Chiu, I. -M., Cheng, C. -Y., Zeng, W. -H., Huang, Y. -H., & Lin, C. -H. R. (2021). Using Machine Learning to Predict Invasive Bacterial Infections in Young Febrile Infants Visiting the Emergency Department. Journal of Clinical Medicine, 10(9), 1875. https://doi.org/10.3390/jcm10091875

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop