Prediction of Suicide-Related Events by Analyzing Electronic Medical Records from PTSD Patients with Bipolar Disorder

Abstract

Around 800,000 people worldwide die from suicide every year and it’s the 10th leading cause of death in the US. It is of great value to build a mathematic model that can accurately predict suicide especially in high-risk populations. Several different ML-based models were trained and evaluated using features obtained from electronic medical records (EMRs). The contribution of each feature was calculated to determine how it impacted the model predictions. The best-performing model was selected for analysis and decomposition. Random forest showed the best performance with true positive rates (TPR) and positive predictive values (PPV) of greater than 80%. The use of Sertraline, Fentanyl, Aripiprazole, Lamotrigine, and Tramadol were strong indicators for no SREs within one year. The use of Haloperidol, Trazodone and Citalopram, a diagnosis of autistic disorder, schizophrenic disorder, or substance use disorder at the time of a diagnosis of both PTSD and bipolar disorder, predicted the onset of SREs within one year. Additional features with potential protective or hazardous effects for SREs were identified by the model. We constructed an ML-based model that was successful in identifying patients in a subpopulation at high-risk for SREs within a year of diagnosis of both PTSD and bipolar disorder. The model also provides feature decompositions to guide mechanism studies. The validation of this model with additional EMR datasets will be of great value in resource allocation and clinical decision making.

1. Introduction

Approximately 800,000 people worldwide die from suicide every year [1]. Suicide is the 10th leading cause of death in the United States, with 48,000 deaths occurring in 2018 [2]. Because the rate of suicide among all deaths has continued to increase since 1999 [2,3,4,5,6,7,8], reversing the suicide rate has been prioritized by the World Health Organization [1].

Multiple studies have identified factors related to suicide. These factors include age, gender, and alcohol abuse [9,10]. However, it is hard to quantify the influence of these factors using traditional statistical methods [11,12,13]. With the rise of machine-learning (ML) algorithms, several successful studies have predicted suicide risk based on different ML-based methods with good accuracy [14,15,16,17].

However, few studies have focused on risk factors among high-risk populations that may require more intensive interventions to prevent suicides. Some research suggest that, among all mental disorders, bipolar disorder contributes most to the risk for suicide [18]. This risk can be higher when comorbidity exists with other psychiatric disorders [19]. Patients with bipolar disorder alternate between manic and depressive episodes [20], leading to considerable impairment of life quality [21,22]. Approximately 1% of the worldwide population suffer from bipolar disorder [23,24].

Although considered a controversial suicide risk factor, post-traumatic stress disorder (PTSD) is one of the most common comorbidities of bipolar disorder [25,26,27], with a lifetime comorbidity rate of 16%–39% [28]. PTSD is a trauma- and stressor-related disorder with the major post-trauma symptoms being fear-based re-experiencing, anhedonia, dysphoric mood stages, or dissociative symptoms [29]. The lifetime prevalence of PTSD is 7.8% [30]. Among patients with bipolar disorder, the time spent with illness as well as SREs (ideation, attempts, and deaths) are significantly higher in patients also diagnosed with PTSD [27,31,32].

Suicide prevention strategies have been proven to substantially decrease the suicide rate [33,34,35,36]. The importance of recognizing depression and suicidal tendencies has been emphasized in several reviews [33,35,37,38]. However, comprehensive suicide prevention programs consume time, labor, and resources that limit the application of care to all patients with bipolar disorder and PTSD. A time-, labor-, and resource-saving quantitative measurement for SREs in this high-risk population is needed to guide clinical decision-making and to help distribute resources to the patients who can most benefit from them.

An ML-based random forest model was constructed to identify and quantify risk factors that induce suicide among patients diagnosed with both bipolar disorder and PTSD. Using factors extracted from electronic medical records (EMRs) of patients with both diagnosis of bipolar disorder and PTSD, patients with a higher risk of an SRE within a year were identified. The model focused primarily on predictors like baseline disease conditions or pharmacy records which can be easily obtained and are less likely to be biased by subjective factors. Extra emphasis was placed on lithium usage, as previous studies have demonstrated that lithium was effective in preventing death by suicide in patients with mental disorders [34,39,40].

2. Materials and Methods

2.1. Data Sources

Data was collected from the EMRs of patients seen at UPMC (University of Pittsburgh Medical Center) facilities between 2004 and 2019. The cohort of patients with PTSD and bipolar disorders were identified based on the International Classification of Diseases (ICD) 9 and 10 codes for these disorders. Records for these patients, including demographics, medication usage, encounters and diagnosis of comorbid diseases, were extracted from the EMR systems as an IRB (Institutional Review Board)-approved HIPAA (Health Insurance Portability and Accountability Act) Limited Data Set. These data included the dates for each transaction, using the University of Pittsburgh’s research data warehouse. After data extraction, a sub-population that was diagnosed with both PTSD and bipolar disorder was created.

SREs for this sub-population were identified from the EMRs. The diagnosis codes included ICD9 and ICD10 for suicidal ideation, attempt, and death based on literature reports [3,41]. The diagnosis table was searched using the keywords ‘suicide’, ‘suicidal’ and ‘intentional self-harm’. Events of undetermined intent (Y10–Y34) were not considered. The lack of well-defined codes like X60–X70 in the list might indicate a bias of coding preference in the UPMC EMR system.

The dates of first PTSD and bipolar disorder diagnoses of patients were extracted, with the later date assigned to each patient as Both Diagnosed Time (BDT). SRE predictions were made at this time point. Patients’ ages when they were diagnosed with both disorders were determined. Patients were excluded that had SREs before BDT since the SREs may have not be casually linked to the diagnosis of both PTSD and bipolar disorder. The time interval between BDT and an SRE was calculated for high-risk patients. SREs within one year after BDT were marked as 1 (event identified). Patients with follow-up times beyond one year that did not have an SRE or had it more than a year after BDT, were marked as 0 (event not identified). Comorbid medical disorders were also documented and categorized into 12 disease categories that used only ICD9 codes [17]. These ICD9 codes were mapped to ICD10 codes using the service provided at http://www.icd10codesearch.com/. Patients having dipolar disorder and PSTD diagnosis codes within a year prior to BDT were considered as having these comorbid diseases.

Three major classes of medications taken into consideration as predictors were mood stabilizers, antipsychotics, and antidepressives, and were extracted from DrugBank [42] (https://www.drugbank.ca/). Patients were marked as taking these medications if they had been prescribed within one year prior to their BDT to find predictor information for SREs at the point of BDT. Included in this study were 75 extracted medications from the three classes that were matched with medications recorded in the EMR system. Medications included in this study are: Almotriptan, Amitriptyline, Amoxapine, Amphetamine, Aripiprazole, Asenapine, Brexpiprazole, Bupropion, Buspirone, Carbamazepine, Cariprazine, Chlorpheniramine, Chlorpromazine, Citalopram, Clomipramine, Clozapine, Desipramine, Desvenlafaxine, Dexmethylphenidate, Dextromethorphan, Dihydroergotamine, Doxepin, Duloxetine, Eletriptan, Escitalopram, Fentanyl, Flibanserin, Fluoxetine, Fluphenazine, Fluvoxamine, Frovatriptan, Haloperidol, Iloperidone, Imipramine, Lamotrigine, Levomilnacipran, Lithium, Loxapine, Lurasidone, Maprotiline, Meperidine, Methadone, Milnacipran, Mirtazapine, Naratriptan, Nefazodone, Nortriptyline, Olanzapine, Paliperidone, Paroxetine, Perphenazine, Phenelzine, Pimozide, Promethazine, Protriptyline, Quetiapine, Rasagiline, Risperidone, Rizatriptan, Ropinirole, Rotigotine, Selegiline, Sertraline, Sumatriptan, Tapentadol, Thiothixene, Tramadol, Tranylcypromine, Trazodone, Trifluoperazine, Venlafaxine, Vilazodone, Vortioxetine, Ziprasidone, and Zolmitriptan.

2.2. Software and Model Setup

The analysis algorithm was written in the Python programming language in a Jupyter notebook [43]. The ML-based models and calibration curves were developed by using scikit-learn 0.20.0 [44]. The key Python libraries used in this analysis were SciPy [45], NumPy [46] and Pandas [47].

Several different ML-based classifiers were tested, including logistic regression [48], random forest [49], decision tree [50], K-nearest neighbors [51], Naïve Bayes [52] and support vector machine [53]. All models were set at a random state of 42 to ensure reproducibility while the other hyper-parameters were left at default settings. The random state seeded the random number generator used in the models. For the final random forest model, we set estimators to 100 and the maximum number of features to the square root of the number of features.

ML-based models frequently encounter datasets that are heavily imbalanced—the number of samples in the different classes are distributed unevenly—which affects their learning phases and subsequent predictions. An over-sampling procedure based on the Synthetic Minority Oversampling Technique (SMOTE) [54] was performed prior to conducting the analysis. The over-sampling procedure creates new samples by connecting inliers and outliers from the original dataset [54]. The resampled dataset was split into training and test datasets randomly in a 4:1 ratio. Only the training set was oversampled with SMOTE so that the test set contained the original subjects in the dataset.

Many socioeconomic factors have been reported to play important roles in suicide prediction [55]. However, data from only the EMR were used as the predictors, variables, or features for modeling: (a) demographic data, including gender and age at BDT; (b) number of emergency department (ED) visits and diagnoses within one year prior to the BDT; (c) medication usage within one year prior to the BDT, including medication orders, dispenses, and fills. Medication usage data was coded by whether patients had taken these medications within one year prior to their BDT.

Predictor or variable importance was calculated to assess key factors in SRE prediction. In the random forest algorithm, predictor importance was quantified by evaluating the decrease in “node impurity” at each split across all decision trees in the forest [56]. In the simplest case, node impurity can be considered as the difference in measurement from controls at a node. The random forest module uses these measures to estimate variances in nodes across trees. The nodes with maximized response variances are those that have greater contributions to the differences in categories of cases and have a greater impact on the model’s ability to predict outcomes.

Since patients with SREs are a minor class in our dataset, model performance was based on true positive rate (TPR), positive predictive value (PPV), and negative predictive value (NPV) calculated as follows (Equation (1)):

$$\begin{gathered} \mathrm{TPR}=\frac{\mathrm{True}\mathrm{Positives}}{\left(\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Negatives}\right)} \\ \mathrm{PPV}=\frac{\mathrm{True}\mathrm{Positives}}{\left(\mathrm{True}\mathrm{Positives}+\mathrm{False}\mathrm{Positives}\right)} \\ \mathrm{NPV}=\frac{\mathrm{True}\mathrm{Negatives}}{\left(\mathrm{True}\mathrm{Negatives}+\mathrm{False}\mathrm{Negatives}\right)} \end{gathered}$$

(1)

Random forest results were interpreted using the python package TreeInterpreter 0.2.2 (https://github.com/andosa/treeinterpreter), which allowed the (a) decomposition of each prediction into feature contribution components in the training set mean and (b) identification of those features that affect the difference and their contribution. In the model, all features will make contributions to the predication about an instance whether positive or negative. If the value of a feature’s contribution was positive (SRE), the prediction value was scored as 1. If the feature’s contribution was negative (no SRE), the prediction value was scored as 0.

3. Results

3.1. Model Construction and Performance

A total of 6042 patients with PTSD and bipolar disorder were identified from the EMR system by ICD9 and ICD10 codes (Appendix A). Of this population, 4138 of them had no records of SRE before BDT. Among these 4138 patients, 205 were identified as having SREs within one year after BDT, while 3933 of them did not have SREs in the same time period. Patients with follow up time less than one year and no reported SRE (970) were excluded from this study. The filtered 2963 subjects were oversampled into a balanced dataset by SMOTE as described above. After data resample and split, the training dataset contained 4726 subjects with 2363 subjects marked as 1 and 2363 subjects marked as 0. The inclusion process is described in Figure 1 and the baseline patient characteristics are shown in

Table 1. Baseline Patient Characteristics.

Characteristic	Suicide (Percentage)	Not Suicide (Percentage)	p Value *
	N = 205	N = 2963
Gender
Male	66 (32.2)	688 (23.2)	0.005
Female	139 (67.8)	2275 (76.8)
Lithium Use
Yes	16 (7.8)	221 (7.5)	0.964
Not	189 (92.2)	2742 (92.5)
ED Visits
10 ≤ X	15 (7.3)	93 (3.1)	0.003
5 ≤ X < 10	28 (13.7)	260 (8.8)	0.026
4	9 (4.4)	133 (4.5)	0.999
3	19 (9.3)	213 (7.2)	0.334
2	20 (9.8)	357 (12.0)	0.385
1	43 (21.0)	596 (20.1)	0.836
0	71 (34.6)	1311 (44.2)	0.009
Age
Mean (SD)	35.06 (12.92)	38.45 (13.29)	<0.001

* p Values were generated with chi-square test.

. Significant differences among patients with and without SREs because of gender, age, and ED visits may be contributing variables in this study.

ML-based models were trained and evaluated with the data generated by the resample procedures. Performances of all the models are shown as the means from a 5-fold stratified cross-validation process (

Table 2. Model performance of all models *.

	K-Nearest Neighbors	Naïve Bayes	Decision Tree	Support Vector Machine	Logistic Regression	Random Forest
TP	182	200.6	146	114.2	111.8	171
FP	888	2732.6	103.8	1238.8	1074.6	17
TN	2075	230.4	2859.2	1724.2	1888.4	2946
FN	23	4.4	59	90.8	93.2	34
TPR	0.888	0.979	0.712	0.557	0.545	0.834
PPV	0.17	0.068	0.585	0.084	0.094	0.91
NPV	0.989	0.981	0.98	0.95	0.953	0.989

TP: True positive, TN: True negative, FP: False positive, FN: False negative, TPR: True positive rate or Sensitivity, PPV: Positive predictive value, NPV: Negative predictive value. * Values in the table are means from 5-fold stratified cross validation.

). TPR and PPV were prioritized since the model should be able to identify the high-risk population within the precision constraints relevant to the data. Random forest was superior at retrieving positive cases with less false positives with an exceptional high PPV (Table 2). Random forest achieved an accuracy of 92.4%, an area under curve (AUC) of 95.6%, an F1 score of 0.879, and an area under receiver operating characteristic (ROC) curve of 0.820. The random forest model was chosen as the predictive model in the following analysis.

3.2. Model Decomposition and Feature Importance Analysis

A decomposition analysis on the decision trees generated by the random forest algorithm was conducted to better understand the contributions of each factor on SRE predictions. All features in the model were examined individually to determine if the feature provided positive contributions. Such an approach allowed a minimization of the data volume needed to make an accurate prediction and to reduce computation expenses.

Ninety-two features were used in the model including disease categories 1–12, the seventy-five medications mentioned above, age, gender and ED visits. Among them, only age and ED visits were continuous variables, and all other features were categorical. In order to find the features that are necessary for the model and to minimize the data requirement, feature importance was calculated using the method implemented in the package. Feature importance was calculated as the decrease in node impurity weighted by the probability of reaching that node. The node probability was calculated by the number of samples that reach the node, divided by the total number of samples. The higher the value the more important the feature [57]

Multiple random forest tests, which included the top most important features, were performed to retrain the model and test its performance. The performance of model improved as the number of features with high importance increased (Figure 2). The performance curves reached a plateau at approximately 30 features, then maintained a performance similar to the original model we trained using all 90 features. As a result, the 30 most important features (

Table 3. Feature importance in random forest model.

Feature	Feature Importance
Age at both diagnosed	0.141
Disease category 5 in last year	0.081
Disease category 3 in last year	0.07
Disease category 1 in last year	0.061
Trazodone	0.055
Fentanyl	0.047
Disease category 11 in last year	0.039
Emergency department visits in last year	0.038
Lamotrigine	0.036
Sertraline	0.031
Disease category 6 in last year	0.031
Disease category 2 in last year	0.023
Quetiapine	0.023
Citalopram	0.022
Bupropion	0.021
Tramadol	0.021
Fluoxetine	0.018
Aripiprazole	0.017
Haloperidol	0.016
Venlafaxine	0.016
Lithium	0.015
Duloxetine	0.014
Buspirone	0.012
GENDER	0.012
Risperidone	0.011
Disease category 4 in last year	0.011
Mirtazapine	0.01
Ziprasidone	0.009
Olanzapine	0.009
Promethazine	0.008
Escitalopram	0.008
Amphetamine	0.006
Sumatriptan	0.006
Disease category 9 in last year	0.006
Amitriptyline	0.005
Chlorpromazine	0.005
Carbamazepine	0.005
Disease category 10 in last year	0.004
Paroxetine	0.004
Methadone	0.004
Disease category 12 in last year	0.003
Dextromethorphan	0.003
Lurasidone	0.003
Meperidine	0.003
Rizatriptan	0.003
Asenapine	0.002
Doxepin	0.002
Disease category 8 in last year	0.002
Vilazodone	0.001
Perphenazine	0.001
Nortriptyline	0.001
Thiothixene	0.001
Clomipramine	0.001
Ropinirole	0.001
Paliperidone	0.001
Eletriptan	0.001
Naratriptan	0
Zolmitriptan	0
Desvenlafaxine	0
Selegiline	0
Levomilnacipran	0
Milnacipran	0
Vortioxetine	0
Dihydroergotamine	0
Imipramine	0
Desipramine	0
Tapentadol	0
Clozapine	0
Fluphenazine	0
Disease category 7 in last year	0
Trifluoperazine	0
Almotriptan	0
Rasagiline	0
Brexpiprazole	0
Chlorpheniramine	0
Cariprazine	0
Fluvoxamine	0
Loxapine	0
Rotigotine	0
Dexmethylphenidate	0
Protriptyline	0
Tranylcypromine	0
Flibanserin	0
Amoxapine	0
Frovatriptan	0
Iloperidone	0
Maprotiline	0
Phenelzine	0
Pimozide	0
Nefazodone	0

) were used to train a simplified random forest model.

A simplified random forest model was built using the top 30 features. The ROC and the Precision-Recall graphs of the new model were plotted (Figure 3). The random forest model outperforms the no skill (random) model in both graphs (Figure 3). The simplified model yielded an accuracy of 98.3%, an AUC of 95.9% (similar to the original model performance), an F1 score of 0.868, and an ROC of 0.811. The performance parameters for the retrained model achieved a high TPR and PPV with the 30 selected features, again similar to the original model performance (Table 2;

Table 4. Performance of model retrained on selected features.

	TP	FP	TN	FN	TPR	PPV	NPV
Retrained model	171	14	2949	34	0.834	0.924	0.988

TP: True positive, TN: True negative, FP: False positive, FN: False negative, TPR: True positive rate or Sensitivity, PPV: Positive predictive value, NPV: Negative predictive value.

). These results indicate that the random forest model is sensitive to patients who had SREs and can predict SREs correctly. Every feature that impacted the final prediction using random forest was processed through the decomposition algorithms from treeinterpreter.

The random forest model was used to predict how each feature could impact the possibility of having a SRE within one year after being BDT on all 3168 patients in the dataset. Of the 3168 patients, the model correctly predicted SREs from 3120 of them. Contribution values (negative and positive) of the features to correctly predicted presence of SREs within 1 year were calculated.

The distributions for two continuous datasets, age and ED visits were investigated (Figure 4). The age and ED distributions between positive and negative scores were significantly different (p < 0.001) (Figure 4). Younger ages and more ED visits are associated with a higher risk of having SREs.

The distribution of the 28 categorical features provided an insight into how the individual features impacted the SREs of individual cases (Figure 5). Generally speaking, value 1 tended to make a positive contribution compared to 0 across all features. Specifically, features such as Fentanyl, Aripiprazole, Disease category 11, Disease category 2 and Disease Category 6 showed obvious associations between contributing groups and feature values. The value distributions of features are different in positive and negative contributing groups (Figure 4) and these shifts can provide information about the impact a feature may have on SREs. The difference in value distributions of features were examined using a chi-square test (

Table 5. Feature value distribution significance in positive and negative contributing groups.

Features	T	P	Percentage of 1 in Whole Population	Percentage of 1 with Positive Contribution	Percentage of 1 with Negative Contribution	FDR Adjusted q Value	Direction of Effect
Disease category 2 in last year	2065.444	<0.001	0.064	0.919	0.017	<0.001	SREs
Disease category 11 in last year	2239.822	<0.001	0.098	0.996	0.027	<0.001	SREs
Disease category 6 in last year	1750.073	<0.001	0.066	0.846	0.021	<0.001	SREs
Disease category 1 in last year	2193.804	<0.001	0.232	1	0.068	<0.001	SREs
Trazodone	1248.659	<0.001	0.17	0.996	0.101	<0.001	SREs
Sertraline	1205.388	<0.001	0.116	0	0.459	<0.001	No SREs
GENDER	681.776	<0.001	0.237	0.062	0.463	<0.001	No SREs
Haloperidol	489.011	<0.001	0.046	0.32	0.021	<0.001	SREs
Fentanyl	486.882	<0.001	0.188	0.002	0.317	<0.001	No SREs
Aripiprazole	428.686	<0.001	0.089	0.003	0.219	<0.001	No SREs
Lamotrigine	424.696	<0.001	0.146	0.001	0.264	<0.001	No SREs
Disease category 4 in last year	422.183	<0.001	0.034	0.261	0.014	<0.001	SREs
Ziprasidone	348.145	<0.001	0.037	0.202	0.013	<0.001	SREs
Risperidone	326.949	<0.001	0.063	0.378	0.043	<0.001	SREs
Mirtazapine	166.917	<0.001	0.063	0.186	0.037	<0.001	SREs
Quetiapine	127.63	<0.001	0.135	0.32	0.109	<0.001	SREs
Venlafaxine	119.404	<0.001	0.082	0.214	0.06	<0.001	SREs
Buspirone	111.025	<0.001	0.079	0.024	0.126	<0.001	No SREs
Disease category 5 in last year	108.425	<0.001	0.46	0.989	0.443	<0.001	SREs
Duloxetine	104.213	<0.001	0.069	0.175	0.048	<0.001	SREs
Tramadol	76.116	<0.001	0.131	0.038	0.162	<0.001	No SREs
Citalopram	63.62	<0.001	0.124	0.233	0.103	<0.001	SREs
Bupropion	18.005	<0.001	0.109	0.066	0.122	<0.001	No SREs
Fluoxetine	14.305	<0.001	0.092	0.065	0.107	<0.001	No SREs
Disease category 3 in last year	14.02	<0.001	0.749	0.907	0.744	<0.001	SREs
Promethazine	7.757	0.005	0.052	0.038	0.061	0.006	No SREs
Lithium	5.165	0.023	0.076	0.058	0.083	0.024	No SREs
Olanzapine	1.026	0.311	0.054	0.048	0.057	0.311	N/A

) and as a percentage in positive and negative contributing groups. If a feature has no or little association with the final prediction, the percentages of patients taken medication or have the comorbid disease in positive and negative contributing groups should be similar to the percentage of 1 in the whole population. If the percentage of patients taken medication or have the comorbid disease in positive or negative contributing group significantly differs from that of the whole population and each other, it suggests a possible mechanistic association between this feature and the potential risk for an SRE. For example, 11.6% of the participants have taken Sertraline. They account for 0% of the positive contributing groups and 45.9% of negative contributing groups. It can be concluded that taking Sertraline is predictive for no SREs within one year. High-importance features with an obvious separation pattern among the population groups have also been identified (Table 3). This indicates that the values of these features can greatly impact the final SRE predictions and may inform future mechanism studies.

All features except Olanzapine showed a significant difference between their distributions in positive and negative contributing groups. This is the result we are expecting because all the features in Figure 5 have been selected through the drop column feature importance test and were identified as important for the model to make the prediction. If the value of a certain feature does not provide significant differences in the percentages among the groups, it is likely that it has no benefit in for predicting SREs and will be dropped in the previous step. The results shown in Table 5 provided additional support to our feature selection process above.

4. Discussion and Conclusions

Prior studies have found that the ML-based methods perform better at identifying suicide risks in large populations of patients than traditional methods. The accuracies of these studies are reported to be between 0.76–0.79 with AUCs generally between 0.80–0.90 [15,17,58]. The objective of this study was to find an ML-based method that identifies patients at high-risk for SREs—patients diagnosed with both PTSD and bipolar disorder. This study demonstrated an accuracy of 0.92 and an AUC of 0.956 using the random forest method. The random forest model can accurately predict those patients at higher risk of SREs as evaluated with TPR and PPV tests. Sub-populations suffering from certain disorders and taking certain medications can be distinguished from a larger population as having a higher risk for SREs.

Different features have different contributions to the prediction of SREs. These features are mental disorders and drug administration history within one year. As discussed by Sanderson et al. [17], mental health diagnoses were separated into twelve disease categories based on their ICD9 codes (Appendix B). Patients suffering from comorbid diseases at BDT are more likely to have SRE within a year. These comorbid diseases include: Category 11, autistic disorder-current and disturbance of conduct; Category 3, mood disorders and adjustment disorders; Category 4, other psychotic disorders; and Category 5, acute stress reactions. Several studies have reported results that diseases in Category 11 are more likely to trigger SRE [59,60,61,62]. Numerous studies have provided evidence that mental disorders have the potential to increase the risk of SREs [63,64]. This evidence is supported by the results of the random forest model presented here.

For the unselected features, it does not mean that these features may not be useful predictors of SREs. The aim of the feature selection process to use a sufficient but minimal number of features for the model to achieve optimal prediction results. It was found that optimal results were found using 30 features and that the addition of additional features did not affect the results. The distribution of all categorical features is attached (Appendix C). The impurity-based feature importance can be misleading for high cardinality features and continuous variables (age and ED visits) [65]. For this reason, the distribution of these two variables were examined first to ensure that their association with SREs are not the result of biased algorithms.

With medication usage as the feature, some of them showed a much higher proportion in the negative contributing group compared to either the whole population or the positive contributing groups (Fentanyl, Levomilnacipran, Sertraline, Aripiprazole, Tramadol, Lamotrigine, Sertraline, and Fluoxetine). These medications are considered to reduce the risk of SREs within one year in our model. Other investigators have shown similar beneficial effects in clinical trials [66,67]. However, some studies have found that Tramadol, Aripiprazole, and Fentanyl have not been associated with risk reduction in SREs. Thus, our results may provide support for further investigations. The model identified several medications that increased the risk of SREs. Such medications have also been reported to increase the risk of SREs in other studies [68]. Caution must be taken in interpreting the effect of medications on the prediction of SREs in that the model’s results do not account for drugs that may be indicators of comorbidities, e.g., sleep problems that may alter the risk of SREs.

The results of our study made it possible for clinicians to identify patients who have a higher risk of SREs and have additional insight of how to reduce this risk by identified risk factors. Clinicians will be able to adjust the medications to replace some drugs which increase the risk of SREs with drugs with the same class with less risk or focus on relieving the symptoms which may contribute most to the suicide risk.

The ML-based random forest model provides a basis for clinicians to build similar models for different populations facing different disease risks. Our model is built with open source Python packages and trained based on EMR data. This means other researchers can test our model in other clinical samples. Also, our study can provide guidance for clinical institutions or other researchers to build their own models for other kinds of populations.

Unavoidably, there are limitations to this study. (a) the data was collected from hospitals affiliated with UPMC. External data for validation was not used and, if included, may have led to overfitting; (b) most clinicians prefer to treat diseases and disorders with particular combinations of drugs different from those used by other clinicians. This may cause bias in the results among institutions if such preferences are widely used in the hospital, despite alternative drug choices; (c) the high prediction performance of the model may due to the unique characteristics of the BDT patient subpopulation. The model may need further adjustment and optimization to apply it to other high-suicide risk populations or other disease states; (d) mis-diagnoses and biased prescriptions are two problems may cause errors in the predictions of SREs. PTSD and bipolar disorder may be mis-diagnosed as other diseases in their early stages, which may cause bias in our model, especially with younger patients. However, the ability to identify mis-diagnoses and biased prescriptions from the EMR is beyond the capability of our model; (e) though some medications, like lithium, may not be indicated for SREs, clinicians prescribe them for bipolar disorders to a greater extent due to its known anti-suicidal properties. This may be the situation in many clinical practices.

The ML-based random forest model makes it possible for clinicians to identify subpopulations of patients who have a higher risk of SREs and to have additional insights to reducing this risk by identifying individual risk factors. Medications that increase the risk of SREs can be substituted with drugs having a lower risk or that focus on relieving symptoms that may contribute most to SREs.

Using EMR information, a ML-based random forest model was constructed that predicts, with an accuracy of around 90%, if a patient will have an SRE within the following year of the diagnosis of both bipolar disorder and PSTD. The model extracts features that make contributions to the risk of SREs, which can be further utilized in mechanism studies. The model has great potential as a clinical tool that can aid clinicians in identifying high-risk individuals and to better guide patient clinical care.