Validation of the REGARDS Severe Sepsis Risk Score
by
,
Henry E. Wang
1,*
,
John P. Donnelly
2,
Sachin Yende
3,
Emily B. Levitan
4,
Nathan I. Shapiro
5,
Yuling Dai
4,
Hong Zhao
4,
Gerardo Heiss
6,
Michelle Odden
7,
Anne Newman
8 and
Monika Safford
9 1
Department of Emergency Medicine, The University of Texas Health Science Center at Houston, Houston, TX 77030, USA
2
Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI 48109, USA
3
Department of Critical Care Medicine, University of Pittsburgh, Veterans Affairs Pittsburgh Healthcare System, Pittsburgh, PA 15213, USA
4
Department of Epidemiology, University of Alabama at Birmingham, Birmingham, AL 35294, USA
5
Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Boston, MA 02215, USA
6
Department of Epidemiology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
7
Department of Health Research and Policy, Stanford University, Stanford, CA 94305, USA
8
Department of Epidemiology, University of Pittsburgh, Pittsburgh, PA 15213, USA
9
Department of Medicine, Weill-Cornell Medical Center, New York, NY 10065, USA
*
Author to whom correspondence should be addressed.
J. Clin. Med. 2018, 7(12), 536; https://doi.org/10.3390/jcm7120536
Submission received: 5 November 2018
/
Revised: 4 December 2018
/
Accepted: 7 December 2018
/
Published: 11 December 2018
(This article belongs to the Section Hematology)
Abstract
:There are no validated systems for characterizing long-term risk of severe sepsis in community-dwelling adults. We tested the ability of the REasons for Geographic and Racial Differences in Stroke-Severe Sepsis Risk Score (REGARDS-SSRS) to predict 10-year severe sepsis risk in separate cohorts of community-dwelling adults. We internally tested the REGARDS-SSRS on the REGARDS-Medicare subcohort. We then externally validated the REGARDS-SSRS using (1) the Cardiovascular Health Study (CHS) and (2) the Atherosclerosis Risk in Communities (ARIC) cohorts. Participants included community-dwelling adults: REGARDS-Medicare, age ≥65 years, n = 9522; CHS, age ≥65 years, n = 5888; ARIC, age 45–64 years, n = 11,584. The primary exposure was 10-year severe sepsis risk, predicted by the REGARDS-SSRS from participant sociodemographics, health behaviors, chronic medical conditions and select biomarkers. The primary outcome was first severe sepsis hospitalizations, defined as the concurrent presence of ICD-9 discharge diagnoses for a serious infection and organ dysfunction. Median SSRS in the cohorts were: REGARDS-Medicare 11 points (IQR 7–16), CHS 10 (IQR 6–15), ARIC 7 (IQR 5–10). Severe sepsis incidence rates were: REGARDS-Medicare 30.7 per 1000 person-years (95% CI: 29.2–32.2); CHS 11.9 (10.9–12.9); ARIC 6.8 (6.3–7.3). SSRS discrimination for first severe sepsis events were: REGARDS-Medicare C-statistic 0.704 (95% CI: 0.691–0.718), CHS 0.696 (0.675–0.716), ARIC 0.697 (0.677–0.716). The REGARDS-SRSS may potentially play a role in identifying community-dwelling adults at high severe sepsis risk.
1. Introduction
The sepsis syndrome is a major public health problem responsible for over 850,000 emergency department visits, 200,000 deaths and $16.7 billion in medical expenditures in the United States (US) each year [1,2,3,4]. Current scientific and clinical initiatives focus almost entirely on the early recognition and treatment of acute sepsis [5,6]. However, the onset of acute disease is often influenced by underlying precursors. For example, acute myocardial infarction and stroke were once viewed as acute conditions triggered by random exposures or processes that could not be anticipated or prevented. The recognition of myocardial infarction and stroke as the end result of underlying chronic medical conditions led to new strategies of cardiovascular risk prediction, management and prevention, contributing to dramatic declines in heart disease and stroke mortality [7,8,9]. Like myocardial infarction and stroke in the past, relatively little attention has been paid to the factors that may heighten an individual’s risk of acute sepsis. An understanding of the factors that influence an individual’s baseline sepsis risk could provide new opportunities for mitigating the societal burden of sepsis [10].
Fundamental steps in disease prevention include identifying vulnerable individuals and quantifying their degree of risk. In prior studies, we used the population-based REasons for Geographic and Racial Differences in Stroke (REGARDS) cohort to show that sepsis hospitalizations are predictable and associated with sociodemographic factors, health behaviors, chronic diseases, and perturbations in the immune system [11]. We used these findings to derive a risk prediction rule—the REGARDS Severe Sepsis Risk Score (REGARDS-SSRS)—quantifying the 10-year risk of severe sepsis hospitalizations [12] (Table 1 and Appendix Table A1).
To validate their accuracy, robustness and generalizability, risk prediction rules must be tested in independent cohorts [13]. In this study we first tested the ability of the REGARDS-SSRS to identify severe sepsis events in a subcohort of REGARDS participants enrolled in Medicare. We then sought to externally validate the REGARDS-SSRS in two large population-based cohorts: The Cardiovascular Health Study (CHS) and the Atherosclerotic Risk in Communities (ARIC) Study.
2. Experimental Section
2.1. Study Design and Overview
We used a population-based cohort design. This study was approved by the Institutional Review Boards of the University of Alabama at Birmingham and the University of Texas Health Science Center at Houston.
While we originally derived the REGARDS-SSRS using severe sepsis events identified by adjudicated review of medical records, there were no other existing cohorts with similarly identified severe sepsis events available for validation of the prediction rule. However, many population-based cohorts have discharge diagnoses available for all hospitalizations, providing an alternate strategy for identifying severe sepsis events. We therefore developed strategies to validate the REGARDS-SSRS in discharge-diagnosis based data sets.
As an important initial step, we first internally applied the SSRS to the REGARDS-Medicare cohort; this subcohort linked Medicare-enrolled REGARDS participants with their respective Medicare claims. This initial analysis allowed us to evaluate the ability of the SRSS to predict sepsis cases in the REGARDS cohort using hospital discharge diagnoses. We then sought to externally validate the SSRS using the CHS and ARIC cohorts, two of the most widely recognized longitudinal population-based cohorts in the US.
2.2. Study Setting and Participants—The REGARDS-Medicare, CHS and ARIC Cohorts
REGARDS is one of the nation’s largest ongoing population-based cohorts [14]. Consisting of 30,239 community-dwelling adults aged ≥45 years from across the United States, the REGARDS cohort was designed to evaluate the predictors of racial and geographic differences in stroke mortality. Recruitment of REGARDS participants occurred between January 2003 and October 2007 with ascertainment of baseline participant information including medical history, functional status, health behaviors, physical characteristics, physiologic measures, current medications, diet, family history of diseases, psychosocial factors and prior residences. The study also obtained comprehensive biologic specimens at the time of enrollment. Among REGARDS participants, 42% are African American, 45% are male, and 69% are over 60 years old.
The REGARDS-Medicare subcohort consists of 9,522 REGARDS participants who were enrolled in Medicare during 2003–2012 [15,16,17,18]. For this study, we limited the analysis to REGARDS participants age ≥65 years enrolled in Medicare Parts A (inpatient services) and B (outpatient services) throughout the entire REGARDS follow-up period 2003–2012. We excluded those with Medicare Part C coverage, who transitioned into or out of Medicare coverage during the study period. The parent REGARDS study identified hospitalizations through adjudicated chart review of medical records for each participant-reported health event. In contrast, the REGARDS-Medicare subcohort identified hospitalization events using discharge diagnoses for each claim. The REGARDS-Medicare linkage strategy has been previously described [18,19,20]. The characteristics of the REGARDS-Medicare subcohort are similar to participants in the Medicare national sample [18]. While the REGARDS-Medicare subcohort contains individuals used in the original SSRS derivation, the method of severe sepsis event identification is different, entailing use of discharge diagnoses. REGARDS included individuals ≥45 years old while Medicare was limited to persons ≥65 years old; we accepted this limitation as there was no large data source characterizing insurance claims for all REGARDS participants.
The Cardiovascular Health Study (CHS) is a population-based longitudinal study of coronary heart disease and stroke in adults aged ≥65 years [21]. Since 1987 the cohort CHS recruited a total of 5888 participants from Forsyth County (North Carolina), Sacramento County (California), Washington County (Maryland) and Pittsburgh (Pennsylvania). The study conducted extensive physical and laboratory evaluations at baseline. Follow-up of participants entailed semi-annual clinic examination (enrollment through 1999) and phone contact (enrollment through present). Identification of hospitalization events in CHS occurred through semi-annual interviews, additional self-reports by participants, and periodic surveillance of Medicare records. The study obtained discharge summaries and diagnoses for all hospitalizations.
The Atherosclerosis Risk in Communities Study (ARIC) is a prospective epidemiologic study conducted in Forsyth County (North Carolina), Jackson (Mississippi), Minneapolis (Minnesota), and Washington County (Maryland) [22]. In 1987 ARIC recruited a cohort a total of 15,792 individuals aged 45–64 years, conducting an extensive examination, and determination of medical, social, and demographic data. Study personnel abstracted hospitalization records (including diagnosis and procedure codes) for events identified through annual telephone contact with study participants or surveillance in study hospitals.
We chose CHS and ARIC cohorts because of their longitudinal design, completeness of follow-up, the availability of comprehensive discharge diagnoses for all hospitalizations, and the depth of available baseline variables and biomarkers needed to calculate the SSRS for each participant.
2.3. Exposure—Calculation of REGARDS-Severe Sepsis Risk Score
The REGARDS-Severe Sepsis Risk Score (SSRS) characterizes long-term severe sepsis risk using combinations of individual sociodemographics, health behaviors, chronic medical conditions, and biomarkers. (Table 1) Derivation and internal validation of the SSRS has been previously described [12]. Briefly, for each participant reported serious infection hospitalizations, two trained abstractors independently reviewed the corresponding medical records for the first 28-h of hospitalization to confirm the presence of a serious infection, sepsis and organ dysfunction. Using Cox regression, we identified clinical characteristics associated with severe sepsis events, converting the regression model into a 0–45 point severe sepsis risk score (SSRS).
While our original derivation effort included a prediction model for sepsis events (Appendix Table A1), for this study we focused on the model identifying severe sepsis events because of its higher discrimination and clearer link with the contemporary definition of sepsis as infection plus organ dysfunction [23]. We also focused on the version of the SSRS incorporating biomarkers (hsCRP, Cystatin-C, eGFR, ACR) because of its higher discrimination.
We calculated REGARDS-SSRS scores for each participant in the REGARDS-Medicare, CHS and ARIC cohorts using baseline comorbidity data available for each participant. For CHS and ARIC, we used previously published serum creatinine adjustments in order to calibrate these values for use with the CKD-EPI equation in the calculation of estimated glomerular filtration rates (eGFR) [24]. Baseline serum creatinine values for REGARDS participants were previously IDMS standardized [25].
For the ARIC cohort, ACR and Cystatin-C were not available until participant visit (examination) No. 4 (1996–1998). Therefore, for ARIC, we used data collected at visit 4 to calculate risk score points. The visit 4 examination date was used as the baseline for all ARIC participants in calculations of time to sepsis, with individuals excluded if the visit 4 examination was not completed for any reason or if sepsis occurred before this date.
2.4. Outcomes—Identification of Severe Sepsis Events
The primary outcome was first severe sepsis hospitalization. Following the method of Angus, et al., we defined severe sepsis as hospitalizations with the concurrent presence of discharge diagnoses for a serious infection and organ dysfunction [2] (Table A2 and Appendix Table A3). We identified severe sepsis events during the following follow-up periods: REGARDS-Medicare 2003–2012, CHS 1989–2002 and ARIC 1996–2008. Although ARIC enrolled subjects starting in 1987, we limited the analysis to sepsis events identified during 1996–2008 because certain exposure variables (ACR, Cystatin-C and history of atrial fibrillation) were not available until participant examination No. 4 (1996–1998).
2.5. Data Analysis
We compared participant characteristics between the REGARDS-Medicare, CHS and ARIC cohorts. We assessed the distribution of the SSRS in each cohort using quantile plots and histograms.
To test the ability of the SSRS to detect discharge-diagnosis-identified severe sepsis events, we first internally applied the SSRS to the REGARDS-Medicare subcohort. We used the original SSRS model and point system—we did not derive a new risk score for the analysis. Using Cox proportional hazards models, we determined severe sepsis incidence and relative hazards across risk groups (very low, low, medium, high, and very high, as previously reported in the initial SSRS derivation effort). To assess calibration, we conducted a validation in the manner described by Royston [26]. We estimated predicted and observed survival functions for SSRS risk groups from Cox models, with the smoothed baseline function estimated using fractional polynomials. We assessed goodness of fit using the Groennesby and Borgan score test (Stata ‘stcoxgof’ command) and reported the observed and predicted number of events over predicted risk deciles [27]. We determined discrimination using Harrell’s C-statistic [28]. Lastly, we reported the percent positive, the true positive rate, and the true negative rate over the full ten years of follow-up.
We then externally validated application of the SSRS to the ARIC and CHS data sets. We followed a similar strategy as with the REGARDS-Medicare subcohort, examining incidence and relative hazards across risk groups, observed and predicted survival functions and events, goodness-of-fit and discrimination. For the ARIC cohort we excluded participants who died or experienced severe sepsis events prior to 1996.
In the primary analysis, we explicitly coded missing exposure variables as “normal”. We conducted all analyses using Stata 13.1 (Stata, Inc., College Station, TX, USA).
3. Results
From the REGARDS-Medicare, CHS and ARIC cohorts, we included 9522, 5888, and 11,584 participants, respectively (Table 2). While the REGARDS-Medicare and CHS cohorts included only persons ≥65 years of age, 60.4% of the ARIC cohort participants were <65 years old at baseline (Table 2). The REGARDS-Medicare cohort contained a slightly higher proportion of males and blacks. Current tobacco use was more common in ARIC participants. The REGARDS-Medicare cohort exhibited a higher comorbid burden than the CHS and ARIC cohorts. History of atrial fibrillation, coronary artery disease, diabetes, deep vein thrombosis, hypertension and stroke were most prevalent in the REGARDS-Medicare cohort. Chronic lung disease was most prevalent in the CHS cohort. Peripheral artery disease was most prevalent in ARIC. Prevalence of obesity was similar between the cohorts. Among biomarkers, CHS participants exhibited a slightly higher prevalence of abnormal hsCRP and eGFR. Elevated Cystatin-C and ACR were most common among REGARDS-Medicare participants.
Median SSRS were: REGARDS-Medicare 11 points (IQR 7–16), CHS 10 (IQR 6–15), ARIC 7 (IQR 5–10). Quantile plot and histograms patterns suggested similar SSRS distributions were between the REGARDS-Medicare and CHS cohorts (Figure 1). The ARIC cohort generally exhibited lower SSRS risk scores. The number of events and incidence rates of severe sepsis events in each cohort were: REGARDS-Medicare, n = 1593, IR 30.7 per 1000 person-years (95% CI: 29.2–32.2); CHS n = 586, IR 11.9 (10.9–12.9); ARIC n = 735, IR 6.8 (6.3–7.3) (Table 3).
The SSRS showed adequate discrimination in the REGARDS-Medicare subcohort; C-statistic 0.704 (95% CI: 0.691–0.718) (Table 3). The SSRS showed good calibration in the REGARDS-Medicare subcohort across risk groups based on plots of observed and predicted survival functions (Figure 2). The SSRS also demonstrated good fit in the REGARDS-Medicare subcohort.
In the ARIC and CHS cohorts, the hazard ratios for severe sepsis events were similar across SSRS risk categories (Table 3). The SSRS showed adequate discrimination; CHS 0.696 (0.675–0.716), ARIC 0.697 (0.677–0.716) (Table 4). In ARIC and CHS, the SSRS showed good calibration across risk groups based on plots of observed and predicted survival functions (Figure 2). For the CHS cohort, the p-value for goodness of fit was statistically significant, but the observed and predicted number of events across deciles of predicted risk were similar for all cohorts. The application of the SSRS to ARIC demonstrated good fit. The observed and predicted number of events across deciles of predicted risk were similar for the ARIC and CHS cohorts (Table 4).
Across all cohorts, less than 6% of participants in the very low risk group developed severe sepsis over ten years of follow-up, while over 20% of participants in the very high risk group had an event (Table 5). In addition, less than 7% of all participants with severe sepsis in REGARDS-Medicare and CHS were classified as very low risk. In ARIC, a higher proportion of severe sepsis cases were classified as low risk, but this may not be comparable to the other cohorts due to differences in baseline risk. Less than 15% of participants without severe sepsis were classified as very high risk across all cohorts.
4. Discussion
We previously derived and internally validated the REGARDS-SRSS using severe sepsis events identified by adjudicated review of medical records [12]. In this study, we verified the ability of the REGARDS-SSRS to predict long-term severe sepsis events identified through Medicare claims in the REGARDS-Medicare subcohort. We also externally validated the SSRS using the CHS and ARIC cohorts. Our results illustrate the robustness and generalizability of the REGARDS-SSRS, supporting its potential application in the prediction of individual and community severe sepsis risk.
The potential application of the REGARDS-SSRS is in frameworks that may be unfamiliar to acute care practitioners. Existing decision rules predict sepsis or adverse outcomes among acutely hospitalized patients [29]. In contrast, the SSRS predicts long-term severe sepsis risk among community-dwelling adults. Our study underscores the concept that risk prediction and prevention could play pivotal roles in reducing the public health burden of sepsis [10]. For example, sepsis entails infection complicated by systemic inflammation and organ dysfunction, and individuals with chronic conditions may be less able to tolerate or respond to this stress. The SSRS related severe sepsis risk to the presence of chronic conditions such as chronic lung disease, peripheral artery disease, tobacco use, coronary artery disease, obesity, hypertension, deep vein thrombosis and chronic kidney disease; optimal management of these conditions might mitigate long-term sepsis risk. Statin therapy has been proposed for reduce cardiovascular risk among individuals with elevated hsCRP [30]. Similar approaches may be possible with sepsis.
Individuals with known high SSRS risk may merit personalized approaches to prevention, acute sepsis care and even health education; for example, these individuals may receive increased emphasis on vaccines and tailored education on recognizing early signs of sepsis. The SSRS may also have important public health applications, identifying community groups or clusters for organized deployment of sepsis prevention measures such as those described above. Additional study must determine if modification of these risk factors in fact alters sepsis risk.
An important aspect of our study was the application of the REGARDS-SSRS to the detection of sepsis using hospital discharge diagnoses. We originally derived the SSRS using severe sepsis events identified through structured medical record review and adjudication. While best practices would have entailed external validation with an analogously structured data set, there are presently no cohorts with severe sepsis events identified in a similar manner. Instead we validated a pragmatic and efficient alternate approach using severe sepsis events identified through Medicare claims and discharge diagnoses. This study highlights the robust nature of the REGARDS-SSRS. National health care data sets in Denmark and Taiwan have been used to characterize sepsis epidemiology in these respective countries [31,32]. Our findings suggest that potential additional inquiry or validation of the SSRS could occur using these other claims- or discharge-diagnosis based data sets.
Experts customarily recommend that clinical decision rules exhibit a discrimination (C-statistic) value of at least 0.70 on derivation and validation [28,33]. The C-statistic values observed in this study (REGARDS-Medicare 0.704, CHS 0.696, ARIC 0.697) fell just short of this threshold. One potential explanation may entail the inherent limitations of the Angus severe sepsis criteria. Discharge diagnoses do not incorporate physiologic or laboratory measures and may be affected by biases in the documented discharge diagnoses. The Angus taxonomy presumes a clinical connection between a serious infection diagnosis and an organ dysfunction diagnosis. While the original SSRS validation identified only community-acquired severe sepsis events, discharge diagnoses cannot differentiate between community-acquired, hospital-acquired or healthcare-associated severe sepsis. In a separate effort using a hospital discharge data set with “present-on-admission” flags for each diagnosis, we found that Angus severe sepsis hospitalizations encompassed approximately 62.8% community-acquired, 25.9% hospital-acquired and 11.3% healthcare associated severe sepsis cases [34]. Additional opportunities for improving the discrimination of the SSRS may encompass the identification of additional relevant clinical variables or biomarkers.
This study has limitations. Our original efforts resulted in separate risk prediction models for both sepsis (infection + ≥2 systemic inflammatory response syndrome criteria) and severe sepsis (sepsis + ≥1 SOFA organ failure point) [12]. In the current study we focused on severe sepsis as the primary endpoint because it best aligned with the discharge data available in the test cohorts. Furthermore, current Sepsis-3 consensus guidelines advocate defining sepsis as the combination of [infection + organ dysfunction], a definition that closely aligns with the definition for severe sepsis [23]. We previously noted that the predictors of the REGARDS-SRS and SSRS were nearly identical, and that 70% of the derivation sepsis events also fulfilled severe sepsis criteria. Some cases presenting to the hospital with sepsis may have later fulfilled severe sepsis criteria. In the original derivation effort, we did not focus on septic shock due to the smaller number of events in the REGARDS cohort.
In the original REGARDS-SSRS derivation we developed models both with and without biomarker variables. For the current study, we focused on the REGARDS-SSRS with biomarker elements due its higher discriminatory power and the availability of identical biomarkers in the test cohorts.
The original REGARDS-sepsis effort identified sepsis hospitalizations through manual chart review and included only events occurring within the first 28-h of hospitalization. The current analysis utilized an alternate approach entailing ICD-9 discharge diagnoses appearing in Medicare claims, and may have encompassed sepsis events at any point of hospitalization. Comparison of the agreement between manual chart review and discharge diagnoses is the objective of a separate analysis and not the focus of the current study. Prior studies suggest the low sensitivity and high sensitivity of discharge diagnoses for identifying severe sepsis [35]. Also, changes in coding practices and reimbursement incentives may influence sepsis coding [36]. These points are evident by the higher severe sepsis incidence seen in current study. However, even with these limitations, we were able to apply and validate the SSRS using these alternate sepsis identification methods. We acknowledge that the ARIC and CHS data used for validation are 10–20 years old; validation with more recent data is an important future direction.
The current effort using the REGARDS-Medicare, CHS and ARIC cohorts illustrates potential alternate strategies for detecting sepsis events. For example, Medicare claims data may capture the most significant health events for an individual. Additional study with cohorts linked to hospital discharge data may lead to additional insights regarding sepsis event detection and classification. The original SSRS was based upon the outcome of community-acquired severe sepsis. In the current study we could not differentiate community-acquired from hospital-acquired severe sepsis because the study data sets do not indicate diagnoses that were present on admission. Additional validation with a cohort incorporating present-on-admission diagnosis flags is an important future goal. While some of the biomarkers in the SSRS are relatively new (specifically, Cystatin-C), the assay can be potentially incorporated into clinical practice.
The REGARDS cohort included only African Americans and whites ≥45 years. REGARDS also did not include nursing home patients, who may be more vulnerable to sepsis than community dwelling individuals. While we used biomarkers that were readily available in REGARDS. Other biomarkers may have potentially improved model discrimination. We could not account for changes in participant characteristics over time. We did not have information on the presence of select immunosuppressive comorbidities such as human immunodeficiency virus infection or liver disease. The risk score does not include markers of future infection risk such as prior infections, antimicrobial use, or use of urinary catheters. Re-derivation of the SSRS with a broader cohort is necessary to evaluate the impact of these factors. While we conceptualized the SSRS as having roles in primary prevention, further study is also needed to confirm if risk modification in fact leads to reduced sepsis risk. We did not account for the competing risk of death because this would have complicated the interpretation of sepsis risk.
5. Conclusions
In this study we tested the ability of the REGARDS-Severe Sepsis Risk Score to predict 10-year risk of severe sepsis when applied to the REGARDS-Medicare, CHS and ARIC cohorts. The REGARDS-SSRS may potentially play a role in community sepsis prevention or mitigation efforts.
Author Contributions
H.E.W. and J.P.D. conceived the study. H.E.W. and M.S. obtained funding. H.E.W. and M.S. oversaw data collection. J.P.D. conducted the analysis. H.E.W. drafted the manuscript and all authors contributed to its critical review. H.E.W. assumes overall responsibility for the paper.
Funding
This study was supported by award R01-NR012726 from the National Institute for Nursing Research, UL1-RR025777 from the National Center for Research Resources, as well as by grants from the Center for Clinical and Translational Science and the Lister Hill Center for Health Policy of the University of Alabama at Birmingham. The parent REGARDS study was supported by cooperative agreement U01-NS041588 from the National Institute of Neurological Disorders and Stroke, National Institutes of Health, Department of Health and Human Service. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies. Representatives of the funding agencies have been involved in the review of the manuscript but not directly involved in the collection, management, analysis or interpretation of the data. Donnelly received support from grant F31-GM122180 from the National Institute of General Medical Sciences. The Cardiovascular Health Study was supported by contracts HHSN268201200036C, HHSN268200800007C, HHSN268201800001C, N01HC55222, N01HC85079, N01HC85080, N01HC85081, N01HC85082, N01HC85083, N01HC85086, and grants U01HL080295 and U01HL130114 from the National Heart, Lung, and Blood Institute (NHLBI), with additional contribution from the National Institute of Neurological Disorders and Stroke (NINDS). Additional support was provided by R01AG023629 from the National Institute on Aging (NIA). A full list of principal CHS investigators and institutions can be found at CHS-NHLBI.org. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The Atherosclerosis Risk in Communities Study is carried out as a collaborative study supported by National Heart, Lung, and Blood Institute contracts (HHSN268201100005C, HHSN268201100006C, HHSN268201100007C, HHSN268201100008C, HHSN268201100009C, HHSN268201100010C, HHSN268201100011C, and HHSN268201100012C). The funders had no role in the design or execution of this particular paper.
Acknowledgments
The authors thank the other investigators, the staff, and the participants of the REGARDS study for their valuable contributions. A full list of participating REGARDS investigators and institutions can be found at http://www.regardsstudy.org and http://www.regardssepsis.org. The authors thank the staff and participants of the CHS and ARIC studies for their important contributions.
Conflicts of Interest
The authors declare the following potential competing interests: Safford reports the following potential conflicts of interest: Amgen—salary support to study patterns of statin use in Medicare and other large databases; diaDexus—salary support for a research grant on lipids and CHD outcomes; diaDexus—consulting to help with FDA application; NIH, AHRQ—salary support for research grants. Levitan receives research support from Amgen, serves on an advisory boards for Amgen, and provides consulting for a Novartis-sponsored research study. Wang provides methodological consultation for Shire, Inc. (Lexington, MA, USA) Donnelly, Shapiro, Dai, Zhao, Heiss, Odde and Newman do not report any related conflicts of interest.
Appendix A
The REGARDS Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS) [12]. These prediction scores were previously derived using adjudicated sepsis and severe sepsis events. Validation of the SSRS Model 2 was tested in the current study.
Table A1.
The REGARDS Sepsis Risk Score (SRS) and Severe Sepsis Risk Score (SSRS).
| SEPSIS RISK SCORE (SRS) | SEVERE SEPSIS RISK SCORE (SSRS) | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions) | SRS MODEL 2 (Add Biomarkers) | SSRS MODEL 1 (Demographics, Health Behaviors, and Chronic Medical Conditions) | SSRS MODEL 2 (Add Biomarkers) | |||||||||
| Variable | HR (95% CI) | β | SRS Points | HR (95% CI) | β | SRS Points | HR (95% CI) | β | SSRS Points | HR (95% CI) | β | SSRS Points |
| Demographics, Health Behaviors and Chronic Medical Conditions | ||||||||||||
| Chronic lung disease | 2.18 (1.92–2.48) | 0.78 | 5 | 2.09 (1.83–2.38) | 0.74 | 5 | 1.93 (1.65–2.25) | 0.66 | 2 | 1.84 (1.58–2.15) | 0.61 | 5 |
| Age ≥75 years | 1.94 (1.73–2.18) | 0.66 | 4 | 1.62 (1.44–1.83) | 0.48 | 3 | 2.23 (1.96–2.54) | 0.80 | 3 | 1.71 (1.50–1.97) | 0.54 | 4 |
| Peripheral Artery Disease | 1.66 (1.29–2.13) | 0.50 | 3 | 1.50 (1.17–1.93) | 0.41 | 3 | 1.57 (1.18–2.09) | 0.45 | 2 | 1.41 (1.06–1.87) | 0.34 | 3 |
| Diabetes | 1.56 (1.40–1.75) | 0.45 | 3 | 1.41 (1.25–1.58) | 0.34 | 2 | 1.74 (1.53–1.98) | 0.55 | 2 | 1.52 (1.33–1.73) | 0.42 | 3 |
| Tobacco use (current) | 1.56 (1.36–1.78) | 0.44 | 3 | 1.40 (1.22–1.61) | 0.34 | 2 | 1.48 (1.26–1.74) | 0.39 | 1 | 1.33 (1.13–1.57) | 0.29 | 2 |
| White Race | 1.45 (1.29–1.62) | 0.37 | 2 | 1.46 (1.31–1.64) | 0.38 | 3 | 1.34 (1.18–1.52) | 0.29 | 1 | 1.34 (1.17–1.52) | 0.29 | 2 |
| Stroke | 1.42 (1.20–1.69) | 0.35 | 2 | 1.30 (1.10–1.54) | 0.27 | 2 | 1.52 (1.26–1.83) | 0.42 | 2 | 1.36 (1.13–1.63) | 0.30 | 2 |
| Atrial Fibrillation | 1.39 (1.19–1.61) | 0.33 | 2 | 1.31 (1.12–1.52) | 0.27 | 2 | 1.42 (1.20–1.68) | 0.35 | 1 | 1.32 (1.11–1.56) | 0.28 | 2 |
| Coronary artery disease | 1.35 (1.19–1.53) | 0.30 | 2 | 1.25 (1.10–1.41) | 0.22 | 2 | 1.54 (1.34–1.76) | 0.43 | 2 | 1.37 (1.20–1.57) | 0.32 | 2 |
| Obesity | 1.34 (1.21–1.50) | 0.30 | 2 | 1.20 (1.07–1.34) | 0.18 | 1 | 1.38 (1.21–1.56) | 0.32 | 1 | 1.21 (1.06–1.38) | 0.19 | 1 |
| Hypertension | 1.31 (1.16–1.46) | 0.27 | 2 | 1.15 (1.02–1.29) | 0.14 | 1 | 1.36 (1.19–1.55) | 0.31 | 1 | 1.15 (1.00–1.31) | 0.14 | 1 |
| Deep Vein Thrombosis | 1.27 (1.05–1.54) | 0.24 | 1 | 1.20 (1.00–1.46) | 0.19 | 1 | 1.31 (1.06–1.63) | 0.27 | 1 | 1.23 (0.99–1.52) | 0.21 | 2 |
| Male Sex | 1.19 (1.07–1.32) | 0.17 | 1 | 1.23 (1.11–1.37) | 0.21 | 1 | 1.43 (1.27–1.62) | 0.36 | 1 | 1.49 (1.32–1.69) | 0.40 | 3 |
| Biomarkers | ||||||||||||
| hsCRP >3.0 mg/dL | 1.48 (1.32–1.65) | 0.39 | 3 | 1.50 (1.31–1.71) | 0.40 | 3 | ||||||
| Cystatin-C ≥1.11 mg/dL | 1.62 (1.43–1.84) | 0.48 | 3 | 1.94 (1.66–2.26) | 0.66 | 5 | ||||||
| eGFR (creatinine) <60 mL/min/1.73 m2 | 1.15 (1.00–1.33) | 0.14 | 1 | 1.36 (1.15–1.59) | 0.30 | 2 | ||||||
| ACR ≥30 mcg/mg | 1.45 (1.27–1.64) | 0.37 | 3 | 1.45 (1.26–1.67) | 0.37 | 3 | ||||||
| TOTAL POINTS | 32 | 38 | 20 | 45 | ||||||||
| Discrimination (C-statistic) | 0.677 (0.677–0.679) | 0.703 (0.702–0.706) | 0.705 (0.703–0.707) | 0.742 (0.740–0.745) | ||||||||
Bolded numbers reflect total possible points for each model. hsCRP = high sensitivity C-reactive protein. ACR = Albumin-to-creatinine ratio, eGFR = estimated glomerular filtration rate.
Table A2.
Angus, et al. ICD-9 codes for serious infection [2].
Table A2.
Angus, et al. ICD-9 codes for serious infection [2].
| Infection Category | ICD-9-CM Code | ICD-9-CM Code Description |
|---|---|---|
| Parasitic | 001 | Cholera |
| 002 | Typhoid/paratyphoid fever | |
| 003 | Other salmonella infection | |
| 004 | Shigellosis | |
| 005 | Other food poisoning | |
| 008 | Intestinal infections due to Escherichia coli | |
| 008.1 | Intestinal infections due to Arizona group of paracolon bacillus | |
| 008.2 | Intestinal infections due to Aerobacter aerogenes | |
| 008.3 | Intestinal infections due to Proteus (mirabilis morganii) | |
| 008.4 | Intestinal infections due to unspecified bacteria | |
| 008.5 | Bacterial enteritis, unspecified | |
| 009 | Ill-defined intestinal infection | |
| 013 | CNS tuberculosis | |
| 018 | Miliary tuberculosis | |
| 020 | Plague | |
| 021 | Tularemia | |
| 022 | Anthrax | |
| 023 | Brucellosis | |
| 024 | Glanders | |
| 025 | Melioidosis | |
| 026 | Rat-bite fever | |
| 027 | Other bacterial zoonoses | |
| 032 | Diphtheria | |
| 033 | Whooping cough | |
| 034 | Streptococcal throat/scarlet fever | |
| 035 | Erysipelas | |
| 036 | Meningococcal infection | |
| 037 | Tetanus | |
| 038 | Septicemia | |
| 039 | Actinomycotic infections | |
| 040 | Other bacterial diseases | |
| 041 | Bacterial infection in other diseases not specified | |
| 098 | Gonococcal infections | |
| 100 | Leptospirosis | |
| 101 | Vincent’s angina | |
| 112 | Candidiasis, of mouth | |
| 112.4 | Candidiasis, of lung | |
| 112.5 | Candidiasis, disseminated | |
| 112.8 | Candidiasis, of other specified sites | |
| 114 | Coccidioidomycosis | |
| 115 | Histoplasmosis | |
| 116 | Blastomycotic infection | |
| 117 | Other mycoses | |
| 118 | Opportunistic mycoses | |
| Nervous | 320 | Bacterial meningitis |
| 321 | Cryptococcal meningitis | |
| 321.1 | Meningitis in other fungal diseases | |
| 324 | CNS abcess | |
| 325 | Phlebitis of intracranial sinus | |
| 360 | Purulent endophthalmitis | |
| 376 | Acute inflammation of orbit | |
| 380.14 | Malignant otitis externa | |
| 383 | Acute mastoiditis | |
| Circulatory | 420.99 | Acute pericarditis due to other specified organisms |
| 421 | Acute or subacute endocarditis | |
| Respiratory | 461 | Acute sinusitis |
| 462 | Acute pharyngitis | |
| 463 | Acute tonsillitis | |
| 464 | Acute laryngitis/tracheitis | |
| 465 | Acute upper respiratory infection of multiple sites/not otherwise specified | |
| 475 | Peritonsillar abscess | |
| 481 | Pneumococcal pneumonia | |
| 482 | Other bacterial pneumonia | |
| 485 | Bronchopneumonia with organism not otherwise specified | |
| 486 | Pneumonia, organism not otherwise specified | |
| 491.21 | Acute exacerbation of obstructive chronic bronchitis | |
| 494 | Bronchiectasis | |
| 510 | Empyema | |
| 513 | Abscess of lung and mediastinum | |
| Digestive | 522.5 | Periapical abscess without sinus |
| 522.7 | Periapical abscess with sinus | |
| 526.4 | Inflammatory conditions of the jaw | |
| 527.3 | Abscess of the salivary glands | |
| 528.3 | Cellulitis and abscess of oral soft tissue | |
| 540 | Acute appendicitis | |
| 541 | Appendicitis not otherwise specified | |
| 542 | Other appendicitis | |
| 562.01 | Diverticulitis of the small intestine without hemorrhage | |
| 562.03 | Diverticulitis of the small intestine with hemorrhage | |
| 562.11 | Diverticulitis of colon without hemorrhage | |
| 562.13 | Diverticulitis of colon with hemorrhage | |
| 566 | Abscess of the anal and rectal regions | |
| 567 | Peritonitis | |
| 569.5 | Intestinal abscess | |
| 569.61 | Infection of colostomy or enterostomy | |
| 569.83 | Perforation of intestine | |
| 572 | Abscess of liver | |
| 572.1 | Portal pyemia | |
| 575 | Acute cholecystitis | |
| Genitourinary | 590 | Kidney infection |
| 599 | Urinary tract infection not otherwise specified | |
| 601 | Prostatic inflammation | |
| 604 | Orchitis and epididymitis | |
| 614 | Female pelvic inflammation disease | |
| 615 | Uterine inflammatory disease | |
| 616.3 | Abscess of Bartholin’s gland | |
| 616.4 | Other abscess of vulva | |
| Pregnancy | 634 | Spontaneous abortion, complicated by genital tract and pelvic infection |
| 635 | Legally induced abortion, complicated by genital tract and pelvic infection | |
| 636 | Illegally induced abortion, complicated by genital tract and pelvic infection | |
| 637 | Unspecified abortion, complicated by genital tract and pelvic infection | |
| 638 | Failed attempted abortion, complicated by genital tract and pelvic infection | |
| 639 | Complications following abortion and ectopic and molar pregnancies | |
| 646.6 | Infections of genitourinary tract in pregnancy | |
| 658.4 | Infection of amniotic cavity | |
| 670 | Major puerperal infection | |
| 675.1 | Abscess of breast | |
| Skin | 681 | Cellulitis, finger/toe |
| 682 | Other cellulitis or abscess | |
| 683 | Acute lymphadenitis | |
| 685 | Pilonidal cyst, with abscess | |
| 686 | Other local skin infection | |
| Musculoskeletal | 711 | Pyogenic arthritis |
| 728.86 | Necrotizing fasciitis | |
| 730 | Osteomyelitis | |
| Other | 790.7 | Bacteremia |
| 958.3 | Posttraumatic wound infection, not elsewhere classified | |
| 996.6 | Infection or inflammation of device/graft | |
| 998.5 | Postoperative infection | |
| 999.3 | Infectious complication of medical care not otherwise classified | |
| 995.91 | Sepsis |
Table A3.
Angus, et al. ICD-9 codes for organ dysfunction [2].
Table A3.
Angus, et al. ICD-9 codes for organ dysfunction [2].
| Organ System | ICD-9 Code | Description |
|---|---|---|
| Cardiovascular | 458 | Orthostatic hypotension |
| 458.8 | Other specified hypotension | |
| 458.9 | Hypotension, unspecified | |
| 785.5 | Shock without mention of trauma | |
| Hematologic | 286.6 | Defibrination syndrome |
| 286.9 | Other and unspecified coagulation defects | |
| 287.4 | Secondary thrombocytopenia | |
| 287.5 | Thombocytopenia, unspecified | |
| Hepatic | 570 | Acute and subacute necrosis of liver |
| 573.4 | Hepatic infarction | |
| Neurologic | 293 | Transient organic psychosis |
| 348.1 | Anoxic brain damage | |
| 348.3 | Encephalopathy | |
| Renal | 584 | Acute renal failure |
| Respiratory | 518.8 | Respiratory failure |
| 786.03 | Apnea | |
| 799.1 | Respiratory arrest |
References
- Levy, M.M.; Fink, M.P.; Marshall, J.C.; Abraham, E.; Angus, D.; Cook, D.; Cohen, J.; Opal, S.M.; Vincent, J.L.; Ramsay, G. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive Care Med. 2003, 29, 530–538. [Google Scholar] [CrossRef] [PubMed]
- Angus, D.C.; Linde-Zwirble, W.T.; Lidicker, J.; Clermont, G.; Carcillo, J.; Pinsky, M.R. Epidemiology of severe sepsis in the United States: Analysis of incidence, outcome, and associated costs of care. Crit. Care Med. 2001, 29, 1303–1310. [Google Scholar] [CrossRef] [PubMed]
- Martin, G.S.; Mannino, D.M.; Eaton, S.; Moss, M. The epidemiology of sepsis in the United States from 1979 through 2000. N. Engl. J. Med. 2003, 348, 1546–1554. [Google Scholar] [CrossRef] [PubMed]
- Wang, H.E.; Jones, A.R.; Donnelly, J.P. Revised National Estimates of Emergency Department Visits for Sepsis in the United States. Crit. Care Med. 2017, 45, 1443–1449. [Google Scholar] [CrossRef] [PubMed]
- Dellinger, R.P.; Levy, M.M.; Rhodes, A.; Annane, D.; Gerlach, H.; Opal, S.M.; Sevransky, J.E.; Sprung, C.L.; Douglas, I.S.; Jaeschke, R.; et al. Surviving Sepsis Campaign: International Guidelines for Management of Severe Sepsis and Septic Shock: 2012. Crit. Care Med. 2013, 41, 580–637. [Google Scholar] [CrossRef] [PubMed]
- Yealy, D.M.; Kellum, J.A.; Huang, D.T.; Barnato, A.E.; Weissfeld, L.A.; Pike, F.; Terndrup, T.; Wang, H.E.; Hou, P.C.; LoVecchio, F.; et al. A randomized trial of protocol-based care for early septic shock. N. Engl. J. Med. 2014, 370, 1683–1693. [Google Scholar] [PubMed]
- Corday, E.; Corday, S.R. Advances in clinical management of acute myocardial infarction in the past 25 years. J. Am. Coll. Cardiol. 1983, 1, 126–132. [Google Scholar] [CrossRef]
- Marshall, R.S.; Mohr, J.P. Current management of ischaemic stroke. J. Neurol. Neurosurg. Psychiatry 1993, 56, 6–16. [Google Scholar] [CrossRef]
- Centers for Disease Control and Prevention (CDC). Ten great public health achievements—United States, 2001–2010. MMWR Morb. Mortal. Wkly. Rep. 2011, 60, 619–623. [Google Scholar]
- Kempker, J.A.; Wang, H.E.; Martin, G.S. Sepsis is a preventable public health problem. Crit. Care 2018, 22, 116. [Google Scholar] [CrossRef]
- Wang, H.E.; Shapiro, N.I.; Griffin, R.; Safford, M.M.; Judd, S.; Howard, G. Chronic medical conditions and risk of sepsis. PLoS ONE 2012, 7, e48307. [Google Scholar] [CrossRef] [PubMed]
- Wang, H.E.; Donnelly, J.P.; Griffin, R.; Levitan, E.B.; Shapiro, N.I.; Howard, G.; Safford, M.M. Derivation of Novel Risk Prediction Scores for Community-Acquired Sepsis and Severe Sepsis. Crit. Care Med. 2016, 44, 1285–1294. [Google Scholar] [CrossRef] [PubMed]
- Stiell, I.G.; Wells, G.A. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann. Emerg. Med. 1999, 33, 437–447. [Google Scholar] [CrossRef]
- Howard, V.J.; Cushman, M.; Pulley, L.; Gomez, C.R.; Go, R.C.; Prineas, R.J.; Graham, A.; Moy, C.S.; Howard, G. The reasons for geographic and racial differences in stroke study: Objectives and design. Neuroepidemiology 2005, 25, 135–143. [Google Scholar] [CrossRef] [PubMed]
- Muntner, P.; Gutierrez, O.M.; Zhao, H.; Fox, C.S.; Wright, N.C.; Curtis, J.R.; McClellan, W.; Wang, H.; Kilgore, M.; Warnock, D.G.; et al. Validation study of medicare claims to identify older US adults with CKD using the Reasons for Geographic and Racial Differences in Stroke (REGARDS) Study. Am. J. Kidney Dis. 2015, 65, 249–258. [Google Scholar] [CrossRef] [PubMed]
- Kumamaru, H.; Judd, S.E.; Curtis, J.R.; Ramachandran, R.; Hardy, N.C.; Rhodes, J.D.; Safford, M.M.; Kissela, B.M.; Howard, G.; Jalbert, J.J.; et al. Validity of claims-based stroke algorithms in contemporary Medicare data: Reasons for geographic and racial differences in stroke (REGARDS) study linked with medicare claims. Circ. Cardiovasc. Qual. Outcomes 2014, 7, 611–619. [Google Scholar] [CrossRef] [PubMed]
- Aaron, K.J.; Colantonio, L.D.; Deng, L.; Judd, S.E.; Locher, J.L.; Safford, M.M.; Cushman, M.; Kilgore, M.L.; Becker, D.J.; Muntner, P. Cardiovascular Health and Healthcare Utilization and Expenditures Among Medicare Beneficiaries: The REasons for Geographic And Racial Differences in Stroke (REGARDS) Study. J. Am. Heart Assoc. 2017, 6, e005106. [Google Scholar] [CrossRef]
- Xie, F.; Colantonio, L.D.; Curtis, J.R.; Safford, M.M.; Levitan, E.B.; Howard, G.; Muntner, P. Linkage of a Population-Based Cohort With Primary Data Collection to Medicare Claims: The Reasons for Geographic and Racial Differences in Stroke Study. Am. J. Epidemiol. 2016, 184, 532–544. [Google Scholar] [CrossRef] [PubMed]
- Kent, S.T.; Safford, M.M.; Zhao, H.; Levitan, E.B.; Curtis, J.R.; Kilpatrick, R.D.; Kilgore, M.L.; Muntner, P. Optimal use of available claims to identify a Medicare population free of coronary heart disease. Am. J. Epidemiol. 2015, 182, 808–819. [Google Scholar] [CrossRef]
- Colantonio, L.D.; Kent, S.T.; Kilgore, M.L.; Delzell, E.; Curtis, J.R.; Howard, G.; Safford, M.M.; Muntner, P. Agreement between Medicare pharmacy claims, self-report, and medication inventory for assessing lipid-lowering medication use. Pharmacoepidemiol. Drug Saf. 2016, 25, 827–835. [Google Scholar] [CrossRef] [Green Version]
- National Heart, Lung and Blood Institute. The Cardiovascular Health Study. Available online: https://chs-nhlbi.org/ (accessed on 10 December 2018).
- Atheroclerosis Risk in Communities Study. Available online: https://www2.cscc.unc.edu/aric/contact_the_coord_ctr (accessed on 10 December 2018).
- Singer, M.; Deutschman, C.S.; Seymour, C.W.; Shankar-Hari, M.; Annane, D.; Bauer, M.; Bellomo, R.; Bernard, G.R.; Chiche, J.D.; Coopersmith, C.M.; et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA 2016, 315, 801–810. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Merkin, S.S.; Coresh, J.; Diez Roux, A.V.; Taylor, H.A.; Powe, N.R. Area socioeconomic status and progressive CKD: The Atherosclerosis Risk in Communities (ARIC) Study. Am. J. Kidney Dis. 2005, 46, 203–213. [Google Scholar] [CrossRef] [PubMed]
- Kurella Tamura, M.; Wadley, V.; Yaffe, K.; McClure, L.A.; Howard, G.; Go, R.; Allman, R.M.; Warnock, D.G.; McClellan, W. Kidney function and cognitive impairment in US adults: The Reasons for Geographic and Racial Differences in Stroke (REGARDS) Study. Am. J. Kidney Dis. 2008, 52, 227–234. [Google Scholar] [CrossRef] [PubMed]
- Royston, P. Tools for checking calibration of a Cox model in external validation: Prediction of population-averaged survival curves based on risk groups. Stata J. 2015, 15, 275–291. [Google Scholar] [CrossRef]
- Hosmer, D.W.; Lemeshow, S.; May, S. Applied Survival Analysis: Regression Modeling of Time-To-Event Data, 2nd ed.; Wiley-Interscience: Hoboken, NJ, USA, 2008. [Google Scholar]
- Harrell, F.E. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis; Springer: New York, NY, USA, 2001. [Google Scholar]
- Shapiro, N.I.; Wolfe, R.E.; Moore, R.B.; Smith, E.; Burdick, E.; Bates, D.W. Mortality in Emergency Department Sepsis (MEDS) score: A prospectively derived and validated clinical prediction rule. Crit. Care Med. 2003, 31, 670–675. [Google Scholar] [CrossRef] [PubMed]
- Ridker, P.M.; Cannon, C.P.; Morrow, D.; Rifai, N.; Rose, L.M.; McCabe, C.H.; Pfeffer, M.A.; Braunwald, E.; Atorvastatin, E. Infection Therapy-Thrombolysis in Myocardial Infarction I: C-reactive protein levels and outcomes after statin therapy. N. Engl. J. Med. 2005, 352, 20–28. [Google Scholar] [CrossRef]
- Shen, H.-N.; Lu, C.-L.; Yang, H.-H. Epidemiologic Trend of Severe Sepsis in Taiwan from 1997 through 2006. Chest 2010, 138, 298–304. [Google Scholar] [CrossRef]
- Schonheyder, H.C.; Sogaard, M. Existing data sources for clinical epidemiology: The North Denmark Bacteremia Research Database. Clin. Epidemiol. 2010, 2, 171–178. [Google Scholar] [CrossRef]
- Steyerberg, E.W. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating; Springer: New York, NY, USA, 2009. [Google Scholar]
- Page, D.B.; Donnelly, J.P.; Wang, H.E. Community-, Healthcare-, and Hospital-Acquired Severe Sepsis Hospitalizations in the University HealthSystem Consortium. Crit. Care Med. 2015, 43, 1945–1951. [Google Scholar] [CrossRef] [Green Version]
- Rhee, C.; Dantes, R.; Epstein, L.; Murphy, D.J.; Seymour, C.W.; Iwashyna, T.J.; Kadri, S.S.; Angus, D.C.; Danner, R.L.; Fiore, A.E.; et al. Incidence and Trends of Sepsis in US Hospitals Using Clinical vs. Claims Data, 2009–2014. JAMA 2017, 318, 1241–1249. [Google Scholar] [CrossRef]
- Lindenauer, P.K.; Bernheim, S.M.; Grady, J.N.; Lin, Z.; Wang, Y.; Wang, Y.; Merrill, A.R.; Han, L.F.; Rapp, M.T.; Drye, E.E.; et al. The performance of US hospitals as reflected in risk-standardized 30-day mortality and readmission rates for medicare beneficiaries with pneumonia. J. Hosp. Med. 2010, 5, E12–E18. [Google Scholar] [CrossRef] [PubMed]
Figure 1.
Normality (panels A, C, E) and distribution (panels B, D, F) of the REGARDS Severe Sepsis Risk Score in REGARDS-Medicare, CHS and ARIC cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Figure 1.
Normality (panels A, C, E) and distribution (panels B, D, F) of the REGARDS Severe Sepsis Risk Score in REGARDS-Medicare, CHS and ARIC cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Figure 2.
Calibration of the REGARDS Severe Sepsis Risk Score in the REGARDS-Medicare (A), CHS (B) and ARIC (C) cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Figure 2.
Calibration of the REGARDS Severe Sepsis Risk Score in the REGARDS-Medicare (A), CHS (B) and ARIC (C) cohorts. REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Table 1.
The REGARDS Severe Sepsis Risk Score (SSRS).
| Variable | SSRS Points |
|---|---|
| Chronic lung disease | 5 |
| Age ≥75 years | 4 |
| Peripheral Artery Disease | 3 |
| Diabetes | 3 |
| Male Sex | 3 |
| Tobacco use (current) | 2 |
| White Race | 2 |
| Stroke | 2 |
| Atrial Fibrillation | 2 |
| Coronary artery disease | 2 |
| Deep Vein Thrombosis | 2 |
| Obesity | 1 |
| Hypertension | 1 |
| Cystatin-C ≥1.11 mg/dL | 5 |
| hsCRP >3.0 mg/dL | 3 |
| ACR ≥30 mcg/mg | 3 |
| eGFR (creatinine) <60 mL/min/1.73 m2 | 2 |
| TOTAL SSRS POINTS | 45 |
hsCRP = high sensitivity C-reactive protein. ACR = Albumin-to-creatinine ratio, eGFR = estimated glomerular filtration rate.
Table 2.
Baseline participant characteristics by cohort.
| Characteristic | Internal Application | External Validation | |
|---|---|---|---|
| REGARDS-Medicare (n = 9522) | CHS (n = 5888) | ARIC (n = 11,584) | |
| Age | |||
| <65 years | 0 | 0 | 6998 (60.4) |
| 65–74 | 6120 (64.3) | 3894 (66.1) | 4583 (39.6) |
| ≥75 | 3402 (35.7) | 1994 (33.9) | 3 (0.0) |
| Gender | |||
| Male | 4692 (49.3) | 2495 (42.4) | 5106 (44.1) |
| Female | 4830 (50.7) | 3393 (57.6) | 6478 (55.9) |
| Race | |||
| White | 6440 (67.6) | 4925 (83.6) | 8899 (76.8) |
| Black | 3082 (32.4) | 963 (16.4) | 2685 (23.2) |
| Tobacco Use | |||
| Current | 910 (9.6) | 700 (11.9) | 1700 (14.7) |
| No Current Use | 8612 (90.4) | 5188 (88.1) | 9884 (85.3) |
| Chronic Medical Conditions | |||
| Atrial Fibrillation | 1025 (10.8) | 236 (4.0) | 324 (2.8) |
| Chronic Lung Disease | 1013 (10.6) | 915 (15.5) | 929 (8.0) |
| Coronary Artery Disease | 2364 (24.8) | 1,154 (19.6) | 965 (8.3) |
| Diabetes | 2276 (23.9) | 989 (16.8) | 1924 (16.6) |
| Deep Vein Thrombosis | 648 (6.8) | 312 (5.3) | 348 (3.0) |
| Hypertension | 6200 (65.1) | 2619 (44.5) | 5510 (47.6) |
| Obesity | 4746 (49.8) | 2747 (46.7) | 7592 (65.5) |
| Peripheral Artery Disease | 275 (2.9) | 120 (2.0) | 385 (3.3) |
| Stroke | 796 (8.4) | 249 (4.2) | 265 (2.3) |
| High-Sensitivity C-Reactive Protein | |||
| >3.0 mg/dL | 3449 (36.2) | 2481 (42.1) | 4909 (42.4) |
| ≤3.0 mg/dL | 6073 (63.8) | 3407 (57.9) | 6675 (57.6) |
| Cystatin-C | |||
| ≥1.11 mg/dL | 3486 (36.6) | 1569 (26.7) | 1504 (13.0) |
| <1.11 mg/dL | 6036 (63.4) | 4319 (73.4) | 10,080 (87.0) |
| eGFR (creatinine) | |||
| <60 mL/min/1.73 m2 | 1723 (18.1) | 1368 (24.1) | 1112 (9.6) |
| ≥60 mL/min/1.73 m2 | 7799 (81.9) | 4520 (76.8) | 10,472 (90.4) |
| Albumin-to-Creatinine Ratio | |||
| ≥30 mcg/mg | 1612 (16.9) | 654 (11.1) | 956 (8.3) |
| <30 mcg/mg | 7910 (83.1) | 5234 (88.9) | 10,628 (91.8) |
| SSRS Risk Group | |||
| Very Low (<6 Points) | 1616 (17.0) | 1137 (19.3) | 3761 (32.5) |
| Low (6–9 Points) | 2308 (24.2) | 1582 (26.9) | 4388 (37.9) |
| Medium (10–13 Points) | 2127 (22.3) | 1358 (23.1) | 2037 (17.6) |
| High (14–17 Points) | 1687 (17.7) | 988 (16.8) | 900 (7.8) |
| Very High (>17 Points) | 1784 (18.7) | 823 (14.0) | 498 (4.3) |
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Table 3.
Severe sepsis incidence for each cohort, overall and stratified by REGARDS-Severe Sepsis Risk Score risk category.
Table 3.
Severe sepsis incidence for each cohort, overall and stratified by REGARDS-Severe Sepsis Risk Score risk category.
| Risk Group | Internal Application | External Validation | ||||
|---|---|---|---|---|---|---|
| REGARDS-Medicare (n = 9522) | CHS (n = 5888) | ARIC (n = 11,584) | ||||
| Severe Sepsis Events (n) IR (95% CI) | HR (95% CI) | Severe Sepsis Events (n) IR (95% CI) | HR (95% CI) | Severe Sepsis Events (n) IR (95% CI) | HR (95% CI) | |
| Full Cohort | 1593 30.7 (29.2–32.2) | --- | 586 11.9 (10.9–12.9) | --- | 735 6.8 (6.3–7.3) | --- |
| Stratified by REGARDS-SSRS Risk Category | ||||||
| Very Low (<6 Points) | 96 9.9 (8.1–12.1) | Reference | 39 3.6 (2.6–4.9) | Reference | 114 3.1 (2.6–3.8) | Reference |
| Low (6–9 Points) | 230 17.0 (14.9–19.3) | 1.73 (1.36–2.19) | 99 6.9 (5.7–8.4) | 1.94 (1.34–2.82) | 208 5.0 (4.3–5.7) | 1.60 (1.27–2.01) |
| Medium (10–13 Points) | 299 24.8 (22.2–27.8) | 2.53 (2.01–3.19) | 143 12.8 (10.8–15.0) | 3.71 (2.61–5.29) | 179 9.7 (8.4–11.2) | 3.15 (2.49–3.99) |
| High (14–17 Points) | 374 42.0 (37.9–46.5) | 4.36 (3.48–5.45) | 138 18.4 (15.6–21.8) | 5.51 (3.86–7.86) | 118 15.6 (13.0–18.7) | 5.20 (4.02–6.73) |
| Very High (>17 Points) | 594 76.8 (70.9–83.3) | 8.27 (6.66–10.26) | 167 30.2 (25.9–35.1) | 9.46 (6.67–13.41) | 116 33.7 (28.1–40.4) | 11.98 (9.24–15.52) |
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities. IR = incidence rate per 1000 person-years. HR = hazard ratio. CI = confidence interval.
Table 4.
Discrimination, calibration, and goodness-of-fit of the REGARDS-Severe Sepsis Risk Score for each cohort.
Table 4.
Discrimination, calibration, and goodness-of-fit of the REGARDS-Severe Sepsis Risk Score for each cohort.
| Measure | REGARDS-Medicare | CHS | ARIC |
|---|---|---|---|
| Discrimination—C-statistic (95% CI) | 0.704 (0.691–0.718) | 0.696 (0.675–0.716) | 0.697 (0.677–0.716) |
| Decile of Predicted Risk (Observed/Predicted Events) | |||
| 1 | 60/59.9 | 29/30.7 | 41/41.0 |
| 2 | 89/95.1 | 29/47.1 | 31/23.2 |
| 3 | 97/108.1 | 30/25.5 | 42/47.9 |
| 4 | 80/68.1 | 50/55.4 | 56/61.9 |
| 5 | 150/147.5 | 27/28.4 | 54/56.6 |
| 6 | 149/162.9 | 80/61.1 | 44/48.3 |
| 7 | 179/172.3 | 78/69.5 | 54/48.2 |
| 8 | 195/183.3 | 67/61.2 | 93/90.6 |
| 9 | 302/298.8 | 80/83.9 | 124/116.8 |
| 10 | 292/297.0 | 116/123.2 | 196/200.4 |
| Goodness of Fit—Grønnesby and Borgan test (Chi-square; p-value) | 7.66 (0.57) | 18.95 (0.03) | 5.72 (0.77) |
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities.
Table 5.
Percentage positive with severe sepsis, true positive rate, and true negative rate by REGARDS-Severe Sepsis Risk Score risk category.
Table 5.
Percentage positive with severe sepsis, true positive rate, and true negative rate by REGARDS-Severe Sepsis Risk Score risk category.
| Risk Group | REGARDS-Medicare | CHS | ARIC | ||||||
|---|---|---|---|---|---|---|---|---|---|
| PP (% with Severe Sepsis) | TPR (% of Cases) | TNR (% of Non-Cases) | PP (% with Severe Sepsis) | TPR (% of Cases) | TNR (% of Non-Cases) | PP (% with Severe Sepsis) | TPR (% of Cases) | TNR (% of Non-Cases) | |
| Very Low (<6 Points) | 5.9 | 6.0 | 19.2 | 3.4 | 6.7 | 20.7 | 3.0 | 15.5 | 33.6 |
| Low (6–9 Points) | 10.0 | 14.4 | 26.2 | 6.3 | 16.9 | 28.0 | 4.7 | 28.3 | 38.5 |
| Medium (10–13 Points) | 14.1 | 18.8 | 23.1 | 10.5 | 24.4 | 22.9 | 8.8 | 24.4 | 17.1 |
| High (14–17 Points) | 22.2 | 23.5 | 16.6 | 14.0 | 23.5 | 16.0 | 13.1 | 16.1 | 7.2 |
| Very High (>17 Points) | 33.3 | 37.3 | 15.0 | 20.3 | 28.5 | 12.4 | 23.3 | 15.8 | 3.5 |
REGARDS = Reasons for Geographic and Racial Differences in Stroke. CHS = Cardiovascular Health Study. ARIC = Atherosclerosis Risk in Communities. PP = percent positive. TPR = true positive rate. TNR = true negative rate.
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
MDPI and ACS Style
Wang, H.E.; Donnelly, J.P.; Yende, S.; Levitan, E.B.; Shapiro, N.I.; Dai, Y.; Zhao, H.; Heiss, G.; Odden, M.; Newman, A.; et al. Validation of the REGARDS Severe Sepsis Risk Score. J. Clin. Med. 2018, 7, 536. https://doi.org/10.3390/jcm7120536
AMA Style
Wang HE, Donnelly JP, Yende S, Levitan EB, Shapiro NI, Dai Y, Zhao H, Heiss G, Odden M, Newman A, et al. Validation of the REGARDS Severe Sepsis Risk Score. Journal of Clinical Medicine. 2018; 7(12):536. https://doi.org/10.3390/jcm7120536
Chicago/Turabian StyleWang, Henry E., John P. Donnelly, Sachin Yende, Emily B. Levitan, Nathan I. Shapiro, Yuling Dai, Hong Zhao, Gerardo Heiss, Michelle Odden, Anne Newman, and et al. 2018. "Validation of the REGARDS Severe Sepsis Risk Score" Journal of Clinical Medicine 7, no. 12: 536. https://doi.org/10.3390/jcm7120536
APA StyleWang, H. E., Donnelly, J. P., Yende, S., Levitan, E. B., Shapiro, N. I., Dai, Y., Zhao, H., Heiss, G., Odden, M., Newman, A., & Safford, M. (2018). Validation of the REGARDS Severe Sepsis Risk Score. Journal of Clinical Medicine, 7(12), 536. https://doi.org/10.3390/jcm7120536
Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.
