Original Investigation
Health Informatics
October 15, 2024
Early Warning Scores With and Without Artificial Intelligence
Dana P. Edelson, Matthew M. Churpek, Kyle A. Carey, et al
JAMA Netw Open. 2024;7(10):e2438986. doi:10.1001/jamanetworkopen.2024.38986
Question How do hospital early warning scores compare with one another?
Findings In this cohort study that compared 6 early warning scores across 362 926 patient encounters, eCARTv5, a machine learning model, identified clinical deterioration best with an area under the receiver operating characteristics curve (AUROC) of 0.895 and the highest positive predictive values at both the moderate- and high-risk matched thresholds. The National Early Warning Score, a non–artificial intelligence score with an AUROC of 0.831, was the second-best performer at both thresholds, while the Epic Deterioration Index was one of the worst, with an AUROC of 0.808 and the lowest positive predictive values.
Meaning Given the wide variation in accuracy, these findings suggest that additional transparency and oversight of early warning tools may be warranted.
Abstract
Importance Early warning decision support tools to identify clinical deterioration in the hospital are widely used, but there is little information on their comparative performance.
Objective To compare 3 proprietary artificial intelligence (AI) early warning scores and 3 publicly available simple aggregated weighted scores.
Design, Setting, and Participants This retrospective cohort study was performed at 7 hospitals in the Yale New Haven Health System. All consecutive adult medical-surgical ward hospital encounters between March 9, 2019, and November 9, 2023, were included.
Exposures Simultaneous Epic Deterioration Index (EDI), Rothman Index (RI), eCARTv5 (eCART), Modified Early Warning Score (MEWS), National Early Warning Score (NEWS), and NEWS2 scores.
Main Outcomes and Measures Clinical deterioration, defined as a transfer from ward to intensive care unit or death within 24 hours of an observation.
Results Of the 362 926 patient encounters (median patient age, 64 [IQR, 47-77] years; 200 642 [55.3%] female), 16 693 (4.6%) experienced a clinical deterioration event. eCART had the highest area under the receiver operating characteristic curve at 0.895 (95% CI, 0.891-0.900), followed by NEWS2 at 0.831 (95% CI, 0.826-0.836), NEWS at 0.829 (95% CI, 0.824-0.835), RI at 0.828 (95% CI, 0.823-0.834), EDI at 0.808 (95% CI, 0.802-0.812), and MEWS at 0.757 (95% CI, 0.750-0.764). After matching scores at the moderate-risk sensitivity level for a NEWS score of 5, overall positive predictive values (PPVs) ranged from a low of 6.3% (95% CI, 6.1%-6.4%) for an EDI score of 41 to a high of 17.3% (95% CI, 16.9%-17.8%) for an eCART score of 94. Matching scores at the high-risk specificity of a NEWS score of 7 yielded overall PPVs ranging from a low of 14.5% (95% CI, 14.0%-15.2%) for an EDI score of 54 to a high of 23.3% (95% CI, 22.7%-24.2%) for an eCART score of 97. The moderate-risk thresholds provided a median of at least 20 hours of lead time for all the scores. Median lead time at the high-risk threshold was 11 (IQR, 0-69) hours for eCART, 8 (IQR, 0-63) hours for NEWS, 6 (IQR, 0-62) hours for NEWS2, 5 (IQR, 0-56) hours for MEWS, 1 (IQR, 0-39) hour for EDI, and 0 (IQR, 0-42) hours for RI.
Conclusions and Relevance In this cohort study of inpatient encounters, eCART outperformed the other AI and non-AI scores, identifying more deteriorating patients with fewer false alarms and sufficient time to intervene. NEWS, a non-AI, publicly available early warning score, significantly outperformed EDI. Given the wide variation in accuracy, additional transparency and oversight of early warning tools may be warranted.
