Original Investigation 

Infectious Diseases

Syndromic Analysis of Sepsis Cohorts Using Large Language Models

Theodore R. Pak, Sanjat Kanjilal, Caroline S. McKenna, et al

JAMA Netw Open 2025;8;(10):e2539267. doi:10.1001/jamanetworkopen.2025.39267

Key Points

Question  Can large language models (LLMs) accurately extract presenting signs and symptoms from clinical notes to identify associations between symptoms, multidrug-resistant infections, and in-hospital mortality in large cohorts of patients with possible sepsis?

Findings  In this cohort study of 104 248 patients with possible infection, LLMs extracted signs and symptoms from admission notes with accuracy comparable to that of physicians performing manual medical records review. Hierarchical clustering identified 7 symptom-based syndromes that correlated with infection sources, risk for methicillin-resistant Staphylococcus aureus and multidrug-resistant gram-negative organisms, and in-hospital death.

Meaning  Findings of this study suggest that LLMs can enable the efficient, large-scale extraction of signs and symptoms from clinical notes and the differential correlation of syndromes with infection sources, multidrug-resistant infections, and mortality; the value of large-scale sign-and-symptom data in models of antibiotic choice, effectiveness, and outcomes in patients with sepsis warrants further study.

Abstract

Importance  Presenting signs and symptoms affect the care of patients with possible sepsis. However, signs and symptoms are not incorporated into most large observational studies because they are difficult to extract from clinical notes at scale.

Objective  To assess the use of large language models (LLMs) to extract presenting signs and symptoms from admission notes and characterize their associations with infectious diagnoses, multidrug-resistant infections, and mortality.

Design, Setting, and Participants  This retrospective cohort study obtained data from 5 Massachusetts hospitals within 1 health care system between June 1, 2015, and August 1, 2022. Participants were hospitalized adult patients with possible infection (determined by blood culture drawn and intravenous antibiotics administered within 24 hours of arrival). An LLM (LLaMA 3 8B; Meta) was used to extract up to 10 presenting signs and symptoms from each patient’s history-and-physical admission notes. LLM-generated labels were validated by blinded review of 303 random admission notes. Data analyses were performed from July 2023 to August 2025.

Exposures  Thirty most common signs and symptoms were retained as exposures, and unsupervised clustering was used to create syndromes, which were compared with infection sources derived from the International Statistical Classification of Diseases, Tenth Revision, Clinical Modification discharge codes.

Main Outcomes and Measures  Outcomes included positive cultures for methicillin-resistant Staphylococcus aureus (MRSA), positive cultures for multidrug-resistant gram-negative (MDRGN) organisms, and in-hospital mortality. Multivariable logistic regression was used to adjust for demographics, comorbidities, physiologic markers of severity of illness, and time to antibiotics.

Results  Among the 104 248 patients (median [IQR] age, 66 [52-78] years; 54 137 males [51.9%]) included, 23 619 (22.7%) had sepsis without shock, 25 990 (24.9%) had septic shock, and 94 913 (91.0%) had 1 or more admission note within 24 hours. The LLM labeled the notes of 93 674 of 94 913 patients (98.7%). On manual validation, LLM labels had an accuracy of 99.3% (95% CI, 99.2%-99.3%), balanced accuracy of 84.6% (95% CI, 83.5%-85.8%), positive predictive value of 68.4% (95% CI, 66.0%-70.7%), sensitivity of 69.7% (95% CI, 67.3%-72.0%), and specificity of 99.6% (95% CI, 99.6%-99.6%) compared with the physician medical record reviewer. The 30 most common signs and symptoms were clustered into syndromes that correlated with infection sources. Presence of skin and soft tissue symptoms (adjusted odds ratio [AOR], 1.73; 95% CI, 1.49-2.00) and absence of gastrointestinal (AOR, 0.63; 95% CI, 0.54-0.73) or urinary tract symptoms (AOR, 0.34; 95% CI, 0.22-0.50) were associated with MRSA culture positivity; inverse associations were seen for MDRGN organisms. Cardiopulmonary symptoms were associated with increased mortality (AOR, 1.30; 95% CI, 1.17-1.45).

Conclusions and Relevance  This cohort study found that an LLM accurately extracted presenting signs and symptoms from admission notes that clustered into syndromes differentially correlated with infection sources, multidrug-resistant infections, and mortality. Further research is warranted to evaluate the value of large-scale sign-and-symptom data in models of antibiotic choice, effectiveness, and outcomes in patients with possible sepsis.