Original Investigation
May 6, 2024
Associations of Internal Medicine Residency Milestone Ratings and Certification Examination Scores With Patient Outcomes
Bradley M. Gray, Jonathan L. Vandergrift, Jennifer P. Stevens, et al
JAMA. Published online May 6, 2024. doi:10.1001/jama.2024.5268
Question Internal medicine residents’ competency is evaluated with milestone ratings and the American Board of Internal Medicine’s certification examination. Is physicians’ performance on either of these measures related to their hospitalized patients’ outcomes?
Findings We analyzed 6898 newly trained hospitalists treating Medicare fee-for-service beneficiaries during 455 120 hospitalizations occurring in 2017 to 2019. We found no associations between overall milestone ratings or medical knowledge ratings and hospitalization outcomes, but certification examination score was associated with reduced 7-day mortality and readmissions.
Meaning Among newly trained hospitalists, certification examination score, but not residency milestone ratings, was associated with improved outcomes among hospitalized Medicare beneficiaries.
Abstract
Importance Despite its importance to medical education and competency assessment for internal medicine trainees, evidence about the relationship between physicians’ milestone residency ratings or the American Board of Internal Medicine’s initial certification examination and their hospitalized patients’ outcomes is sparse.
Objective To examine the association between physicians’ milestone ratings and certification examination scores and hospital outcomes for their patients.
Design, Setting, and Participants Retrospective cohort analyses of 6898 hospitalists completing training in 2016 to 2018 and caring for Medicare fee-for-service beneficiaries during hospitalizations in 2017 to 2019 at US hospitals.
Main Outcomes and Measures Primary outcome measures included 7-day mortality and readmission rates. Thirty-day mortality and readmission rates, length of stay, and subspecialist consultation frequency were also assessed. Analyses accounted for hospital fixed effects and adjusted for patient characteristics, physician years of experience, and year.
Exposures Certification examination score quartile and milestone ratings, including an overall core competency rating measure equaling the mean of the end of residency milestone subcompetency ratings categorized as low, medium, or high, and a knowledge core competency measure categorized similarly.
Results Among 455 120 hospitalizations, median patient age was 79 years (IQR, 73-86 years), 56.5% of patients were female, 1.9% were Asian, 9.8% were Black, 4.6% were Hispanic, and 81.9% were White. The 7-day mortality and readmission rates were 3.5% (95% CI, 3.4%-3.6%) and 5.6% (95% CI, 5.5%-5.6%), respectively, and were 8.8% (95% CI, 8.7%-8.9%) and 16.6% (95% CI, 16.5%-16.7%) for mortality and readmission at 30 days. Mean length of stay and number of specialty consultations were 3.6 days (95% CI, 3.6-3.6 days) and 1.01 (95% CI, 1.00-1.03), respectively. A high vs low overall or knowledge milestone core competency rating was associated with none of the outcome measures assessed. For example, a high vs low overall core competency rating was associated with a nonsignificant 2.7% increase in 7-day mortality rates (95% CI, −5.2% to 10.6%; P = .51). In contrast, top vs bottom examination score quartile was associated with a significant 8.0% reduction in 7-day mortality rates (95% CI, −13.0% to −3.1%; P = .002) and a 9.3% reduction in 7-day readmission rates (95% CI, −13.0% to −5.7%; P < .001). For 30-day mortality, this association was −3.5% (95% CI, −6.7% to −0.4%; P = .03). Top vs bottom examination score quartile was associated with 2.4% more consultations (95% CI, 0.8%-3.9%; P < .003) but was not associated with length of stay or 30-day readmission rates.
Conclusions and Relevance Among newly trained hospitalists, certification examination score, but not residency milestone ratings, was associated with improved outcomes among hospitalized Medicare beneficiaries.
