Invited Commentary
March 13, 2024
Standardizing Resident Assessment and Competency—Milestones Matter
Amanda C. Filiberto, Amalia Cochran
JAMA Surg. Published online March 13, 2024. doi:10.1001/jamasurg.2024.0045
Smith et al1 present findings from an observational cohort study of early-career vascular surgeons and patient outcomes following endovascular aneurysm repair (EVAR), where lower mean Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings were associated with major complications in early surgical practice. Milestone ratings were corrected for program effect using a deviation score from the program mean and surgeon performance was assessed controlling for patient-level and health care professional–level covariates. Of note, this association was limited to those programs with lower program-level mean Milestone ratings. For graduates with Milestone mean ratings of 3.5, the odds ratio (OR) of major complications was 0.50 (OR, 0.50; 95% CI, 0.26-0.95), which is equivalent to a 2 times increased risk for every 1-point decrease in Milestone rating.
The authors1 point out that the ACGME first implemented a requirement for all training programs to report Milestone ratings for trainees a decade ago, yet to date, there are limited data to support any improvement in the quality of resident education associated with this requirement. The authors succinctly summarize the available literature, which focuses on board examination performance and certification status as a surrogate for competence, with results that vary widely depending on specialty and outcomes database analyzed.2 Importantly, the authors highlight their results in the context of implications for training residents and fellows, what it means for patients, and appreciate the potential for Milestones to identify struggling trainees, with an opportunity to intervene prior to them starting independent clinical practice as a way to improve patient outcomes.
As the only other study evaluating Milestones and patient outcomes, Kendrick et al3 analyzed patient outcomes of early career surgeons using Centers for Medicare and Medicaid Services claims data and found no association between mean Milestone ratings during training and patient outcomes. Smith et al1 underscore important differences between their study and that of Kendrick et al: (1) vascular surgery Milestones provide greater specificity within the competency domains than the general surgery Milestones, (2) the Vascular Quality Initiative registry allows for more detailed analysis of procedure-specific outcomes, and (3) program effects were controlled, as the training program may result in systematic bias of ratings of an individual trainee’s performance. Another salient difference not discussed is that Kendrick et al evaluated nonfellowship-trained general surgeons performing high-risk general surgery procedures in an older patient population, which differs greatly from vascular surgery fellowship and integrated vascular surgery residency–trained surgeons doing elective endovascular aortic repairs on patients of all ages.
Consistent with the move toward competency-based education and assessment, the present study1 suggests that the ACMGE’s efforts to standardize resident assessment and competency with Milestones are on point. For the first time, albeit in a single specialty and for a specific patient population, we have longitudinal data on resident performance during training linked to patient outcomes following training. Milestones, in combination with entrustable professional activities,4 should be used to create a unifying framework and to provide more learner-centered assessment and feedback to surgery residents and fellows. Full implementation of both Milestones and entrustable professional activities will benefit both our profession and our patients.
