Research Letter
November 20, 2024
The Quality of Evidence Supporting Clinical Practice Guidelines in General Surgery: A Meta-Analysis
Ali B. Abbasi, Adrian Valderrama, Hannah C. Decker, et al
JAMA Surg. Published online November 20, 2024. doi:10.1001/jamasurg.2024.4751
Clinical practice guidelines can help patients and physicians make decisions using the best available evidence. Unfortunately, the best available evidence is often of low quality.1 In many areas of surgery, evidence may be lacking because randomized clinical trials (RCTs), the gold standard of evidence, are relatively challenging to conduct2 and can be at high risk for bias.3 However, the evidence shortage in surgery has not been systematically quantified. Here, we examined the quality of evidence supporting clinical practice guidelines in general surgery.
Methods
We identified US-based professional societies that publish clinical practice guidelines in general surgery as defined by the American College of Surgeons (ACS)4 (eMethods and eTable 2 in Supplement 1). We searched guidelines published by these societies between January 1, 2014, and December 31, 2023, that were related to clinical care and used the Grading of Recommendations, Assessment, Development and Evaluations (GRADE) method5 (eFigure and eTable 1 in Supplement 1).
GRADE is a commonly used framework for guideline creation, with rating scales for both recommendation strength (strong, weak) and evidence quality (very low, low, medium, high). RCTs start at high quality while observational studies start at low quality, and evidence quality can be decreased (eg, for risk of bias) or increased (eg, if the effect size is large). From each guideline, we extracted all recommendations, including GRADE ratings according to guideline authors. We classified recommendations by phases of surgical care, as defined by the ACS6 or relating to nonoperative care. At least 2 reviewers independently classified each recommendation, and disagreements between reviewers were resolved by consensus. We summarized evidence quality and recommendation strength, stratified by phase of surgical care, and cross-tabulated them. Data analysis was performed with R, version 4.3 (R Project for Statistical Computing).
Results
We identified 7 professional societies that created guidelines using GRADE. These societies published 153 guidelines, 90 of which met our inclusion criteria, containing 1410 individual recommendations. Evidence quality was rated by guideline authors as high for 143 (10.1%), moderate for 525 (37.2%), low for 499 (35.4%), and very low for 173 (12.3%) recommendations and was not graded for 70 (5.0%) recommendations. Evidence quality was least likely to be high in recommendations for preoperative (17 of 290 [5.9%]) and intraoperative (53 of 568 [9.3%]) phases of care and most likely to be high for perioperative (16 of 86 [18.6%]) and postoperative (10 of 76 [13.2%]) care (Table 1).
Regarding recommendation strength, 722 (51.2%) were strong and 609 (43.2%) were weak, with the remaining 79 (5.6%) not rated. Among strong recommendations, 129 (17.9%) were supported by high-quality evidence compared with 14 (2.3%) weak recommendations (Table 2).
Discussion
In this assessment of clinical practice guidelines in general surgery, we found that a minority of all recommendations were backed by high-quality evidence, while almost half of all recommendations and over a third of strong recommendations were backed by low- or very low–quality evidence. In the GRADE framework, low quality suggests the true outcome may be markedly different from the estimated outcome, and very low quality indicates the true outcome is probably markedly different.5 These findings highlight that physicians and patients must frequently make decisions despite substantial uncertainty regarding actual risks and benefits of surgical treatment options, even for a course of action strongly recommended by professional societies.
Practices concerning the intraoperative phase of care, such as choice or timing of operation, are less commonly supported by high-quality evidence than practices concerning perioperative care, such as pain or anesthetic management, or nonoperative care. This finding suggests a need to prioritize generating evidence to address intraoperative decision-making.
A study limitation is our focus on guidelines that used GRADE, which enabled us to compare levels of evidence across different guidelines. As a result, this study may not generalize to guidelines that used different methods.
