Original Investigation
Cardiology
April 30, 2024
Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review
Christoph B. Olivier, Lasse Struß, Nathalie Sünnen, et al
JAMA Netw Open. 2024;7(4):e248818. doi:10.1001/jamanetworkopen.2024.8818
Question How accurate are estimated event rates and effect sizes in contemporary cardiovascular randomized clinical trials (RCTs)?
Findings In this systematic review of 344 contemporary cardiovascular RCTs, overestimation of event rates was common. In addition, 4 of 5 trials overestimated the effect size of the tested intervention.
Meaning These findings suggest that event rates and effect sizes in contemporary cardiovascular RCTs are frequently overestimated, which may contribute to the inability to answer the trial hypothesis.
Abstract
Importance For the design of a randomized clinical trial (RCT), estimation of the expected event rate and effect size of an intervention is needed to calculate the sample size. Overestimation may lead to an underpowered trial.
Objective To evaluate the accuracy of published estimates of event rate and effect size in contemporary cardiovascular RCTs.
Evidence Review A systematic search was conducted in MEDLINE for multicenter cardiovascular RCTs associated with MeSH (Medical Subject Headings) terms for cardiovascular diseases published in the New England Journal of Medicine, JAMA, or the Lancet between January 1, 2010, and December 31, 2019. Identified trials underwent abstract review; eligible trials then underwent full review, and those with insufficiently reported data were excluded. Data were extracted from the original publication or the study protocol, and a random-effects model was used for data pooling. This review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline. The primary outcome was the accuracy of event rate and effect size estimation. Accuracy was determined by comparing the observed event rate in the control group and the effect size with their hypothesized values. Linear regression was used to determine the association between estimation accuracy and trial characteristics.
Findings Of the 873 RCTs identified, 374 underwent full review and 30 were subsequently excluded, resulting in 344 trials for analysis. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of −12.3% (95% CI, −16.4% to −5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. Accuracy of event rate estimation was associated with a higher likelihood of refuting the null hypothesis (0.13 [95% CI, 0.01 to 0.25]; P = .03). The median observed effect size in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). A total of 216 trials (82.1%) overestimated the effect size.
Conclusions and Relevance In this systematic review of contemporary cardiovascular RCTs, event rates of the primary end point and effect sizes of an intervention were frequently overestimated. This overestimation may have contributed to the inability to adequately test the trial hypothesis.
