Guide to Statistics and Methods <\/p>\n\n\n\n
Surgical Education Research<\/p>\n\n\n\n
January 3, 2024<\/p>\n\n\n\n
Introduction<\/p>\n\n\n\n
Over the past 2 decades, surgical education literature has seen tremendous growth driven by changes in graduate medical education, such as work hour restrictions, a focus on competency-based education, the incorporation of simulation, and proper assessment. Some authors have criticized the quality of education research and indicated a need for improvement.1<\/a><\/sup> Quality issues have been attributed to decreased funding for education research, limiting the ability to conduct rigorous, multicenter trials.2<\/a><\/sup> The objective of this article is to describe common methodological flaws in surgical education research to help prospective authors avoid errors and to help reviewers better recognize them (Box<\/a>).<\/a><\/a><\/p>\n\n\n\n Box. <\/p>\n\n\n\n Using the Methodology<\/p>\n\n\n\n This article addresses design and statistical flaws that are common across methodologies. We recommend adhering to reporting guidelines as outlined by the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network (https:\/\/www.equator-network.org\/<\/a>).<\/a><\/p>\n\n\n\n Statistical Considerations<\/p>\n\n\n\n Common Flaw 1: Type II Error<\/p>\n\n\n\n What Is the Issue?<\/p>\n\n\n\n Not unique to surgical education research, type II errors frequently plague papers in the field.3<\/a><\/sup> A type II error occurs when one fails to reject a null hypothesis that is false, primarily due to insufficient power of a study, which is determined by sample size, effect size, significance level, and variance. The sample size is low in many education research studies due to the target population of learners typically being limited in a department of surgery. The effect size reflects the magnitude of the true underlying difference between groups that is clinically or practically important; the larger the effect size is, the more relevant the results are. It is critical to determine the clinically or educationally significant effect size (not just the statistically significant difference) prior to performing any sample size calculation or power analysis. The desired clinically or educationally important effect estimate (risk difference, risk ratio, odds ratio, mean or median difference) and the associated confidence interval determine the sample size. The statistical significance level (generally set at \u03b1\u2009=\u2009.05), the power of the study (generally set at \u03b2\u2009=\u2009.20, or 80% power), and variance (usually not known at the study\u2019s beginning) are less under investigator control.<\/a><\/p>\n\n\n\n How to Recognize It<\/p>\n\n\n\n Determine if the authors identified a clinically significant effect estimate and calculated a sample size based on the desired power to detect this effect. If the authors have not described an effect size or determined a sufficient sample size, then failure to detect a statistically and clinically significant effect may not indicate a lack of a difference or association but rather a type II error due to insufficient power. Conclusions may not be valid if the power analysis and sample size calculation were performed post hoc.<\/a><\/p>\n\n\n\n How to Avoid It<\/p>\n\n\n\n Investigators must first identify what constitutes a minimum clinically significant effect and calculate a sample size based on the desired power to detect this effect. Rather than the P<\/em> value, investigators should report the effect estimate and the associated confidence interval for the reader. Study participants should be representative of the population of interest and without selection bias.<\/a><\/p>\n\n\n\n Common Flaw 2: Poor Participant Matching at Baseline for Studies Comparing 2 Cohorts<\/p>\n\n\n\n What Is the Issue?<\/p>\n\n\n\n Flaw 1 described earlier is amplified when study groups are not well balanced at baseline, especially as it relates to the primary outcome of the study or the factors that may directly affect the primary outcome. Systematic differences between comparison cohorts indicate selection bias.<\/a><\/p>\n\n\n\n How to Recognize It<\/p>\n\n\n\n Whether an observational study or a randomized clinical trial, one should look at the baseline differences between compared groups and determine whether the groups are similar as it relates to the outcome of interest. Consider a study investigating the impact of laparoscopic simulator training on learner performance in 2 groups using 2 different approaches. At baseline, the mean (SD) scores were 40 (15) for group A and 15 (18) for group B (P<\/em>\u2009=\u2009.15). After training completion, scores were 90 (9) and 100 (7), respectively (P<\/em>\u2009=\u2009.09). The author\u2019s conclusion that there was no difference in the effectiveness of the 2 approaches (based on the P<\/em> value) is flawed as group A\u2019s performance improved by 125% while group B\u2019s performance improved by 633%, reflecting an important effect size that should have been delineated a priori. In this instance, a difference-in-differences approach should have been described.<\/a><\/p>\n\n\n\n How to Avoid It<\/p>\n\n\n\n Randomized clinical trials will avoid a type II error if a sufficient sample size is used.4<\/a><\/sup>However, in education research, which often has smaller study cohorts, one consideration may be to use stratified methods, such as those based on similar baseline experience or performance. For observational studies, adjusting for baseline differences using regression or matching may mitigate this error.<\/a><\/p>\n\n\n\n Common Flaw 3: Inappropriate Use of Validity<\/p>\n\n\n\n What Is the Issue?<\/p>\n\n\n\n The term validity<\/em> is frequently used in relation to assessment tools and simulators. Unfortunately, many researchers in surgical education have not followed the transformative changes in validity concepts that have occurred over time, and they continue using the term inappropriately by stating that they have validated an assessment tool, a curriculum, a simulator, etc. Validity is not currently regarded as an inherent property of a test but instead refers to the specified uses of a test for a particular purpose. Importantly, the evaluation of validity is neither static nor a 1-time event but a continuing process.<\/a><\/p>\n\n\n\n How to Recognize It<\/p>\n\n\n\n One should look for language that implies that a tool, simulator, curriculum, etc has been validated. Evidence for validity can be collected to support the use of a tool, simulator, curriculum, etc for a particular purpose, but the tool itself cannot be validated. Also, look for obsolete terms such as face validity<\/em> or studies that purport to assess the performance of experts and novices on a simulator to demonstrate its construct validity. The difference in surgical skill between these 2 groups is so large that such a comparison is meaningless and should be avoided.<\/a><\/p>\n\n\n\n How to Avoid It<\/p>\n\n\n\n Authors should be aware of Messick and Kane\u2019s frameworks5<\/a><\/sup>,6<\/a><\/sup> and use these to guide studies that investigate validity. When collecting construct validity evidence for a tool or simulator, include several groups of variable skill in your comparison rather than just experts and novices.<\/a><\/p>\n\n\n\n Common Flaw 4: Pre-Post Studies<\/p>\n\n\n\n What Is the Issue?<\/p>\n\n\n\n While pre-post study designs are an acceptable and useful methodology, in the context of education research these are less meaningful. Immediate improvements inevitably seen after an educational intervention should be expected and are not noteworthy; rather, retention of knowledge, attitude, or skills should be determined.<\/a><\/p>\n\n\n\n How to Recognize It<\/p>\n\n\n\n Be aware of the limitations of single-group pre-post study design when studying the impact of an educational intervention. Determine whether a meaningful control group has been included or whether retention data are reported to document the lasting outcomes of the intervention.<\/a><\/p>\n\n\n\n How to Avoid It<\/p>\n\n\n\n The benefit of an educational intervention should be judged not on immediate postintervention improvements but rather by including a retention assessment offered weeks or months after the intervention to detect knowledge, skill, or attitude decay. Another approach is to include a meaningful control group that allows comparisons with the intervention group.<\/a><\/p>\n\n\n\n The aforementioned flaws are not the only ones that occur in the surgical education literature. We focused on these due to the frequency with which they occur and the fact that they are easily correctable with appropriate study design. We believe that preventing such flaws will improve the quality of surgical education research.<\/p>\n","protected":false},"excerpt":{"rendered":" Guide to Statistics and Methods Surgical Educatio […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[13,14,32,23],"tags":[],"_links":{"self":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts\/25341"}],"collection":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=25341"}],"version-history":[{"count":2,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts\/25341\/revisions"}],"predecessor-version":[{"id":25350,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts\/25341\/revisions\/25350"}],"wp:attachment":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=25341"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=25341"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=25341"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Summary<\/h4>\n\n\n\n
\n