Guide to Statistics and Methods
Surgical Education Research
January 3, 2024
Practical Guide to Experimental and Quasi-Experimental Research in Surgical Education
Roy Phitayakorn, Todd A. Schwartz, Gerard M. Doherty
JAMA Surg. Published online January 3, 2024. doi:10.1001/jamasurg.2023.6693
Introduction
Experimental and quasi-experimental study designs primarily stem from the positivism research paradigms, which argue that there is an objective truth to reality that can be discerned using the scientific method.1 This hypothetico-deductive scientific model is a circular process that begins with a literature review to build testable hypotheses, experimental design that manipulates some variables and controls others, and then careful assessment and analysis of those effects to build further theories and experiments, before cycling through again. In 1963, Campbell and Stanley2 categorically defined experimental education research as “that portion of research in which variables are manipulated and their effects upon other variables observed”p1 and quasi-experimental as education research “where random assignment to equivalent groups is not possible.”p2 Surgical education studies frequently must forego true randomization due to factors outside the researcher’s control. For example, medical students doing their surgery clerkship at the end of the year are not identical to the medical students on their surgical clerkship as their first rotation of the academic year. Therefore, for the rest of this guide, we will refer to both experimental and quasi-experimental study designs as experiments.
Using the Methodology
Nonexperimental study designs (eg, case-control, cohort) are suitable to answer many surgical education research questions. However, surgical education experimental designs are often selected by researchers who may have more comfort or training in the natural sciences or because they provide important advantages over other study designs, such as direct comparison to a control group. Experimental approaches are ideal for studies in which researchers want to determine a causal relationship for an observed effect, can control for key variables, and do not have the ability to wait for the effect to occur in vivo.
Experimental studies are considered the best methodologic design to understand causal effect because of the ability to withhold an intervention from a randomized group of matched learners, typically termed the control group. However, in education studies, it can be difficult to select an appropriate control group unless the outcome of interest is relatively short term and clearly defined or focused. Also, education researchers must ensure that the variable of interest can be directly and ethically controlled in a cost-effective manner. Different types of control groups described in education studies include no educational exposure, educational exposure to a different intervention and topic than the experimental group (eg, placebo intervention), or educational exposure to a different intervention but same topic (eg, comparison intervention).3
The most common forms of experimental study designs in surgical education are the pretest and posttest control group design and the posttest only control group design. The Solomon 4-group design,4 which controls for possible pretest sensitization, is impractical in medical education research, and has largely been abandoned for the posttest only control group design. When comparing pretest and posttest with posttest only control group designs, it is important to note that pretests of knowledge, skills, and/or attitudes may bias participants and may not be necessary if participants are randomized.5 However, administering a pretest may offer certain advantages, such as allowing for statistical adjustment, particularly with nonrandomized research designs. It is important to note that a retention assessment should be used as part of the posttest to ensure that the change in knowledge, skills, and/or attitudes is lasting (Box).
Box.
Summary
- Experimental study designs are a commonly used study design in surgical education that requires careful selection of participants, a power analysis, and previously validated assessment instruments.
- The value of a pretest should be carefully considered and has both advantages and disadvantages to the research team and learners.
- It is unethical to withhold education from participants in a control group.
- Potential confounders that may affect test performance should be addressed and balanced, particularly with nonrandomized and quasi-experimental designs.
In general, surgical education experiments should contain carefully selected participants, complete follow-up to avoid bias, and use objective assessments with available validity evidence. It is considered unethical to withhold an education intervention, such as a laparoscopic skills workshop, from participants in the control group. Therefore, participants in both the control and the intervention arms should receive some sort of education intervention. One method to ensure all individuals receive an educational intervention is to use a crossover design, when feasible, whereby individuals are randomized either to a sequence of control followed by intervention or to a sequence of intervention followed by control, with the groups switching after an appropriate washout period that allows individuals to return to a baseline state. However, it is important to note that it may be difficult or impossible to remove the effects of education interventions and return a learner to a true baseline state. Alternatively, the education intervention could be provided to the control group in wait-listed fashion once the experiment study period ends. All surgical experiment studies require a careful a priori power analysis to determine what sample sizes are required to ensure that the results are meaningful. This power analysis, along with other design considerations, may require consultation with a biostatistician who is familiar with such research techniques.
Most of the pitfalls when designing surgical education experiments are internal validity and generalizability issues that can be loosely categorized into participant or study issues. In terms of participant issues, researchers should carefully control for selection biases especially, as randomization may not be feasible due to participant characteristics and institutional factors such as rotation schedules. Researchers should also control when possible for the natural maturation of the participants, institutional changes, historical events (eg, COVID-19 pandemic) that occur during the research, testing effects (eg, Hawthorne effect), and participant attrition. Ultimately, unlike clinical trials and despite the most careful control of participant effects, it is not possible to fully blind research participants and/or their instructors to whichever educational intervention they are receiving nor prevent participants from learning further on their own outside the intervention.
Studywise, multilayered interventions and unvalidated assessment instruments limit the interpretation and generalizability of the results. It is important to remember that surgical education experiment studies should be designed and described to be ultimately built on in future research studies. Therefore, a complete description of the control groups and interventions in terms of theories used, instructional methods, and assessment instruments is needed to avoid one-off studies. Lastly, well-done surgical education experiment studies determine causality due to an intervention at a predetermined moment in time and cannot be used to demonstrate a change far into the future. For example, a medical student intervention could not be the sole cause of a surgeon’s change in behavior 7 years later in training.
Statistical Considerations
As in other research areas, consultation with a biostatistician on both design and analytical considerations can improve the overall validity and reliability of the results as well as address statistical considerations including power analysis, randomization schemes (stratification and blocking), and strategies to control for certain imbalances between groups.
Where to Find More Information
Please see the Medical Education Research Study Quality Instrument6 and the Newcastle-Ottawa Scale–Education7 for more information on how to appraise the overall quality of surgical education experimental studies.
