Guide to Statistics and Methods 

Surgical Education Research

January 3, 2024

Practical Guide to Surgical Simulation Research

Aimee K. Gardner, Amy H. Kaji, Marja Boermeester

JAMA Surg. Published online January 3, 2024. doi:10.1001/jamasurg.2023.6684

Introduction

Simulation has become a crucial component of surgical education and practice. Not only has simulation been widely integrated into surgical training curricula,^1,2 but the impact of simulation-based education (SBE) on surgeon performance has been widely documented. For example, SBE has been linked to actual performance in the operating room.³ Compelling findings such as these have inspired a significant increase in the quantity of SBE use although its quality remains highly variable. Increased emphasis on study design and optimizing simulation research methodologies are necessary to maximize the impact of simulation—besides practice-based learning—on surgical training and personal learning curve.

Using the Methodology

The goal of surgical simulation research (SSR) is to advance our knowledge and inform practice, and it thereby often falls within 1 of 3 categories: description, justification, or clarification.⁴ Descriptive studies based on observational data may include research that describes a new simulation curriculum, proposes a new method for assessing surgical skills, or the development of an in situ simulation program. These types of studies do not necessarily need to have outcome data or make comparisons and are intended to answer the question “What was done?” Justification studies, in contrast, compare the impact of one intervention to another or to no intervention. Justification studies aim to answer the question “Did it work?” or “Which one worked better?” Clarification studies are the most robust and seek to answer the questions of “How does it work?” and “Why does it work?” These studies incorporate each of the components of the scientific method, starting with observations, incorporating relevant models or theories, making predictions, and testing them. Although experimental designs align themselves very well with this purpose, correlational and qualitative research can be used as well. Because of their robustness and reliance on frameworks and theory, clarification studies have the greatest potential of elucidating the role simulation plays in surgical teaching, learning, and assessment.

Based on the scientific method, SSR uses data from observations (such as those collected from prior descriptive studies), theoretical models, and/or conceptual frameworks. These then can inform the design of the simulation activity or scenario, research methods (eg, intervention, assessment type and frequency), and hypothesis development. Assessment data and relevant outcomes are used to examine the extent to which those hypotheses can be supported or refuted. Ideally, outcomes go beyond attitudes and reactions to measure higher-level behavioral change. Findings may support certain methods and frameworks or uncover ways in which the frameworks or theories need to be modified or improved. Implementing surgery simulation research in this way can help deepen our understanding and advance the art and science of SBE in surgery.

To perform surgical simulation research optimally, several resources are required. Expertise in research design and assessment as well as contextual knowledge is needed. Collaborators, including those who will contribute to study design, implementation, and evaluation, should represent an array of diverse perspectives and backgrounds. Additionally, the critical role of simulation instructors and support staff in simulation research is sometimes overlooked. These individuals are the “boots on the ground” team members who can be valuable research partners by enacting the program and ensuring adherence to research design and standardization. Depending on the scope and modality of the simulation, adequate physical space, technology, and relevant logistics to conduct research are important. Finally, the value and contribution of leadership should not be overlooked. Outward support from department chairs, hospital administrators, and educational leaders can ensure the purpose and value of simulation research is well understood by all.

Being able to study human behavior and learning within a simulation-based setting has advantages. One of the most important advantages to performing research in simulation-based settings over clinical environment is the opportunity for standardization. The control over design and implementation of interventions that exists within a simulation environment is unique. Being able to standardize scenarios and patient presentations involved, while also controlling for other factors, is a substantial benefit over the unpredictability of clinical education. SSR also allows for predictable recruitment of participants and the ability to measure an array of outcomes. For example, data can be extracted from the simulator itself, audio and video recordings, and observer ratings. Even more, SSR can help us more accurately understand and predict all the dimensions of a surgeon by being able to measure procedural skills, communication skills, decision-making, teamwork, leadership, and other key attributes and competencies. Finally, and most importantly, SSR allows for the testing of new technology, protocols, clinical spaces, or practitioner competency with no potential patient harm.

Of course, no type of research is without its potential challenges and pitfalls. Even within surgery simulation research, confounding variables can serve as a threat to the accuracy and generalizability of findings. Randomization of participants may be one approach to mitigate potential confounders. Creating truly authentic environments and contexts for studying human behavior and learning can be challenging. Although predictable recruitment is an advantage, there can be a selection bias and barriers to getting surgeons and learners to participate in a comprehensive simulation study. Depending on the scope and type of surgery simulation research, ethical issues in using trainees as study participants and the high costs limited by lack of funding limit performance of robust studies. Finally, although simulation can allow for rigorous assessment of learning while in the simulation experience, linking these outcomes to actual patient outcomes and surgical practitioners can be complex (Box).

Box. 

Summary

• Surgery simulation research can fall into the categories of description, justification, or clarification.
• A multidisciplinary research team is optimal for robust design, implementation, and impact.
• It is important to state why, in particular, simulation is well suited to achieve the objective and answer the research question (eg, low frequency, high-stakes procedure, need to study in different scenarios) compared to other approaches.
• Surgery simulation research should follow standard reporting guidelines.

Statistical Considerations

• Report the study design and why it is appropriate for the research question—SSR may use nearly all types of study methodologies: randomized clinical trials, observational studies, diagnostic and prognostic studies, qualitative research, quality improvement, economic evaluations, and be included as one of the studies that shape a clinical practice guideline or systematic review. The Enhancing the Quality and Transparency of Health Research (EQUATOR) website⁵ provides a comprehensive site that describes reporting guidelines for various study methodologies. Although each of these study types has published reporting guidelines that should be adhered to, as background for the study, it is important to state why in particular simulation is well suited to achieve the objective and answer the research question. For example, is it a low-frequency, high-stakes event in actual patients that makes in vivo research nearly impossible? Is there a need to compare 2 simulation techniques in terms of differences in fidelity of the model to in vivo performance? Or is it important to test a surgical technique in different scenarios or according to process changes? Perhaps surgical performance via simulation can be used as a prognostic predictive factor of in vivo surgical performance? Simulation may be helpful for economic modeling to determine if a novel technique may be completed in a shorter time period, thereby decreasing operating room costs.
• Frequent flaws to avoid in the analysis—SSR is subject to the common flaws of other types of research. However, unique to SSR may be assuming high fidelity of the simulation model to actual patient care or patient-centered outcomes. As SSR is based on simulation, defined by Merriam-Webster⁶ as “the act or process of simulating, a sham object, counterfeit,” its studies should be viewed as exploratory, rather than a definitive study. Often overlooked is the necessity to adjust for within-person correlations when comparing pre-post performances, as well as not drawing any conclusions about future performance over time if degradation of skills was not assessed. Finally, operator experience or familiarity with the simulation model must be accounted for if 2 cohorts are being compared in terms of performance quality.

Where to Find More Information

Cheng et al⁷ provide reporting guidelines for health care simulation research that can be highly relevant for surgical simulation as well. Cook et al⁴ offer a thoughtful framework for the classification of the purposes of education research and what types of impact each has. Finally, the annual conferences of the Society for Simulation in Europe⁸ and the Association for Surgical Education⁹ are both highly recommended.