Guide to Statistics and Methods
Surgical Education Research
January 3, 2024
Practical Guide to Curricular Development Research
Kevin Y. Pei, Todd A. Schwartz, Marja A. Boermeester
JAMA Surg. Published online January 3, 2024. doi:10.1001/jamasurg.2023.6699
Introduction
There has been a significant uptick of interest in curricular development, with more than 2500 publications in the last decade devoted to this area of health care education research. Effective curricular development research embodies pillars of excellence similar to those of curricular development itself, that is, identifying a gap in curriculum to address a health care or learner need, developing a curriculum to address the gap, and evaluating the impact of the curricular intervention. Curricular development research not only is central to one’s ability to disseminate innovative programs but more importantly provides a scaffold by which to assess the quality of curricular content and outcomes of interest for learners (Box).
Box.
Summary
- Perform a thorough literature search for existing and evaluated published curricula, which can serve as an evidence base for the proposed curricular innovation.
- Follow an education theory–based curricular development model.
- Map the curricular design process, and report this cycle in scholarly work.
- Measure and report learner or patient outcomes related to the curricular intervention using Kirkpatrick’s model of training and learner evaluation.
- Be open to disseminating and sharing the curriculum to maximize its impact.
Using the Methodology
Curricular development research can be subdivided into (1) design and dissemination and (2) evaluation, both integrally related. It is highly desirable not only to report the curriculum but also to include measures of training and/or learner evaluation.
Design and Dissemination
It is best practice to follow an algorithmic, reproducible, education theory–based approach to curricular design. Following a systematic design workflow will not only improve the learner’s understanding and retention but also will facilitate scientific dissemination in peer-reviewed journals. One such curricular design model is Kern’s model of curricular development.1 Kern’s model integrates a 6-step approach through an iterative process including the following: (1) identifying a problem, (2) performing a needs assessment, (3) establishing clear goals and objectives, (4) determining the effective educational strategies, (5) implementation, and (6) evaluation. By following a systematic approach, the curricular design team ensures that the new program addresses a specific need and commits to a process that is learner centered and mindful of continuous quality improvement.
While it is tempting to assume that one’s curriculum is novel and innovative, it is prudent to pursue a thorough assessment of the published curriculum landscape. This should not dissuade the educator from creating the said curriculum, but rather it should encourage optimizing or tailoring it to one’s unique needs based on existing, effective, peer-reviewed curricula. MedEdPORTAL and myriad education journals serve as important repositories of peer-reviewed curricula.
Sharing a curriculum through an open-source platform may amplify its impact and reach by facilitating improved learner outcomes beyond organizational borders while also documenting scholarly work and effort.
Evaluation
Dissemination is only half of the curricular development research journey. Increasingly, it is important and impactful to report the learner and patient outcomes of the curricular intervention, specifically based on the original needs assessment.2 Human memory studies suggest rapid deterioration of knowledge, skills, and attitudes without deliberate practice; consequently, it is helpful to measure long-term as well as short-term outcomes for understanding and retention of knowledge, skills, and attitudes.3 To mitigate such natural deterioration of memory, it may be helpful to replicate the curricular intervention and measure behavioral changes in multiple populations.
One option is to use Kirkpatrick’s model for learning evaluation to design and report learner outcomes.4 Kirkpatrick’s model of training and learner evaluation progresses from reaction (end-of–curricular session survey about whether the course objectives were met) to learning (whether there has been any change in knowledge, skills, and attitudes due to the intervention) to behavior (whether the learner has incorporated the curricular lesson into daily practice) to results (whether patient outcomes improved because of a curricular intervention).
End-of-session surveys (Kirkpatrick’s reaction level) are often used to measure satisfaction with the course in Likert scales. Continuing medical education evaluation forms are a good example. Learning assessments (Kirkpatrick’s learning level) may entail a postcurricular standardized examination or objective skills clinical examination to evaluate the impact. A study incorporating Kirkpatrick’s behavior level may include on-the-job observation, self-assessment of incorporating lessons learned into daily practice, or multisource evaluations for a specific skill or attitude. Learning retention is best assessed when some time is allowed to pass before conducting the behavioral assessment. Last, an example of measuring Kirkpatrick’s result level is assessing how the curriculum may have affected an organizational priority, such as improving diversity, inclusion, and equity through a curricular intervention.
There are multiple options to quantitatively or semiquantitatively measure learner outcomes. For example, one might use existing attitudinal scales or validated technical checklists for procedures when assessing the behavioral level of learning. Similarly, semiquantitative measures incorporating qualitative studies (mixed methods) may reveal findings that may otherwise escape scales and surveys.
Each level of the model is useful. Progressive measures of learner evaluation encourage education scientists to promote outcomes-based education research through both learner and patient outcomes.
Common pitfalls of curricular development research involve dedicating significant effort to create a curriculum that already exists or not following an education theory–based design workflow that provides a continuous cycle of curricular improvement. Also, failing to collect appropriate data to analyze outcomes of curricular intervention is a missed opportunity, particularly at the behavior and learning levels of learner evaluation in Kirkpatrick’s model.
Statistical Considerations
As with other forms of research, generalizability of curricular scholarship is important. Where possible, demographic information of the sample should be reported using descriptive statistics. The participation rate of survey responses (in the context of those who participated) is a helpful measure of engagement and reliability.
Authors measuring the impact of innovative curricula on learner outcomes may consider inferential statistics (eg, regression models), which are able to adjust for differences in learners’ demographic characteristics or previous experience. Where possible, it is helpful to compare learner outcomes between new and existing curricula.
Nonparametric analysis should be considered when curricular research outcomes do not satisfy distributional assumptions for their parametric counterparts, as would occur with small sample sizes. Consequently, when comparing learners’ outcomes in pretests or posttests based on Likert scales, descriptive data might include the counts and percentages for each level, in addition to presenting the median (and IQR). For a Likert response, odds ratios and their corresponding 95% CIs can serve as a meaningful measure to assess and report associations among the levels; this response might be reduced to a dichotomy if this is clinically meaningful.
Depending on the specific design of the assessment, statistical tests should be appropriately tailored. For example, reporting interrater reliability via Cohen κ index (or another measure of interrater reliability) may be important for studies in which multiple assessors are used.5
While sample size is often a limitation of curricular design research, this can be mitigated with multi-institutional participation or through collecting data over multiple study years.
Where to Find More Information
Options for publishing curricula include the following:
- MedEdPORTAL
- The Cutting Edge of Surgical Education
- Academic Medicine
- Medical Teacher
Using Kern’s model for curricular development, Green et al6 created a robotic curriculum for residents that mapped specifically to an education theory–based model. In a well-done study incorporating Kirkpatrick’s model of training and learner evaluation, Abdulghani et al7 not only focused on short-term outcomes but also measured the impact of their curriculum on learner behavior in practice.
