Guide to Statistics and Methods <\/p>\n\n\n\n
Surgical Education Research<\/p>\n\n\n\n
January 3, 2024<\/p>\n\n\n\n
Introduction<\/p>\n\n\n\n
There has been a significant uptick of interest in curricular development, with more than 2500 publications in the last decade devoted to this area of health care education research. Effective curricular development research embodies pillars of excellence similar to those of curricular development itself, that is, identifying a gap in curriculum to address a health care or learner need, developing a curriculum to address the gap, and evaluating the impact of the curricular intervention. Curricular development research not only is central to one\u2019s ability to disseminate innovative programs but more importantly provides a scaffold by which to assess the quality of curricular content and outcomes of interest for learners (Box<\/a>).<\/a><\/a><\/p>\n\n\n\n Box. <\/p>\n\n\n\n Using the Methodology<\/p>\n\n\n\n Curricular development research can be subdivided into (1) design and dissemination and (2) evaluation, both integrally related. It is highly desirable not only to report the curriculum but also to include measures of training and\/or learner evaluation.<\/a><\/p>\n\n\n\n Design and Dissemination<\/p>\n\n\n\n It is best practice to follow an algorithmic, reproducible, education theory\u2013based approach to curricular design. Following a systematic design workflow will not only improve the learner\u2019s understanding and retention but also will facilitate scientific dissemination in peer-reviewed journals. One such curricular design model is Kern\u2019s model of curricular development.1<\/a><\/sup> Kern\u2019s model integrates a 6-step approach through an iterative process including the following: (1) identifying a problem, (2) performing a needs assessment, (3) establishing clear goals and objectives, (4) determining the effective educational strategies, (5) implementation, and (6) evaluation. By following a systematic approach, the curricular design team ensures that the new program addresses a specific need and commits to a process that is learner centered and mindful of continuous quality improvement.<\/a><\/p>\n\n\n\n While it is tempting to assume that one\u2019s curriculum is novel and innovative, it is prudent to pursue a thorough assessment of the published curriculum landscape. This should not dissuade the educator from creating the said curriculum, but rather it should encourage optimizing or tailoring it to one\u2019s unique needs based on existing, effective, peer-reviewed curricula. MedEdPORTAL<\/em> and myriad education journals serve as important repositories of peer-reviewed curricula.<\/a><\/p>\n\n\n\n Sharing a curriculum through an open-source platform may amplify its impact and reach by facilitating improved learner outcomes beyond organizational borders while also documenting scholarly work and effort.<\/a><\/p>\n\n\n\n Evaluation<\/p>\n\n\n\n Dissemination is only half of the curricular development research journey. Increasingly, it is important and impactful to report the learner and patient outcomes of the curricular intervention, specifically based on the original needs assessment.2<\/a><\/sup> Human memory studies suggest rapid deterioration of knowledge, skills, and attitudes without deliberate practice; consequently, it is helpful to measure long-term as well as short-term outcomes for understanding and retention of knowledge, skills, and attitudes.3<\/a><\/sup> To mitigate such natural deterioration of memory, it may be helpful to replicate the curricular intervention and measure behavioral changes in multiple populations.<\/a><\/p>\n\n\n\n One option is to use Kirkpatrick\u2019s model for learning evaluation to design and report learner outcomes.4<\/a><\/sup> Kirkpatrick\u2019s model of training and learner evaluation progresses from reaction (end-of\u2013curricular session survey about whether the course objectives were met) to learning (whether there has been any change in knowledge, skills, and attitudes due to the intervention) to behavior (whether the learner has incorporated the curricular lesson into daily practice) to results (whether patient outcomes improved because of a curricular intervention).<\/a><\/p>\n\n\n\n End-of-session surveys (Kirkpatrick\u2019s reaction level) are often used to measure satisfaction with the course in Likert scales. Continuing medical education evaluation forms are a good example. Learning assessments (Kirkpatrick\u2019s learning level) may entail a postcurricular standardized examination or objective skills clinical examination to evaluate the impact. A study incorporating Kirkpatrick\u2019s behavior level may include on-the-job observation, self-assessment of incorporating lessons learned into daily practice, or multisource evaluations for a specific skill or attitude. Learning retention is best assessed when some time is allowed to pass before conducting the behavioral assessment. Last, an example of measuring Kirkpatrick\u2019s result level is assessing how the curriculum may have affected an organizational priority, such as improving diversity, inclusion, and equity through a curricular intervention.<\/a><\/p>\n\n\n\n There are multiple options to quantitatively or semiquantitatively measure learner outcomes. For example, one might use existing attitudinal scales or validated technical checklists for procedures when assessing the behavioral level of learning. Similarly, semiquantitative measures incorporating qualitative studies (mixed methods) may reveal findings that may otherwise escape scales and surveys.<\/a><\/p>\n\n\n\n Each level of the model is useful. Progressive measures of learner evaluation encourage education scientists to promote outcomes-based education research through both learner and patient outcomes.<\/a><\/p>\n\n\n\n Common pitfalls of curricular development research involve dedicating significant effort to create a curriculum that already exists or not following an education theory\u2013based design workflow that provides a continuous cycle of curricular improvement. Also, failing to collect appropriate data to analyze outcomes of curricular intervention is a missed opportunity, particularly at the behavior and learning levels of learner evaluation in Kirkpatrick\u2019s model.<\/a><\/p>\n\n\n\n Statistical Considerations<\/p>\n\n\n\n As with other forms of research, generalizability of curricular scholarship is important. Where possible, demographic information of the sample should be reported using descriptive statistics. The participation rate of survey responses (in the context of those who participated) is a helpful measure of engagement and reliability.<\/a><\/p>\n\n\n\n Authors measuring the impact of innovative curricula on learner outcomes may consider inferential statistics (eg, regression models), which are able to adjust for differences in learners\u2019 demographic characteristics or previous experience. Where possible, it is helpful to compare learner outcomes between new and existing curricula.<\/a><\/p>\n\n\n\n Nonparametric analysis should be considered when curricular research outcomes do not satisfy distributional assumptions for their parametric counterparts, as would occur with small sample sizes. Consequently, when comparing learners\u2019 outcomes in pretests or posttests based on Likert scales, descriptive data might include the counts and percentages for each level, in addition to presenting the median (and IQR). For a Likert response, odds ratios and their corresponding 95% CIs can serve as a meaningful measure to assess and report associations among the levels; this response might be reduced to a dichotomy if this is clinically meaningful.<\/a><\/p>\n\n\n\n Depending on the specific design of the assessment, statistical tests should be appropriately tailored. For example, reporting interrater reliability via Cohen \u03ba index (or another measure of interrater reliability) may be important for studies in which multiple assessors are used.5<\/a><\/sup><\/a><\/p>\n\n\n\n While sample size is often a limitation of curricular design research, this can be mitigated with multi-institutional participation or through collecting data over multiple study years.<\/a><\/p>\n\n\n\n Where to Find More Information<\/p>\n\n\n\n Options for publishing curricula include the following:<\/a><\/p>\n\n\n\n Using Kern\u2019s model for curricular development, Green et al6<\/a><\/sup> created a robotic curriculum for residents that mapped specifically to an education theory\u2013based model. In a well-done study incorporating Kirkpatrick\u2019s model of training and learner evaluation, Abdulghani et al7<\/a><\/sup> not only focused on short-term outcomes but also measured the impact of their curriculum on learner behavior in practice.<\/p>\n","protected":false},"excerpt":{"rendered":" Guide to Statistics and Methods Surgical Educatio […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[25,23],"tags":[],"_links":{"self":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts\/25363"}],"collection":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=25363"}],"version-history":[{"count":1,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts\/25363\/revisions"}],"predecessor-version":[{"id":25364,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=\/wp\/v2\/posts\/25363\/revisions\/25364"}],"wp:attachment":[{"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=25363"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=25363"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/csccm.org.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=25363"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Summary<\/h4>\n\n\n\n
\n
\n