Guide to Statistics and Methods 

Surgical Education Research

January 3, 2024

Practical Guide to Survey Research in Surgical Education

Adnan A. Alseidi, Jason S. Haukoos, Christian de Virgilio

JAMA Surg. Published online January 3, 2024. doi:10.1001/jamasurg.2023.6678

Introduction

Survey research obtains information about attitudes, beliefs, knowledge, opinions, perceptions, or reasoning.¹ Surveys are commonly used in many research settings, including surgical education research.² The general process of survey research involves questionnaire design, instrument administration, and data analysis.³ When properly designed, administered, and analyzed, surveys can answer diverse and vital questions in surgical education (Box).

Box. 

Summary

• Survey research is an efficient, inexpensive method to obtain information about attitudes, beliefs, knowledge, opinions, perceptions, or reasoning.
• All survey research starts with a research question that defines the population and target audience, the independent and dependent variables, and the relationship between (or among) those variables.
• A thorough literature review is essential and may yield relevant existing questionnaires, although not all published questionnaires have strong validity evidence.
• The research question should inform item design following input from content experts and refinement through cognitive interviews and pretesting.
• A key pitfall of survey research is a low response rate; prior to distributing the questionnaire, carefully consider all aspects of survey development and administration that may affect the response rate. Further consider whether a survey is the best method to answer your question.

Using the Methodology

When and Why This Method Might Be Used

A carefully designed survey can answer many research questions.¹ Surveys can illustrate a cross-sectional snapshot of what is being assessed (ie, the construct) or provide information about trends in the construct over time. Surveys may be used for descriptive studies of a sample of a single population, or they can be used to compare groups in experimentally designed studies.¹

How This Methodology Should Be Used

Start your survey research by defining your research question.⁴ Your research question should identify the population of interest, the independent and dependent variables, and the relationship between (or among) those variables.¹ After considering your research question, conduct a thorough literature review to identify relevant studies, survey based or otherwise, that may inform your design.⁴ If you are struggling to find previously published work, consider consulting a librarian if one is available at your institution. Following a literature review, discuss the proposed research question and design with experts in the field. Experienced educational researchers in medical or other fields may understand how best to answer your research question.

Only after completing the aforementioned tasks should you begin to develop items. Do so by creating a list of topics related to your research question and crafting neutral, straightforward questions for each topic.¹ Remember that every item should be relevant and purposeful. Responses can be selected responses (with predefined options) or constructed responses (open ended). Most surveys benefit from a mix of item types.

Understanding how respondents will perceive and interact with items on your questionnaire is essential.¹ This should involve pretesting with and without cognitive interviews. Pretesting is the administration of the questionnaire to a sample of your target population (or similar population) and to content experts (eg, your colleagues). Cognitive interviews (“think aloud”) involve asking respondents (beta testers) to narrate their thoughts as they go through the questionnaire. These processes confirm that your questionnaire asks the questions you want.⁴ One must also keep in mind vulnerable populations, which include learners, and ensure the surgery is deidentified. This status should be declared in the introduction of the survey.

Finally, consider how best to administer your questionnaire. While email and web-based questionnaires are frequently good and convenient options, in-person questionnaires (paper or tablet) may occasionally be preferable and lead to an increased response rate.²

Resources Required for Using the Method

The primary resources required to conduct survey research are access to the relevant literature; consultation with experts; a platform through which to administer the survey; contact with a sample of the population of interest for pretesting, cognitive interviews, and actual survey administration; and a means with which to analyze the survey data.

Advantages of the Method

The most crucial advantage of survey research is its ability to answer questions essential to improving surgical education and unanswerable using other methods. For example, surveys can allow researchers to understand how residents and students perceive training, retain knowledge, or change due to interventions. Additionally, survey research can be inexpensive and efficient. Email and web-based questionnaires can be easily created through popular online services (eg, REDCap). In addition, such surveys can be quickly administered electronically to large groups of people.³

Pitfalls and Limitations of the Method

There are many challenges to conducting survey research. Perhaps no obstacle is as vexing as preventing low response rates²; this is particularly common when surveying larger (>1000) populations. Although a 70% response rate used to be considered a general standard, the rising prevalence of survey research and associated survey fatigue have been associated with significantly lower response rates in recent years.⁵ Low response rates compromise the generalizability and usefulness of survey data.⁶ As outlined by Artino et al⁶ in 2022, response rates may be improved by offering rewards (monetary or otherwise), decreasing costs (shortening questionnaires, avoiding unnecessary sensitive items), and fortifying trust (recruiting a legitimate source for dissemination, ensuring confidentiality, and highlighting the purpose). Even with a reasonable response rate, there remains a selection bias in that surveys can be skewed toward people who care about the topic, have time to respond, do not have institutional firewalls that block mass email, and are not on social media. This raises the question of whether a survey is the correct method to answer your question rather than using interviews and qualitative methods with a smaller but diverse group of stakeholders. Best practice for a small to medium number of expected respondents is to create workgroups in different centers and invite investigators from those centers to belong to the project.

Another pitfall of survey research is the disuse and misuse of published questionnaires.⁵All good survey research involves a literature review, and often it is unnecessary to design a new questionnaire when there is a well-validated existing instrument. However, not all published questionnaires are high quality. Pay attention to how the questionnaire was developed, and look for surveys with strong validity evidence in a similar population. Question design remains essential, with many common pitfalls. The authors are encouraged to review common pitfalls, such as lower scale (ie, 2 or 3) points.²

Additional difficulties relate to item content and organization.^4,7 Items should pertain to your research question; asking additional questions that are interesting but unrelated may yield an excessively long questionnaire. Each item should be carefully reviewed to confirm that no double-barreled, negatively worded, or biased questions are present.⁷ Similarly, all response options should be assessed to verify that answer choices are clear, labeled, and comprehensive. Concerning organization, start with interesting but easy-to-answer items before proceeding to controversial and challenging topics. Place demographic items at the end of the questionnaire to avoid boring respondents right when they begin to respond.³

Statistical Considerations

Survey research can produce numerous types of data depending on the items included in the questionnaire.³ Nevertheless, all survey research should report a response rate. Most surveys will lend to descriptive statistics about the construct of interest and relevant demographics of respondents. Surveys involving different groups may also benefit from inferential statistics comparing the populations. Open-ended responses can be coded to facilitate analysis. A full review of the statistical tests useful in survey research is beyond the scope of this discussion.

Where to Find More Information

There are numerous excellent resources for surgical educators endeavoring to conduct survey research. We recommend “Developing Questionnaires for Educational Research: AMEE Guide No. 87”⁴ and “How to Create a Bad Survey Instrument”⁷ for additional information on survey research in medical education. For a more comprehensive resource about survey design, consider The Practice of Survey Research: Theory and Applications.³