EDUCATION AND CLINICAL PRACTICE: ORIGINAL RESEARCH

Quality of Human Expert vs Large Language Model-Generated Multiple-Choice Questions in the Field of Mechanical Ventilation

Sami Safadi, Roxana Amirahmadi, Burton W. Lee, et al

Chest 2025; 168: 1425-1432

Abstract

Background

Although mechanical ventilation (MV) is a critical competency in critical care training, standardized methods for assessing MV-related knowledge are lacking. Traditional multiple-choice question (MCQ) development is resource intensive, and prior studies have suggested that generative AI tools could streamline question creation. However, the quality of AI-generated MCQs remains unclear.

Research Question

Are MCQs generated by ChatGPT noninferior to human expert (HE)-created questions in terms of quality and relevance for MV education?

Study Design and Methods

Three key MV topics were selected: Equation of Motion and Ohm’s Law, Tau and Auto-PEEP, and Oxygenation. Fifteen learning objectives were used to generate 15 AI-written MCQs via a standardized prompt with ChatGPT-o1 (preview model; made available September 12, 2024). A group of 31 faculty experts, all of whom regularly teach MV, evaluated both AI- and HE-generated MCQs. Each MCQ was assessed based on its alignment with learning objectives, accuracy of chosen answer, clarity of the question stem, plausibility of distractor options, and difficulty level. The faculty members were blinded to the provenance of the MCQ questions. The noninferiority margin was predefined as 15% of the total possible score (–3.45).

Results

AI-generated MCQs were statistically noninferior to the HE-written MCQs (95% upper CI, [–1.15, ∞]). In additions, respondents were unable to reliably differentiate AI-generated MCQs from HE-written MCQs (P = .32).

Table 1. Topics and Learning Objectives

Topic	Learning Objective
Equation of Motion and Ohm’s Law	•Use a clinical vignette to calculate static compliance•Use a clinical vignette to apply the Equation of Motion to a patient with high airway pressures•Use a clinical vignette to identify changes in lung static compliance and alveolar pressure•Use a clinical vignette to demonstrate how the Equation of Motion explains the relationships among alveolar pressure, peak pressure, and respiratory muscular pressure (Pmusc)•Understand the components of the Equation of Motion in a square wave volume control setting
Tau and Auto-PEEP	•Understand the factors that determine how much volume remains in the lung at the end of expiration•Use a clinical vignette to identify the relationship between compliance, resistance, and risk of auto-PEEP•Recognize the most effective change a clinician can make to acutely treat auto-PEEP•Understand that the expiratory time constant represents the time needed to exhale until 37% of tidal volume remains in the lungs•Understand that a patient needs at least 3 expiratory time constants to fully exhale tidal volume
Oxygenation	•Understand how to calculate the stress index in a patient who is mechanically ventilated•Understand oxygen toxicity in a patient who is mechanically ventilated•Understand the impact of higher PEEP on oxygenation and survival in patients with ARDS•Understand the appropriate application of stress index monitoring for adjusting PEEP in a patient with severe ARDS. Present a clinical vignette•Identify a clinical strategy to improve oxygenation and reduce mortality in ARDS

Table 2. Raters’ Ability to Identify Source of Multiple-Choice Question

Group	Guess: Artificial Intelligence	Guess: Human Expert	Total
Group: Artificial Intelligence	256 (55.1%)	209 (45.0%)	465
Group: Human Expert	241 (51.8%)	224 (48.2%)	465
Total	497	433	930

Interpretation

Our results suggest that AI-generated MCQs using ChatGPT-o1 are comparable in quality to those written by HEs. Given the time and resource-intensive nature of human MCQ development, AI-assisted question generation may serve as an efficient and scalable alternative for medical education assessment, even in highly specialized domains such as MV.