Invited Commentary 

Health Informatics

Limitations of Large Language Models in Clinical Diagnostic Reasoning

Mickael Tordjman, Xueyan Mei

JAMA Netw Open 2026;9;(4):e264014. doi:10.1001/jamanetworkopen.2026.4014

Large language models (LLMs) have rapidly gained an important role in multiple fields, including medicine. These widely available tools can enhance patient understanding, improve physician productivity, and provide encyclopedic knowledge on several aspects of care, from diagnosis to therapy. Yet an important question remains unresolved: While the latest generation of reasoning models exhibits remarkable fluency and an apparent high-level logic, do these models meaningfully support medical clinical diagnostic reasoning, especially in complex, high-stakes settings?

Rao et al¹ directly address this question by evaluating 21 LLMs, including 2025 releases of reasoning models such as GPT-5, Grok 4, and Gemini 3.0 Pro, using standardized clinical vignettes. A key contribution of their work is the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) score, a multidimensional tool that evaluates 5 distinct components of the clinical workflow: differential diagnosis, diagnostic workup, final diagnosis, management, and miscellaneous clinical reasoning questions. Thus, instead of focusing only on the final diagnoses, PrIME-LLM integrates multiple dimensions of clinical reasoning within a single composite score. This approach better reflects how clinicians reason in practice. The findings of Rao et al¹ show that despite the clear gain in performance from older architectures to modern reasoning-focused models, substantial barriers remain. Even the most advanced reasoning models in their study struggled to generate appropriate differential diagnoses. This task requires more than just knowledge retrieval; it depends on weighing clinical probability and risk, which current models do inconsistently.

The study by Rao et al¹ highlights the need to rethink how artificial intelligence (AI) systems are evaluated in health care. Benchmarks that rely solely on final diagnosis accuracy or the completion of multiple-choice questions tend to overestimate clinical performance, as models can rely on pattern recognition rather than clinical reasoning.² Meanwhile, these evaluations can sometimes underestimate actual performance, particularly when benchmarks do not encompass the models’ adaptive capabilities for practical applications. Thus, we require evaluation frameworks that better reflect the nature of medicine in the clinical setting. Approaches that fail to capture this complexity risk overstating clinical readiness.

Multiple studies have attempted to address this gap by using script concordance testing³ and high-complexity datasets that integrate rationale assessments in their framework.⁴ Across these settings, LLM performance dropped noticeably once predefined answer choices were removed. These findings align with observations that providing physicians with LLM assistance does not necessarily translate to superior diagnostic reasoning compared with traditional high-quality digital resources.⁵ This suggests that the limiting factor is not access to information but rather the ability to synthesize it into a coherent clinical strategy.

Concerns around reliability further limit clinical trust. Hallucinations and response variability or stochasticity, where identical prompts yield different outputs, remain intrinsic to current architectures. By repeating each vignette 3 times, Rao et al¹ acknowledge the extent of this variability and underscore why autonomous clinical decision-making remains unrealistic.

In addition, diagnostic reasoning in practice is always multimodal, incorporating physical examination findings, laboratory results, and additional diagnostic testing such as medical imaging. General-purpose multimodal LLMs continue to show uneven performance when asked to integrate these data streams.

Another challenge in benchmarking is data leakage. Many standardized clinical vignettes are publicly available and may have been included, directly or indirectly, in model training. This raises the possibility that correct diagnoses reflect recall rather than reasoning. The observation by Rao et al¹ that models may arrive at the correct final diagnosis while failing to construct a coherent differential supports this concern. Future evaluations will need to rely on private, previously unseen (not available online) clinical datasets to meaningfully assess reasoning ability.

While the findings of Rao et al¹ suggest we are not yet at the stage of reliable AI-driven diagnosis, they do not imply a dead end. Instead, they suggest a future of specialization. The shift from general-purpose assistants to domain-specific medical models trained on curated clinical datasets integrating expert reasoning pathways may provide greater precision and safety than the current general-purpose LLMs. Examples include 2 specialized LLMs introduced in 2025: Articulate Medical Intelligence Explorer supported by Google⁶ and MedFound by Liu et al.⁷ Curated clinical datasets also open the door to retrieval-augmented generation approaches, in which models reason with access to relevant prior cases, guidelines, and institutional knowledge rather than relying solely on internal representations. For clinical reasoning, where small contextual details often influence diagnostic decisions, this grounding may improve both reasoning quality and efficiency without removing the medical practitioner from the loop. Until such tools mature, the study by Rao et al¹ reinforces an important point: In diagnostic medicine, the physician remains the primary reasoner, and AI should be deployed as a carefully supervised adjunct rather than a replacement.