Top AI models struggle with medical questions in real-world scenarios, study finds

Overreliance on pattern-matching raises questions about suitability for use in diagnosis

AI systems that boast near-perfect scores on medical licensing exams may not be as clever as they seem.

New research has found that leading large language models (LLMs) falter dramatically when exam questions are subtly reworded, raising doubts about whether these systems are truly capable of clinical reasoning.

The study, led by Stanford University PhD student Suhana Bedi and colleagues, sought to probe whether top AI models truly reason through complex medical scenarios or just mimic answers seen before.

In clinical practice, real patient cases are messy, fragmented, and rarely resemble neatly crafted exam questions. Yet, fewer than 5% of previous AI evaluations used real patient data that reflect this complexity, the researchers highlight.

To test AI reasoning under more realistic conditions, Bedi's team modified 100 questions from the MedQA medical licensing exam. They replaced correct answers in a subset of questions with "None of the other answers" (NOTA), forcing models to rely on actual medical reasoning rather than pattern recognition.

All changes were reviewed by clinicians to ensure medical appropriateness.

Sixty-eight such questions that required selecting the best next step in diagnosis or treatment were used to test six popular AI models, including GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, DeepSeek-R1, o3-mini and Llama 3.3-70B.

Each model was prompted using the chain-of-thought methodology, designed to promote careful analysis rather than guesswork.

The results showed a significant drop in AI performance when familiar answer options were removed.

Models DeepSeek-R1 and o3-mini showed more modest accuracy declines of 9% to 16%, while GPT-4o’s and Claude 3.5 Sonnet's accuracy plunged by 25% and 33%, respectively. Llama 3.3-70B exhibited the largest decline, answering nearly 40% more questions incorrectly under the NOTA condition.

"What surprised us most was the consistency of the performance decline across all models, including advanced reasoning ones," Bedi said.

This uniform drop indicates that despite their advanced architecture, current AI models largely rely on statistical patterns in test formats rather than true understanding of clinical nuances.

The study illustrates this with a clinical vignette involving a newborn baby with an inward-turning foot. Typically this is a symptom of a condition called metatarsus adductus which resolves itself without further intervention.

The original exam answer was "Reassurance," but this was replaced with "None of the other answers" in the modified test, making it harder for the models to guess correctly without real reasoning.

Concerns over AI’s readiness for real-world settings

These findings raise concerns about deploying AI tools in hospitals, where patient presentations are often complex, subtle and unpredictable.

If AI systems struggle when answers are slightly reformulated, they may not be equipped to handle real-world variability and could mislead clinicians.

"These AI models aren't as reliable as their test scores suggest," Bedi said.

While the research involved a relatively small sample of 68 questions, the consistency of the decline across diverse models signals a systemic issue. The authors call for more extensive studies using larger, diverse clinical datasets and evaluation techniques like retrieval-augmented generation or fine-tuning on real medical records.

Looking forward, the team identifies three key priorities: creating evaluation tools that distinguish genuine reasoning from pattern matching, increasing transparency about AI limitations with novel medical problems, and developing models explicitly designed to prioritise reasoning skills.