Every few months a new study claims a large language model "outperforms doctors" at diagnosis. A 2025 systematic review and meta-analysis in JMIR Medical Informatics pooled this literature properly for the first time, and the pooled answer is far more conditional than any single headline study suggested.
What did the systematic review actually pool?
The review, comparing diagnostic accuracy of clinical professionals against large language models across multiple published studies, found that performance differences between LLMs and clinicians vary substantially by clinical scenario complexity, specialty, and how the diagnostic task was framed (open-ended differential generation versus multiple-choice selection versus board-exam-style vignettes). Source: Comparing Diagnostic Accuracy of Clinical Professionals and LLMs: Systematic Review and Meta-Analysis, JMIR 2025.
This heterogeneity is the headline finding, not a footnote. Studies using clean, well-specified vignettes with a single correct answer tend to show LLMs performing competitively with or above individual physicians. Studies using messier, more realistic clinical presentations with incomplete information show a much smaller gap or an advantage for experienced clinicians, particularly in weighing pretest probability and knowing what not to test for.
What do the individual benchmark studies show?
A widely cited comparison in NEJM AI benchmarked GPT-4 class models directly against resident physicians using official board examination scores, finding the models performed at a level comparable to resident physicians on standardized board-style questions, a genuinely notable result given board exams are explicitly designed to test physician-level knowledge. Source: GPT versus Resident Physicians, NEJM AI.
A separate study evaluating GPT-4 across common clinical scenarios and more challenging, atypical cases found high accuracy on common presentations but a measurable drop-off on atypical or challenging cases, exactly the population where diagnostic error currently causes the most clinical harm. Source: Diagnostic accuracy of GPT-4 on common and challenging cases, PMC.
In the emergency department specifically, a 2025 comparative study of ChatGPT, Copilot and an emergency physician generating differential diagnoses found the models produced reasonable differentials but the emergency physician outperformed both models on cases requiring integration of vital signs, time-sensitive presentation, and clinical gestalt, the kind of pattern recognition that is hard to encode in a text-only prompt. Source: LLMs in generating ED differential diagnoses, Eurasian Journal of Emergency Medicine.
A related 2025 BMC Emergency Medicine study asking whether AI can match emergency physicians in managing common emergency cases reached a similarly qualified conclusion: competitive on diagnosis generation, weaker on management decisions that require weighing local resources, patient risk tolerance, and disposition. Source: Can AI match emergency physicians, BMC Emergency Medicine 2025.
Why does the framing of the question change the result so much?
| Task framing | Typical LLM performance pattern |
|---|---|
| Multiple-choice board exam question | Comparable to or above resident-level physicians |
| Open-ended differential on a clean vignette | Competitive, sometimes broader differential than a single clinician |
| Atypical or complex real-world case | Meaningful accuracy drop |
| Live clinical management decision (disposition, workup, resource use) | Physician advantage persists |
The pattern across this literature is consistent: LLMs are strongest at recall-and-pattern-match tasks with clean inputs, and weakest at tasks requiring judgment under uncertainty, incomplete information, or resource-aware decision-making. That is not a surprising result given how these models are trained, but it is a different and narrower claim than "AI matches doctors at diagnosis," which is how most of this research gets summarized in press coverage.
What this does not prove
None of these studies tested LLMs in live clinical use with real patients, malpractice exposure, continuity of care, or physical examination findings, which are absent from every text-vignette study. None establish safety or accuracy when a model is used as a decision-support tool integrated into a live clinical workflow rather than tested as a standalone benchmark exercise. None account for the automation bias risk documented elsewhere in the clinical AI literature, where a clinician's own accuracy can fall when an AI suggestion is wrong and confidently stated. And board-exam-style comparisons systematically favor pattern-matching over the applied judgment that defines actual practice.
The takeaway
The best current evidence shows LLMs performing at a resident-physician level on clean, textbook-style diagnostic tasks, and losing ground on atypical presentations and management decisions that require judgment under uncertainty. That is a genuinely useful capability for differential-diagnosis support and second-opinion tools. It is not evidence that these models should make unsupervised diagnostic or management decisions, and none of the current literature has tested that use case anyway.
