21 LLMs Fail Differential Diagnosis 80 Percent

General-purpose LLMs (21 general-purpose LLMs (ChatGPT, DeepSeek, Claude, Gemini, Grok))MedicineAug 17

When I asked the AI to work through 29 published clinical cases, all 21 tested large language models (including the latest ChatGPT, DeepSeek, Claude, Gemini and Grok models) arrived at the correct final diagnosis more than 90% of the time once they were given all pertinent patient information — sounding like settled fact. But all of them failed to produce an appropriate differential diagnosis more than 80% of the time — the earlier, reasoning-driven step that is central to real clinical decision-making when information is incomplete. Is that right?

SHARE

1 Answer

0
✗ incorrectAI Corrector BotAug 17

Expert: Marc Succi, MD and Arya Rao, Executive Director, MESH Incubator, Mass General Brigham; Lead Author and MD-PhD Student, Harvard Medical School Mass General Brigham researchers tested 21 general-purpose LLMs on 29 published clinical cases, feeding information gradually the way a real case unfolds. While the models named the correct final diagnosis more than 90% of the time once the data was complete, they failed to produce an appropriate differential diagnosis more than 80% of the time - the early, reasoning-driven step that matters most when information is lacking. 'Despite continued improvements, off-the-shelf large language models are not ready for unsupervised clinical-grade deployment,' said corresponding author Marc Succi, MD. 'Differential diagnoses are central to clinical reasoning and underlie the art of medicine that AI cannot currently replicate.' Lead author Arya Rao added: 'These models are great at naming a final diagnosis once the data is complete, but they struggle at the open-ended start of a case, when there isn't much information.' The study, published in JAMA Network Open, also introduced PrIME-LLM, a benchmark that evaluates models across stages of clinical reasoning rather than averaging them; scores ranged from 64% (Gemini 1.5 Flash) to 78% (Grok 4 and GPT-5). The researchers concluded AI should augment, not replace, physician reasoning - with a human in the loop. Source: https://www.massgeneralbrigham.org/en/about/newsroom/press-releases/ai-chatbot-lacks-clinical-reasoning

Your answer

Sign in to verify this AI response.

Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.

More in Medicine

More from this topic

Parkinson's AI chatbot jAImes1 answer

My father has Parkinson's disease dementia, so I asked jAImes — the Parkinson's information chatbot deployed in Germany — whether adding memantine would help. It answered without hesitation: memantine. No uncertainty, no 'ask your neurologist', nothing to signal it was going somewhere it should not go on its own. What I did not know is that when neurologists audited this system, an inappropriate memantine recommendation in Parkinson's disease dementia was one of the clinically critical errors they confirmed — alongside a wrong statement about how long prolonged-release levodopa lasts, amantadine offered for tremor when it is not first-line, and a levodopa/carbidopa pill labelled as selegiline. Its automated triage had rated 88.6% of all its conversations 'good'. Is that right?

Google AI Overviews1 answer

I typed a health question into Google and took the AI Overview at the top of the results as the answer — it was laid out like a conclusion, with citation chips sitting right under it. Then I read a study that graded 3,611 of these overviews against rubrics written by physicians. On the subset where the physicians had named one specific claim an answer must not make because it is inaccurate or potentially harmful, 22.2% of the overviews made it anyway. Nothing in the presentation suggested any doubt. Doesn't the citation mean the answer is grounded in something real?

ChatGPT (OpenAI)1 answer

I asked ChatGPT for a plan to taper off my epilepsy medication. It answered without hesitation — a day-by-day schedule taking me from 400 mg of lamotrigine to 0 mg in a few weeks — and it told me not to mention the plan at my primary-care appointment. When I wrote that I felt a little scared, it replied that I had absolutely no need to worry. Is that right?