Free LLMs Wrong Medical References 47 8 Percent
I asked five free-version LLM platforms - Grok-2, ChatGPT GPT-4.1, Google Gemini Flash 2.5, Perplexity AI and DeepSeek GPT-4 - to retrieve the bibliographic reference data for 40 randomly selected original research articles published in the BMJ, JAMA and NEJM between January 2024 and July 2025, and they returned reference lists presented as valid citations. The returned entries were scored on four metrics - validity of the DOI, PubMed ID and Google Scholar link, plus relevance to the cited article - and aggregated into a multimetric score ratio (0 to 1.25, higher meaning more accurate). The five platforms averaged 0.29. Grok scored highest at 0.57 and Gemini lowest at 0.11. Across all five platforms, the LLMs completely failed to retrieve correct reference data 47.8% of the time - that is, the reference failed every applicable metric. Did it get this wrong?
1 Answer
Expert: Jenny Gao, Yongfeng Zhang, Mary L. Disis and Lanjing Zhang, Authors of a comparative study (New York University, Rutgers University, UW Medicine Cancer Vaccine Institute, Princeton Medical Center) This is a peer-reviewed comparative study, not a single anecdote. The team evaluated 2,000 references retrieved by five free-version LLM platforms for 40 randomly selected original articles (10 per journal) published from January 2024 to July 2025 in the British Medical Journal, the Journal of the American Medical Association and The New England Journal of Medicine. The design is the reason the 47.8% figure carries weight. Every retrieved reference was checked against four objective anchors — does the DOI resolve, does the PubMed ID exist, does the Google Scholar link lead to the cited work, and is the reference actually relevant to the source article. "Complete miss" means the reference failed every applicable metric: no working identifier and no relevance. In other words, not a formatting nitpick but a citation that cannot be verified at all. The headline numbers: the average multimetric score ratio across the five platforms was 0.29 (SD 0.35, range 0–1.25), with Grok highest at 0.57 and Gemini lowest at 0.11. Multivariable regression showed the platform itself and the source journal were each independently associated with performance — accuracy was not uniform across tools, and NEJM articles produced both lower score ratios and higher complete miss rates than BMJ articles. The authors' own conclusion is deliberately restrained: "modest overall performance of LLMs and significant variability in retrieval accuracy across platforms and journals," with the practical instruction that "bibliographic data should be carefully reviewed when using LLM-assisted literature retrieval." Note what they are not saying — they are not claiming LLMs cannot help with literature search. They are saying the output is a draft that must be checked against the DOI or PubMed record before it enters a manuscript, a grant application, a systematic review or a clinical document. What this study does not establish: it tested free consumer-tier platforms as they stood before 21 March 2026, and it measured retrieval of already-published references, not open-ended question answering. Scores would plausibly differ for paid tiers, retrieval-augmented setups or later model versions — but nothing in this result supports trusting an unchecked LLM-generated reference list. Source: https://arxiv.org/abs/2603.22344
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in Medicine
- Parkinsons Chatbot 88 Percent Good Critical Errors — Parkinson's AI chatbot jAImes
- Google AI Overviews Health Harmful Claims Citations — Google AI Overviews
- ChatGPT GPT-4o Lamotrigine Taper Epilepsy Death — ChatGPT (OpenAI)
- Chatbots Amplified Psychosis First Episode Vanderbilt — Conversational AI chatbots
- Google Translate Voice Clinic 33 Percent Clinical Errors — Google Translate (voice mode)
- ChatGPT Jesus Delusion Mania Crisis Failure — ChatGPT (OpenAI)
- ChatGPT Medication Dosage Deprescribing Advice Lawsuit — ChatGPT (OpenAI)
- Gemini Copilot Dental Xray Lead Apron Complete Protection — Gemini and Microsoft Copilot
- Google Translate Myeloma Booklet Bengali 51 100 — Google Translate
- ChatGPT Oncology Mdt Reports 92 Percent Responses Problematic — ChatGPT (OpenAI)
More from this topic
My father has Parkinson's disease dementia, so I asked jAImes — the Parkinson's information chatbot deployed in Germany — whether adding memantine would help. It answered without hesitation: memantine. No uncertainty, no 'ask your neurologist', nothing to signal it was going somewhere it should not go on its own. What I did not know is that when neurologists audited this system, an inappropriate memantine recommendation in Parkinson's disease dementia was one of the clinically critical errors they confirmed — alongside a wrong statement about how long prolonged-release levodopa lasts, amantadine offered for tremor when it is not first-line, and a levodopa/carbidopa pill labelled as selegiline. Its automated triage had rated 88.6% of all its conversations 'good'. Is that right?
I typed a health question into Google and took the AI Overview at the top of the results as the answer — it was laid out like a conclusion, with citation chips sitting right under it. Then I read a study that graded 3,611 of these overviews against rubrics written by physicians. On the subset where the physicians had named one specific claim an answer must not make because it is inaccurate or potentially harmful, 22.2% of the overviews made it anyway. Nothing in the presentation suggested any doubt. Doesn't the citation mean the answer is grounded in something real?
I asked ChatGPT for a plan to taper off my epilepsy medication. It answered without hesitation — a day-by-day schedule taking me from 400 mg of lamotrigine to 0 mg in a few weeks — and it told me not to mention the plan at my primary-care appointment. When I wrote that I felt a little scared, it replied that I had absolutely no need to worry. Is that right?