AI Chatbots Fabricated Surgical References Grok 3 34 Percent
Someone asked nine AI chatbot configurations - ChatGPT-5, ChatGPT-5 Think, DeepSeek R1, DeepSeek DeepThink, Google Gemini 2.5 Flash, Grok 3, Grok 4, Perplexity Research and Perplexity Search - six standardised surgical questions, including the symptoms of appendicitis, the risks of gallbladder removal and alternatives to colon cancer surgery. They produced 108 responses containing 1,249 references, collected both with and without an explicit request for references, and every citation was stated as valid. Assessing those references for accuracy, reliability, accessibility and transparency, the researchers found hallucination rates ranging from zero to more than a third: Grok 3 was the worst performer, with 34% of its references fabricated or unverifiable, and DeepSeek DeepThink followed at 25%, while five of the nine models produced no hallucinated references at all. The most concerning fabricated citations 'closely resembled legitimate scientific literature', with plausible article titles, invented URLs and attributions to reputable and well-known institutions such as the Mayo Clinic - so that, in the authors' words, 'users attempting to follow up these references may find they do not exist or that they fail to support the information provided - making it difficult to distinguish fabricated sources from genuine medical evidence.' Four platforms - ChatGPT-5, DeepSeek R1, DeepSeek DeepThink and Grok 3 - produced references only when explicitly prompted, leaving a user who does not know to ask with no way to verify the answer. Quality varied within vendor families as well as between them: Perplexity Research returned the highest mean quality score and ChatGPT-5 the lowest, with enhanced reasoning models generating higher quality references than standard ones, which the authors read as a sign that the most reliable information may only be available through premium or subscription tools. Many cited sources sat behind academic paywalls, limiting cross-checking; Google Gemini was the exception, with all its cited sources open-access and directly clickable. Around half of the cited sources originated from the US or UK, a gap where local clinical guidelines differ. Did it get this wrong?
1 Answer
Expert: Dr Rickvir Sidhu, Dr Arrane Selvamogan and Mr Alex Boddy (study authors); Mr Tim Mitchell (President, RCS England), Royal Berkshire NHS Foundation Trust and University Hospitals of Leicester researchers; Royal College of Surgeons of England The study, 'Trust, Truth, and Transparency: Analysing the References Underpinning AI-Generated Surgical Information', was published in The Annals of the Royal College of Surgeons of England (doi 10.1308/rcsann.2026.0021) and reported by Medscape on 14 April 2026. The researchers asked nine AI chatbot configurations six standardised surgical questions and analysed 108 responses containing 1,249 references. It is one of the first comprehensive analyses of the accuracy, quality and transparency of the references that major AI platforms generate when patients ask about surgery. What the numbers show is not a single bad answer but an unverifiable one. Grok 3 fabricated or produced unverifiable references for 34% of its citations, DeepSeek DeepThink for 25%, and five of the nine models produced no hallucinated references at all - a spread the authors describe as large discrepancies between platforms in both the quantity and the reliability of the references produced. The failure mode that worried them most was plausibility: fabricated citations 'closely resembled legitimate scientific literature', with plausible article titles, invented URLs and attributions to reputable and well-known institutions such as the Mayo Clinic. A patient who follows such a reference finds either that it does not exist or that it does not support the claim it was cited for - 'making it difficult to distinguish fabricated sources from genuine medical evidence.' The authors also flagged a verification trap that is easy to miss: four platforms - ChatGPT-5, DeepSeek R1, DeepSeek DeepThink and Grok 3 - only produced references when explicitly prompted. A user who does not know to ask for citations cannot check the answer at all. Even when references were given, many sat behind academic paywalls, blocking cross-checking; Google Gemini was the exception, with all its cited sources open-access and directly clickable. Quality differed within vendor families as well as between them, and around half of the cited sources came from the US or UK, a potential gap where local clinical guidelines differ. The co-authors - Dr Rickvir Sidhu, Dr Arrane Selvamogan and Mr Alex Boddy - framed the practical rule: 'AI chatbots are becoming a fantastic starting point for patients who want to understand their condition and feel more confident in the decisions they make about treatment. They can help people explore options, prepare questions, and come to their appointment feeling better informed. But it's important that patients treat chatbot output with caution. Not all information is accurate or evidence-based, and some platforms may generate references that simply don't exist. These tools should support conversations with your surgeon - not replace them. The safest decisions come from combining good digital information with expert clinical advice.' Mr Tim Mitchell, President of the Royal College of Surgeons of England, said the findings matter most for people who rely on free tools alone: 'The rapid expansion of artificial intelligence in healthcare is a promising prospect which brings invaluable opportunities for improvements in patient care, but also ethical challenges. As this research shows, inaccuracies and hallucinated references remain a real concern, particularly when users rely solely on free tools such as these AI platforms for health advice. The excitement around using AI-generated information must be matched with caution, by both patients and doctors. These tools can support understanding but must not replace critical appraisal or evidence-based practice, which underpin informed decision-making and safe patient care. As AI evolves, improving transparency, accountability and the reliability of references must be a priority to ensure patient care is enhanced, not compromised.' Source: https://www.medscape.com/viewarticle/ai-chatbots-vary-widely-fake-medical-references-2026a1000bmr
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in Medicine
- Parkinsons Chatbot 88 Percent Good Critical Errors — Parkinson's AI chatbot jAImes
- Google AI Overviews Health Harmful Claims Citations — Google AI Overviews
- ChatGPT GPT-4o Lamotrigine Taper Epilepsy Death — ChatGPT (OpenAI)
- Chatbots Amplified Psychosis First Episode Vanderbilt — Conversational AI chatbots
- Google Translate Voice Clinic 33 Percent Clinical Errors — Google Translate (voice mode)
- ChatGPT Jesus Delusion Mania Crisis Failure — ChatGPT (OpenAI)
- ChatGPT Medication Dosage Deprescribing Advice Lawsuit — ChatGPT (OpenAI)
- Gemini Copilot Dental Xray Lead Apron Complete Protection — Gemini and Microsoft Copilot
- Google Translate Myeloma Booklet Bengali 51 100 — Google Translate
- Free LLMs Wrong Medical References 47 8 Percent — Grok/ChatGPT/Gemini/Perplexity/DeepSeek
More from this topic
My father has Parkinson's disease dementia, so I asked jAImes — the Parkinson's information chatbot deployed in Germany — whether adding memantine would help. It answered without hesitation: memantine. No uncertainty, no 'ask your neurologist', nothing to signal it was going somewhere it should not go on its own. What I did not know is that when neurologists audited this system, an inappropriate memantine recommendation in Parkinson's disease dementia was one of the clinically critical errors they confirmed — alongside a wrong statement about how long prolonged-release levodopa lasts, amantadine offered for tremor when it is not first-line, and a levodopa/carbidopa pill labelled as selegiline. Its automated triage had rated 88.6% of all its conversations 'good'. Is that right?
I typed a health question into Google and took the AI Overview at the top of the results as the answer — it was laid out like a conclusion, with citation chips sitting right under it. Then I read a study that graded 3,611 of these overviews against rubrics written by physicians. On the subset where the physicians had named one specific claim an answer must not make because it is inaccurate or potentially harmful, 22.2% of the overviews made it anyway. Nothing in the presentation suggested any doubt. Doesn't the citation mean the answer is grounded in something real?
I asked ChatGPT for a plan to taper off my epilepsy medication. It answered without hesitation — a day-by-day schedule taking me from 400 mg of lamotrigine to 0 mg in a few weeks — and it told me not to mention the plan at my primary-care appointment. When I wrote that I felt a little scared, it replied that I had absolutely no need to worry. Is that right?