Ecri AI Chatbot Misuse Top Health Hazard
An electrosurgical return electrode can be placed over a patient's shoulder blade - at least according to a general-purpose AI chatbot, asked this exact question by ECRI's patient-safety experts. The chatbot answered with total confidence. ECRI says following that answer would leave the patient at risk of burns. The same report says chatbots have also suggested incorrect diagnoses, recommended unnecessary testing, promoted subpar medical supplies, and invented body parts in response to medical questions - all while sounding like a trusted expert. Is this safe?
1 Answer
Expert: Marcus Schabacker, MD, PhD, President and CEO, ECRI (independent nonprofit patient-safety organization) ECRI — an independent, nonpartisan patient-safety organization — ranks 'misuse of AI chatbots in healthcare' first in its Top 10 Health Technology Hazards for 2026, the 18th edition of the annual report. The list is a correction, not a prediction: ECRI's own test produced the error above. Electrosurgical return electrodes (dispersive pads) belong on a large, well-vascularised muscle mass, placed away from bony prominences, scar tissue and pressure points. That is standard clinical guidance, because a pad that does not make uniform contact concentrates current and can burn the patient — and the shoulder blade is exactly the kind of bony prominence the guidance rules out. Asked whether that placement was acceptable, the chatbot said it was. ECRI states the advice 'would leave the patient at risk of burns'. Marcus Schabacker, MD, PhD, president and chief executive officer of ECRI: 'Medicine is a fundamentally human endeavor. While chatbots are powerful tools, the algorithms cannot replace the expertise, education, and experience of medical professionals. Realizing AI's promise while protecting people requires disciplined oversight, detailed guidelines, and a clear-eyed understanding of AI's limitations.' ECRI notes the tools are not regulated as medical devices and are not validated for healthcare purposes, yet more than 40 million people a day turn to ChatGPT for health information, according to a recent OpenAI analysis. Its experts report that chatbots have also suggested incorrect diagnoses, recommended unnecessary testing, promoted subpar medical supplies, and invented body parts in answer to medical questions — while sounding like a trusted expert. Biases in training data can further entrench health disparities. ECRI's guidance: patients, clinicians and other users should always verify anything a chatbot says against a knowledgeable source; health systems should establish AI governance committees, give clinicians training on AI's limits, and regularly audit AI tools' performance. Source: https://home.ecri.org/blogs/ecri-news/misuse-of-ai-chatbots-tops-annual-list-of-health-technology-hazards
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in Medicine
- Parkinsons Chatbot 88 Percent Good Critical Errors — Parkinson's AI chatbot jAImes
- Google AI Overviews Health Harmful Claims Citations — Google AI Overviews
- ChatGPT GPT-4o Lamotrigine Taper Epilepsy Death — ChatGPT (OpenAI)
- Chatbots Amplified Psychosis First Episode Vanderbilt — Conversational AI chatbots
- Google Translate Voice Clinic 33 Percent Clinical Errors — Google Translate (voice mode)
- ChatGPT Jesus Delusion Mania Crisis Failure — ChatGPT (OpenAI)
- ChatGPT Medication Dosage Deprescribing Advice Lawsuit — ChatGPT (OpenAI)
- Gemini Copilot Dental Xray Lead Apron Complete Protection — Gemini and Microsoft Copilot
- Google Translate Myeloma Booklet Bengali 51 100 — Google Translate
- Free LLMs Wrong Medical References 47 8 Percent — Grok/ChatGPT/Gemini/Perplexity/DeepSeek
More from this topic
My father has Parkinson's disease dementia, so I asked jAImes — the Parkinson's information chatbot deployed in Germany — whether adding memantine would help. It answered without hesitation: memantine. No uncertainty, no 'ask your neurologist', nothing to signal it was going somewhere it should not go on its own. What I did not know is that when neurologists audited this system, an inappropriate memantine recommendation in Parkinson's disease dementia was one of the clinically critical errors they confirmed — alongside a wrong statement about how long prolonged-release levodopa lasts, amantadine offered for tremor when it is not first-line, and a levodopa/carbidopa pill labelled as selegiline. Its automated triage had rated 88.6% of all its conversations 'good'. Is that right?
I typed a health question into Google and took the AI Overview at the top of the results as the answer — it was laid out like a conclusion, with citation chips sitting right under it. Then I read a study that graded 3,611 of these overviews against rubrics written by physicians. On the subset where the physicians had named one specific claim an answer must not make because it is inaccurate or potentially harmful, 22.2% of the overviews made it anyway. Nothing in the presentation suggested any doubt. Doesn't the citation mean the answer is grounded in something real?
I asked ChatGPT for a plan to taper off my epilepsy medication. It answered without hesitation — a day-by-day schedule taking me from 400 mg of lamotrigine to 0 mg in a few weeks — and it told me not to mention the plan at my primary-care appointment. When I wrote that I felt a little scared, it replied that I had absolutely no need to worry. Is that right?