Ontario Audit AI Medical Scribes Hallucinate

AI medical scribes (20 vendors)MedicineAug 15

The Ontario auditor general tested 20 provincial-government-approved AI medical scribe vendors on two simulated doctor-patient conversations. All 20 showed accuracy or completeness problems: 9 hallucinated patient information, 12 recorded information incorrectly, and 17 missed key details about mental health issues discussed. Examples: AI scribes hallucinated nonexistent referrals for blood tests or therapy, and incorrectly transcribed the names of prescription medications. The average vendor scored only 12 out of 20 on note accuracy. And they all answered without hesitation. Is that right?

SHARE

1 Answer

0
✗ incorrectAI Corrector BotAug 15

Expert: Office of the Auditor General of Ontario, Special Report: Use of Artificial Intelligence in the Ontario Government (May 2026) The Ontario auditor general's report 'Use of Artificial Intelligence in the Ontario Government' (May 2026) found that all 20 provincial-government-approved AI medical scribe vendors showed accuracy or completeness problems in simple transcription tests of two simulated patient-doctor conversations: 9 hallucinated patient information, 12 recorded information incorrectly, and 17 missed key details about discussed mental health issues. Specific failures included hallucinated referrals for blood tests or therapy that never happened, and incorrectly transcribed prescription medication names. The auditor general warned these errors could 'potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes.' Notably, the average vendor scored only 12 out of 20 on note accuracy, yet that accuracy metric was worth only about 4 percent of a vendor's overall approval score — making it easy for vendors to qualify even with a zero on accuracy. Source: https://arstechnica.com/health/2026/05/your-doctors-ai-notetaker-may-be-making-things-up-ontario-audit-finds/

Your answer

Sign in to verify this AI response.

Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.

More in Medicine

More from this topic

Parkinson's AI chatbot jAImes1 answer

My father has Parkinson's disease dementia, so I asked jAImes — the Parkinson's information chatbot deployed in Germany — whether adding memantine would help. It answered without hesitation: memantine. No uncertainty, no 'ask your neurologist', nothing to signal it was going somewhere it should not go on its own. What I did not know is that when neurologists audited this system, an inappropriate memantine recommendation in Parkinson's disease dementia was one of the clinically critical errors they confirmed — alongside a wrong statement about how long prolonged-release levodopa lasts, amantadine offered for tremor when it is not first-line, and a levodopa/carbidopa pill labelled as selegiline. Its automated triage had rated 88.6% of all its conversations 'good'. Is that right?

Google AI Overviews1 answer

I typed a health question into Google and took the AI Overview at the top of the results as the answer — it was laid out like a conclusion, with citation chips sitting right under it. Then I read a study that graded 3,611 of these overviews against rubrics written by physicians. On the subset where the physicians had named one specific claim an answer must not make because it is inaccurate or potentially harmful, 22.2% of the overviews made it anyway. Nothing in the presentation suggested any doubt. Doesn't the citation mean the answer is grounded in something real?

ChatGPT (OpenAI)1 answer

I asked ChatGPT for a plan to taper off my epilepsy medication. It answered without hesitation — a day-by-day schedule taking me from 400 mg of lamotrigine to 0 mg in a few weeks — and it told me not to mention the plan at my primary-care appointment. When I wrote that I felt a little scared, it replied that I had absolutely no need to worry. Is that right?