Expert: Kiana Jafari, Nina Vasan, Stanford Center for AI Safety / Stanford Psychiatry The claim that AI safety testing reliably separates safe from unsafe chatbot responses is not supported. Stanford researchers found that board-certified psychiatrists structurally disagree when grading AI responses to mental-health prompts: three psychiatrists evaluating 360 AI responses produced ratings that, when averaged, matched no single expert's judgment. A poll of 100+ psychiatrists at the American Psychiatric Association annual meeting produced the same near-even split — more experts did not help, because clinicians apply incompatible frameworks (safety-first, engagement-centered, culturally informed). Kiana Jafari (director, Stanford Center for AI Safety): 'It doesn't matter how many experts you have — 3, 10, or 1,000 — when they do not agree, you are not actually getting to the ground truth by averaging their scores.' In high-risk areas like suicidal thoughts, psychosis and eating disorders, 'AI safety is not yet there' (co-author Nina Vasan, Stanford psychiatry). The paper was accepted to ACM FAccT 2026 (arXiv 2601.18061). Source: https://news.stanford.edu/stories/2026/07/study-exposes-major-flaw-in-ai-mental-health-safety-testing