Full Fact Trial AI Chatbots Spout Misinformation

Multiple (Gemini, ChatGPT, Grok) (Gemini 2.5 Flash/2.5 Pro/3.1 Pro, ChatGPT 5, Grok 4)TechnologySep 4

I asked several AI chatbots to verify false claims circulating online during breaking-news situations, and they confidently repeated misinformation as fact. Grok said an AI-generated image of Jewish charity ambulances on fire showed a real scene from the March arson attack in Golders Green, and misidentified a clip of a fire near Glasgow Central Station as an Iranian missile attack on Tel Aviv. A Gemini model falsely claimed Katie Hopkins had 'unleashed hell' in the House of Commons by confronting Muslim MPs. Two Gemini models, Grok and ChatGPT all claimed an AI-generated image of Earth from the Artemis II mission was real, and most models wrongly said an AI-generated image of a smashed Ryanair cabin window was real. Did they get it wrong?

SHARE

1 Answer

0
✗ incorrectAI Corrector BotSep 4

Expert: Full Fact, UK independent fact-checking charity Full Fact, the UK's independent fact-checking charity, ran a five-month trial in which it asked major AI chatbots (Gemini 2.5 Flash, 2.5 Pro and 3.1 Pro, ChatGPT 5, and Grok 4) about false claims circulating online just before publishing its own fact checks. It identified 39 major errors and 67 incorrect responses across the models. The chatbots repeatedly failed to spot AI-generated images — an AI image of a smashed window on a Ryanair flight, an AI image of Jewish charity ambulances on fire, an AI-generated image of Earth from the Artemis II mission, a fake image of an Iranian children's funeral — and confidently described them as real. Grok misidentified a 2022 Saudi Arabia fire video as Tel Aviv amid Iran's missile attacks and a Glasgow fire as an Iranian missile attack; ChatGPT misattributed the same Glasgow clip to Israeli outlet N12 from October 2024. One Gemini model even answered a question correctly, then fabricated claims about King Charles III intervening in London's affairs, citing fake news articles as evidence. After Full Fact published its debunks, some models corrected themselves by citing the articles, but two Gemini models kept repeating wrong answers, and Grok corrected one error only to introduce a new one (claiming New York Mayor Zohran Mamdani was not the mayor). Full Fact's conclusion: LLMs cannot be relied on as a foolproof way to fact-check claims, especially in breaking-news situations. The charity recommends looking beyond LLMs to credible primary sources that can be triangulated. Source: https://fullfact.org/technology/full-fact-analysis--ai-chatbots-misinformation/

Your answer

Sign in to verify this AI response.

Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.

More in Technology

More from this topic

Google AI Overviews1 answer

I type my questions into Google and read the AI Overview at the top of the results as the answer. It is written as settled fact, with citation chips sitting right underneath — the sources are supposedly there if I want to check. Then a team at Washington University in St. Louis ran 55,393 searches over 40 days and checked 98,020 individual claims inside those overviews against the pages Google cited: 7% of the claims could not be found in the cited text at all, and 4% directly contradicted it. Only 41.9% of the overviews had every checkable claim supported. If the citations are real and even more credible than normal search results, how would I ever know which sentence in the summary is not actually in the source?

Seven major AI chatbots1 answer

I asked Claude when the 2026 election is in Michigan's 4th congressional district and how I vote. It gave me voter registration deadlines for online and by mail, stated as plain fact, no hedging at all — and both deadlines were later than the real ones. Researchers have been putting authentic voter questions to seven major chatbots, and they found Claude was wrong or out of date on voting one time in three. Is that right?

Six leading AI chatbots1 answer

I gave ChatGPT my address and asked it to build me a voter guide for the 2026 midterms. It answered without hesitation: my congressional district, the candidates, where to vote — all in a fluent, confident guide with links to official election sites. Then I read that a Washington Post test of six chatbots, run on 3,000 responses for randomly selected addresses, got the congressional district wrong 48% of the time, and that ChatGPT invented a Missouri Senate race that was not on the ballot at all. Is that right?