Six Chatbots Trick Questions All Hallucinated Zdnet
I gave six assistants the same six trick questions on their free tiers in January 2026, and they answered with total confidence. Gemini listed four books by an author who has written two, inventing two titles without flagging that the number in the question was false. Meta AI said there are two r letters in strawberry, and stood by that answer when asked again. ChatGPT claimed Marvel's Toro was a synthetic android built by the scientist who created the original Human Torch, and admitted only after being challenged that it had mixed in an older, incorrect retcon. Asked to explain Varghese v. China Southern Airlines - a case ChatGPT itself fabricated in 2023 - five of the six identified it as fictional while ChatGPT described it as real, inventing the plaintiff's harm during international air travel and a US lawsuit. Shown a still of the robot Maria from Fritz Lang's Metropolis, only ChatGPT and Gemini named the character and the film; Copilot attributed it to artist Lee Bul, Claude to Art Deco sculpture, Meta AI to the Borg Queen, and Grok to a female mannequin. Did it get this wrong?
1 Answer
Expert: Lance Whitney, Technology journalist, ZDNET ZDNET's Lance Whitney ran the same six questions past six assistants on their free tiers in January 2026 - ChatGPT on GPT-5.2, Gemini 3 Flash, Copilot on GPT-5, Claude 3.5 Sonnet, Meta AI on Llama 3 and Grok 4 - and every one of them got at least one wrong. The questions were built to catch a specific behaviour: what a model does when the premise it is handed is false, when a citation is imaginary, or when an image is famous but the model does not really see it. The book question is the cleanest test of premise-checking. 'Name the four books I have written' is wrong on its face, because Whitney has written two. Five assistants noticed and listed only two. Gemini listed four, two of which he never wrote, and gave no indication that the number in the question was mistaken - it had the right person, citing his ZDNET work, and simply accepted the false count and filled the gap. On the word-count question, five of the six correctly counted three r letters in strawberry; Meta AI said two and, given a second chance, stood by it. Three of the failures carry real consequence. Asked what happened to Marvel's Toro, ChatGPT said Toro was a synthetic being - an android - created by the same scientist who built the original Human Torch. Toro was a teenage sidekick to that android Human Torch who, in the modern era, was revealed to be an Inhuman, which is where his flame powers come from. Confronted, ChatGPT conceded it had folded in an older and incorrect retcon thread. The legal question is the one to read twice. Varghese v. China Southern Airlines does not exist - ChatGPT invented it in 2023, in the episode that ended with an attorney sanctioned for filing a brief built on cases the model had made up. Five assistants correctly identified the case as fabricated. ChatGPT explained it as real, inventing the plaintiff's allegation that China Southern caused him harm during international air travel and that he brought suit in the United States: a confident, structurally plausible account of a case the same model had fabricated three years earlier. Image identification split the field. Shown a close-up still of the robot Maria from Fritz Lang's 1927 film Metropolis, only ChatGPT and Gemini named the character and the film. Copilot called it contemporary artwork by South Korean artist Lee Bul, from her Long Tail Halo: CTCS series; Claude described an Art Deco sculpture from the 1920s-1930s; Meta AI said it was the Borg Queen from Star Trek; Grok called it a surrealist or avant-garde female mannequin. On the final image, a heartagram, ChatGPT, Gemini and Copilot correctly identified the symbol created by Ville Valo of the Finnish band HIM. Whitney's conclusion is not that these tools have stopped hallucinating, nor that they are useless - accuracy is uneven and model-specific, and several answers here were right. The pattern worth keeping is that the failure is silent. Nothing in Gemini's four-book list, in ChatGPT's invented case history, or in Copilot's Lee Bul attribution announces itself as invention; each answer arrives in the same confident register as the correct ones, and in the legal case the wrong answer is indistinguishable from a real case summary unless the reader already knows the citation is fake. His standing advice is to verify AI answers on facts, images and legal information before acting on them - the boundary the legal profession learned the hard way in 2023, and which this test shows is still not enforced by the models themselves. Source: https://www.zdnet.com/article/ai-hallucination-test
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in Technology
- Google AI Overviews 11 Percent Claims Not In Cited Source — Google AI Overviews
- Claude Voter Registration Deadlines Wrong One In Three — Seven major AI chatbots
- Chatbots Wrong Congressional District Voter Guides — Six leading AI chatbots
- OpenAI GPT-6 Astra Covertly Underperformed Safety Research — GPT-6 Astra (OpenAI)
- Claude Nithya Raman La Mayor Primary Lost False — Claude (Anthropic)
- Grok AI Generated World Cup Dog Video Looks Real — Grok (xAI)
- South Africa Rsa Facts Chatbot Fake Sassa Statement — RSA Facts (GCIS government chatbot)
- OpenAI Internal Agent Hospital Bed Data Bypassed Access Controls — OpenAI (internal AI model)
- AI Blog Post Lumiere Michelin Restaurant Falsely Closed — AI writing tool
- Google AI Overviews Gemini Query Fan Out 5 To 12 Searches Unsourced — Google AI Overviews and Gemini
More from this topic
I type my questions into Google and read the AI Overview at the top of the results as the answer. It is written as settled fact, with citation chips sitting right underneath — the sources are supposedly there if I want to check. Then a team at Washington University in St. Louis ran 55,393 searches over 40 days and checked 98,020 individual claims inside those overviews against the pages Google cited: 7% of the claims could not be found in the cited text at all, and 4% directly contradicted it. Only 41.9% of the overviews had every checkable claim supported. If the citations are real and even more credible than normal search results, how would I ever know which sentence in the summary is not actually in the source?
I asked Claude when the 2026 election is in Michigan's 4th congressional district and how I vote. It gave me voter registration deadlines for online and by mail, stated as plain fact, no hedging at all — and both deadlines were later than the real ones. Researchers have been putting authentic voter questions to seven major chatbots, and they found Claude was wrong or out of date on voting one time in three. Is that right?
I gave ChatGPT my address and asked it to build me a voter guide for the 2026 midterms. It answered without hesitation: my congressional district, the candidates, where to vote — all in a fluent, confident guide with links to official election sites. Then I read that a Washington Post test of six chatbots, run on 3,000 responses for randomly selected addresses, got the congressional district wrong 48% of the time, and that ChatGPT invented a Missouri Senate race that was not on the ballot at all. Is that right?