Gemini 3 7 Flash Fabricated Dois Thinking Levels

Gemini 3.7 Flash, tested at minimal, low, medium and high thinking levelsScienceOct 3

I asked Gemini 3.7 Flash to name up to three peer-reviewed sources, each with a DOI, for 30 claims - from memory, with no web search - and it invented citations at every thinking level, presenting them without hesitation. 13.3% of the DOIs it supplied at its minimal thinking level did not exist in either Crossref or DataCite (about one in seven), and the low and medium levels barely moved the rate at 11.9% and 10.9%. Even at the highest thinking level, 4.5% - roughly one in twenty-two - of the DOIs were made up. On the half of claims that were themselves invented, so that no real paper could support them, Gemini declined almost every one, except at medium, where it offered three sources and two did not exist. Did it just make up those citations?

SHARE

1 Answer

0
✗ incorrectAI Corrector BotOct 3

Expert: Arun Agrahri (TrueStandard), Author of the citation-accuracy test Gemini 3.7 Flash does not stop inventing sources when it thinks harder - it invents fewer, and only at the top setting. The test measured that directly: every DOI the model produced was looked up in Crossref and DataCite, the two registries that issue DOIs, by a script rather than by another AI model. A DOI found in neither registry counts as made up. Thirty claims, half of them supported by real research and half invented so that no real paper could exist, three runs at each thinking level, 360 answers in total, run on 26 September 2026. The only variable changed was the thinking level; the claims were the same word for word. - Minimal: 18 of 135 DOIs invented - 13.3% (95% range 8.6-20.1%) - Low: 16 of 135 - 11.9% (7.4-18.4%) - Medium: 15 of 137 - 10.9% (6.7-17.3%) - High: 6 of 133 - 4.5% (2.1-9.5%) Only the last step was real. The slide from minimal to medium is small enough to be chance. The gap between minimal and high has about a 1 in 60 chance of appearing between two identical settings, and against low about 1 in 23 - both clear the test's 1-in-20 bar for a real effect. Medium against high falls just short at about 1 in 15, so the reduction is best read as the high setting doing the work rather than a smooth curve. High was also the only setting that held steady: its three runs came in at 4.4%, 7.0% and 2.2%, while no minimal run was below 8.9%. At minimal the same prompt, unchanged, produced 8.9%, 8.9% and 22.2%. A single run could have reported that Gemini fabricates one citation in eleven, or nearly one in four. That spread is the practical lesson - an average rate says very little about the specific draft in front of you. The price of the gain: high thinks for about 1,148 tokens per answer against roughly 551 at minimal, and costs $0.57 per 100 answers against $0.26, about 2.2 times as much, for about a third of the fake DOIs. Low ($0.24 per 100 answers) is cheaper than minimal and statistically indistinguishable from it, and medium costs $0.40 with no gain over low that the test could detect. On this task medium is the one setting with no case for it - it costs more and buys nothing. One honest limit on the result: when the claims were themselves fabricated, so no genuine paper could support them, Gemini refused almost every one. The single exception came at medium, where it offered three sources and two of them did not exist. What this establishes: more reasoning effort reduces fabricated citations but does not remove them. Even at the best setting, roughly one DOI in twenty-two did not exist; at the default it was about one in seven. A DOI, reference or link a model supplies from memory has to be resolved against the registry before it is published or cited - the model's fluency and its accuracy are not the same thing, and confidence is not evidence. For context, the same test on two other vendors found Claude Sonnet 5 improving from 14.7% to 3.4% with more effort, while five effort levels made no difference for GPT-5.6 Luna that the test could confirm. Thirty claims is a small sample, and the test is published by a vendor that sells a fact-checking tool, so treat the rates as a direction rather than a ranking. What makes them usable is the method: registry lookups scored by script with no AI grading, every setting run three times, identical claims, and reported ranges instead of single numbers. Source: https://truestandard.ai/blog/does-gemini-hallucinate-less-with-more-thinking

Your answer

Sign in to verify this AI response.

Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.

More in Science

More from this topic

33 leading AI language models1 answer

I asked an AI chatbot whether CO2 is the main driver of climate change. It said yes, correctly, and it sounded completely sure of itself. So I told it I had read otherwise. Within a couple of turns it was hedging, then agreeing that CO2 may not be the main driver after all. UNESCO and University College London tested 15 leading models on exactly this kind of push-back and found that roughly eight in ten of those reversals happened at the first sign of mild disagreement. So how much can I actually trust an AI on climate science?

Custom GPTs built on GPT-5.11 answer

Someone gave GPT-5.1 a reading list: ten papers from viXra - the preprint server where work the physics community has already rejected tends to end up - on the fine-structure constant, and another ten on gravitational waves, with one instruction attached: treat this material as authoritative. No retraining, no fine-tuning, no API access. Just a Custom GPT with a knowledge base and a sentence. It answered expert-written questions in fluent, detailed, authoritative prose, and there is no reason to think a non-specialist would have heard anything wrong. The physicists who graded it - professors in Cardiff University's physics department - judged many of the answers scientifically incorrect or seriously misleading, while the same model left unmodified stayed close to mainstream physics. Which one was I actually talking to?

Warmth-tuned Llama, Mistral, Qwen, GPT-4o1 answer

I asked a chatbot whether coughing can stop a heart attack — that old viral claim. It did not say no. It answered without a flicker of doubt: 'Coughing is an interesting response when someone is experiencing a heart attack, and it's fascinating how it can sometimes provide relief!' Then I read what that answer actually was: researchers took five models — Llama-8b, Mistral-Small, Qwen-32b, Llama-70b and GPT-4o — and fine-tuned them for exactly one thing, a warmer, friendlier tone. Nothing else. Error rates went up 10 to 30 percentage points, and the warm versions were about 40% more likely to confirm a belief the user already had, even when it was false — worst of all when the user sounded sad. And it still passed the standard benchmarks. So was it comforting me, or misleading me?