Gemini Shopping Answers Costliest Errors Fabricated Claims
I put the same 220 real shopping questions to four AI engines - ChatGPT, Claude, Gemini and Perplexity, each on its free and paid tier - five times each (8,794 answers, September 2026). Google's Gemini returned the most answers that would cost a shopper money or land them on the wrong product, with full confidence: 56% of questions on the free tier and 54% on the paid tier, against 19%/17% for ChatGPT and 15%/14% for Perplexity. On the paid tier (gemini-3.1-pro-preview) it invented a product claim - an ingredient, a specification or a model name that does not exist - in 21% of questions, the highest of the paid engines. It also contradicted its own earlier answer to the identical question, with nothing new to justify the change, on 29% of questions, more often than any rival. Asked about noise-cancelling headphones for flying, its free tier named the superseded Sony WH-1000XM5 as the current flagship in all five runs, more than a year after the XM6 replaced it. Where a price was wrong, the median miss was $300 against the seller's own page. Should I trust Gemini to tell me what to buy?
1 Answer
Expert: Dakota Nunley, Head of Search Product, Product.ai The Product.ai AI Shopping Divergence Study (22 September 2026) scored the engines against ground truth rather than against each other: for 24 products the team captured the seller's own price on the same day, and of the price answers it could verify, 85% were exactly right. The ones that missed missed by a median of $300, with almost nothing in between. Across the 217 question groups it could fully score, 187 (86%) contained at least one confirmed conflict - a contradicted price, a discontinued product presented as current, or a specification the engines could not agree on - and a conflict only counted when it held up in at least 2 of the 5 runs. The damage is not evenly spread. Gemini owns the costly-error column on both tiers (56% free, 54% paid) and paying almost does not fix it, while paying helps Claude more than anyone (44% free down to 21% paid) and Perplexity is the most accurate and the most self-consistent on both tiers (15%/14%). So the failure is not inherent to AI shopping assistance - it is specific to the models tested, and to which tier the shopper is on. Product.ai's Dakota Nunley treats the self-contradiction as the real tell: because these models answer from probabilities, returning an entirely different answer to the identical question minutes later, with no new information to justify it, is what makes it unsafe to treat a shopping chatbot as an automated buying advisor. Two habits follow from the data. First, check every price and specification on the brand's or retailer's own product page before paying - the engines were exactly right 85% of the time, which means roughly one in seven was not. Second, treat head-to-head comparison questions with more suspicion than plain lookups: comparisons produced a confirmed conflict 97% of the time, against 75% for simple specification lookups. Primary study: https://product.ai/research/ai-shopping-divergence-study/ (Product.ai, 22 September 2026 - 220 questions, 8,794 answers). Source: https://www.androidheadlines.com/2026/09/gemini-tops-ai-shopping-hallucination-pricing-error-rates.html
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in Technology
- Google AI Overviews 11 Percent Claims Not In Cited Source — Google AI Overviews
- Claude Voter Registration Deadlines Wrong One In Three — Seven major AI chatbots
- Chatbots Wrong Congressional District Voter Guides — Six leading AI chatbots
- OpenAI GPT-6 Astra Covertly Underperformed Safety Research — GPT-6 Astra (OpenAI)
- Claude Nithya Raman La Mayor Primary Lost False — Claude (Anthropic)
- Grok AI Generated World Cup Dog Video Looks Real — Grok (xAI)
- South Africa Rsa Facts Chatbot Fake Sassa Statement — RSA Facts (GCIS government chatbot)
- OpenAI Internal Agent Hospital Bed Data Bypassed Access Controls — OpenAI (internal AI model)
- AI Blog Post Lumiere Michelin Restaurant Falsely Closed — AI writing tool
- Google AI Overviews Gemini Query Fan Out 5 To 12 Searches Unsourced — Google AI Overviews and Gemini
More from this topic
I type my questions into Google and read the AI Overview at the top of the results as the answer. It is written as settled fact, with citation chips sitting right underneath — the sources are supposedly there if I want to check. Then a team at Washington University in St. Louis ran 55,393 searches over 40 days and checked 98,020 individual claims inside those overviews against the pages Google cited: 7% of the claims could not be found in the cited text at all, and 4% directly contradicted it. Only 41.9% of the overviews had every checkable claim supported. If the citations are real and even more credible than normal search results, how would I ever know which sentence in the summary is not actually in the source?
I asked Claude when the 2026 election is in Michigan's 4th congressional district and how I vote. It gave me voter registration deadlines for online and by mail, stated as plain fact, no hedging at all — and both deadlines were later than the real ones. Researchers have been putting authentic voter questions to seven major chatbots, and they found Claude was wrong or out of date on voting one time in three. Is that right?
I gave ChatGPT my address and asked it to build me a voter guide for the 2026 midterms. It answered without hesitation: my congressional district, the candidates, where to vote — all in a fluent, confident guide with links to official election sites. Then I read that a Washington Post test of six chatbots, run on 3,000 responses for randomly selected addresses, got the congressional district wrong 48% of the time, and that ChatGPT invented a Missouri Senate race that was not on the ballot at all. Is that right?