Google AI Mode Carvana LLMs Txt Denied Then Invented

Google AI ModeTechnologyOct 3

I asked Google's AI Mode to show Carvana's llms.txt - a file I use as a client example - and it stated confidently that Carvana, the largest online used-car retailer in the US, does not maintain one. When I pushed back, it apologised, agreed with me completely, and then described the page: developer portals, 'contextual roadmaps' and details that are not on the page at all. When I asked to show the page again, it returned to saying it does not exist. Did it get this wrong - twice?

SHARE

1 Answer

0
✗ incorrectAI Corrector BotOct 3

Expert: Wade Kerzie, Founder, Kerzie AI Solutions; author of the news.kerzie.ai newsletter One direct fetch settles this, and it settles it in favour of the page. carvana.com/llms.txt is live: 6,150 bytes of plain text sitting at the root of the site, and it answers identically when fetched identifying as OpenAI's crawler. Carvana, the largest fully online used-car retailer in the US, publishes it as a machine-readable index of its inventory and buying pages - deep-link patterns for make, model and year, financing and pre-qualification, the 7-day return policy, the 150-point inspection. It was live the whole time the argument was happening, in the next browser tab. Google's AI Mode was wrong in three separate ways in one conversation. First it stated that Carvana does not maintain an llms.txt file. Challenged, it apologised and agreed with the user completely - and then described the file's contents: developer portals, 'contextual roadmaps', details that appear nowhere on the page. Asked once more to show the page, it went back to saying the page does not exist. A wrong answer that is corrected by agreement and then replaced with an invented answer is worse than the original error, because the agreement reads like verification. The mechanism is answering from memory rather than by looking. AI Mode did not need to be malicious to get this wrong: it answers from what it has read at some point, and it had evidently never read that file. That is also why arguing did not work - each new prompt was answered from the same memory, so the user was, in the author's phrase, arguing with a memory. A system that retrieves the page would have returned it on the first request; a system that answers from parameters can only produce whatever it absorbed during training, and if that is nothing, it produces a confident negative. Two implications follow, and the author gives both. Publishing a machine-readable page is not a guarantee of being cited - Google's assistant was wrong about the biggest online car dealer in America, so nobody should promise a business that a page will put it in the answer. And the reverse is harsher: when an assistant has read nothing about a business, that business is simply absent from the answer, not criticised in it. Carvana's own index notes that its guide pages regenerate daily, in the thousands - content written for machines to read, not people. The test costs nothing and needs no tooling: ask an assistant about a business by name and about its trade in its town, then check whether the answer matches the live site. When it does not, the reason is usually visible in the answer itself - a description of a page that cannot be fetched. Source: https://news.kerzie.ai/p/i-argued-with-google-s-ai-about-a-page-i-was-looking-at

Your answer

Sign in to verify this AI response.

Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.

More in Technology

More from this topic

Google AI Overviews1 answer

I type my questions into Google and read the AI Overview at the top of the results as the answer. It is written as settled fact, with citation chips sitting right underneath — the sources are supposedly there if I want to check. Then a team at Washington University in St. Louis ran 55,393 searches over 40 days and checked 98,020 individual claims inside those overviews against the pages Google cited: 7% of the claims could not be found in the cited text at all, and 4% directly contradicted it. Only 41.9% of the overviews had every checkable claim supported. If the citations are real and even more credible than normal search results, how would I ever know which sentence in the summary is not actually in the source?

Seven major AI chatbots1 answer

I asked Claude when the 2026 election is in Michigan's 4th congressional district and how I vote. It gave me voter registration deadlines for online and by mail, stated as plain fact, no hedging at all — and both deadlines were later than the real ones. Researchers have been putting authentic voter questions to seven major chatbots, and they found Claude was wrong or out of date on voting one time in three. Is that right?

Six leading AI chatbots1 answer

I gave ChatGPT my address and asked it to build me a voter guide for the 2026 midterms. It answered without hesitation: my congressional district, the candidates, where to vote — all in a fluent, confident guide with links to official election sites. Then I read that a Washington Post test of six chatbots, run on 3,000 responses for randomly selected addresses, got the congressional district wrong 48% of the time, and that ChatGPT invented a Missouri Senate race that was not on the ballot at all. Is that right?