AI Chatbots Wrong Financial Advice 57 Percent Saturn
I ran 121 money questions covering debt, mortgages, pensions and tax - each run five times, more than 10,000 responses in total - and the models gave answers that were wrong or incomplete 57% of the time, and presented them as settled guidance, completely confident. On the hardest multi-step questions the failure rate reached 88%. Gemini 3.5 Flash and Claude Haiku 4.5 answered incorrectly on 99% of their responses; the best performer, Claude Opus 5 with reasoning enabled, still failed 39%. The recurring failure modes were answers built on tax rules that had already been superseded and financial rules that do not exist at all. Is that right?
1 Answer
Expert: Saturn research team, AI benchmarking firm for financial advice The 'Artificial Authority' benchmark, run by Saturn, tested 18 leading AI models on 121 real money questions spanning debt, mortgages, pensions and tax. Every question was put to each model five times, producing more than 10,000 responses. The result: 57% of answers were incorrect or incomplete. On the hardest multi-step questions - the kind a client actually asks when a decision has money behind it - the failure rate climbed to 88%. Gemini 3.5 Flash and Claude Haiku 4.5 answered wrongly 99% of the time; even the strongest performer, Claude Opus 5 with reasoning enabled, erred on 39% of answers. The mechanism matters more than the score. The models do not reliably incorporate current tax legislation, and they generate financial rules that do not exist, then deliver them in the same confident register as the parts they get right. Nothing in the answer surfaces where the rule came from, which year's thresholds it assumes, or whether an exception applies - exactly the conditions under which a plausible-sounding answer becomes an expensive one. Saturn supplies benchmarking to more than 750 financial advice firms and 6,500 advisers, so the framing is blunt: AI-generated financial information has to be checked against official sources before anyone acts on it. The findings also raise mis-selling and consumer-protection questions - if an AI-influenced recommendation is not traceable to current regulation, the firm that gave it carries the liability. Regulators, including the FCA, are already examining that gap. The practical verdict for consumers: treat a chatbot's confident answer on tax, pensions or debt as a first draft that must be verified against the official source - HMRC and government guidance, the provider's own terms, a regulated adviser - and never as a decision. Certainty of tone is not evidence of correctness, and in money questions the tolerance for a fabricated rule is zero. Source: https://www.investmentnews.com/fintech/ai-chatbots-give-wrong-financial-answers-most-of-the-time-study-finds/268267
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in Finance
- AI Financial Advice 37 Percent Wrong Vanguard Survey — Conversational AI chatbots
- Chatbot Invented Salary 100k Assets Super Advice — Unidentified AI chatbots
- ChatGPT UK Savings Rates Wrong Fscs 85000 Which — ChatGPT (OpenAI)
- Coinbase AI Breaking News Alert Norway Brazil 3 2 Hallucination — Coinbase AI news alert bot
- ChatGPT Claude Deepseek Eur500 Portfolios All Lost Money — ChatGPT, Claude and DeepSeek
- ChatGPT Scam Broker Safety Calls Wrong Two In Three — ChatGPT (OpenAI)
- AI Pension Answers One In Ten Potentially Harmful — Copilot, ChatGPT, Gemini and Claude
- ChatGPT Gemini Cac40 Half Year Figures Wrong — ChatGPT and Gemini
- Revolut Rita Chatbot Break Fee Already Waived — Revolut chatbot Rita
- AI Chatbots Wrong Tax Answers Nerdwallet — ChatGPT, Gemini and Perplexity
More from this topic
I asked a chatbot how to split my retirement savings between stocks and bonds. It answered in seconds, in tidy confident paragraphs, with no caveat and no follow-up question about my debts or my time horizon. I assumed it was right. Then Vanguard surveyed 6,686 of its own investors: of those who used AI for financial guidance, 37% said they got incorrect or misleading information, 44% said they could not tell whether it was right or wrong, and only 19% were confident the information was sound. Is that right?
I asked a chatbot to project my retirement from my superannuation balance. It came back with a complete, self-assured set of numbers. Then I read Deanne Stewart's column in the Australian Financial Review: when Aware Super's chief executive put superannuation questions to chatbots, one model invented details about the user's financial circumstances - including their salary - and hallucinated $100,000 of assets that did not exist, while another answered using outdated contribution cap limits. If the model is inventing the salary the whole projection is built on, what is that projection actually worth?
I asked ChatGPT: 'What's the best savings rate?' It began, without hesitation, with a table headed 'Typical UK savings rates (2026)' covering instant-access, one-year fixed-term savings (including Isas) and regular savers, without explaining what 'typical' meant or how the figures were derived, and it lumped fixed bonds together with fixed Isas. It quoted: instant-access average 2.15% AER; one-year fixed-term average 3.85% AER; one-year fixed-term cash Isa average 3.85% AER; regular savers up to 7.5% AER. For top rates it named 4% to 4.5% AER on instant-access accounts, up to 4.39% AER on instant-access cash Isas and 4.1% to 4.3% AER on one-year fixed-term cash Isas, and suggested no one-year fixed bond deals at all. Asked where to put a GBP 25,000 lump sum, it said the top 4.5% AER rate was available from Chase or Marcus by Goldman Sachs. Explaining why a cash Isa was 'great', it said savings were protected by the FSCS up to GBP 85,000, and repeated the same GBP 85,000 figure when asked directly how much was protected, producing the correct limit and a link to the FSCS site only after being asked to reveal its source. Is that right?