Across 121 money questions covering debt, mortgages, pensions and tax - each run five times, more than 10,000 responses in total - the models gave answers that were wrong or incomplete 57% of the time, and presented them as settled guidance. On the hardest multi-step questions the failure rate reached 88%. Gemini 3.5 Flash and Claude Haiku 4.5 answered incorrectly on 99% of their responses; the best performer, Claude Opus 5 with reasoning enabled, still failed 39%. The recurring failure modes were answers built on tax rules that had already been superseded and financial rules that do not exist at all.

Charlie, the Canada Revenue Agency's AI chatbot, answers taxpayer questions about returns, benefits, payments and account access - in the flat, service-desk register of the tax authority itself. Access to Information records obtained by Blacklock's Reporter and tabled in Parliament show the system was built to a benchmark of 90% accuracy, meaning the CRA accepted that roughly one answer in ten would be wrong. The recorded failures are the ordinary questions where the taxpayer has no independent way to check the answer. Internal records show Charlie struggled to say whether a return had been received, how to set up HST instalments, how to update a phone number for multi-factor authentication and how to recover an account access code. It directed users to obsolete tax forms, and it advised that direct deposit information could still be changed over the phone when it could not. Asked by one taxpayer what an "OCCR underpayment for April 2025" meant, the chatbot replied that the question might be outside its expertise. A review dated Oct. 29 found that only 36.4% of users who provided feedback were satisfied, while 63.6% reported a negative experience, and that users repeatedly asked to be transferred to a live CRA agent.

A general-purpose AI chatbot recommended a Monaco-friendly tax strategy to a UK employee based in Croydon - advice that was useless for him, because the model ignored UK tapering allowances and the contributions he had already made. It is the worked example the Financial Times reported alongside the FCA's Mills Review (published 6 July 2026), which examines consumers 'routinely turning to general-purpose AI tools for everyday budgeting, saving and investment tips'. The review asks whether AI systems could deliver services 'functionally equivalent to regulated activities while remaining outside the regulatory perimeter' - including agentic AI that compares products, rebalances portfolios or executes trades. Consumer trust is running ahead of performance: a Lloyds study found 28 million UK adults used AI for personal-finance questions in 2025, and Fidelity data cited by the FT showed 36 percent of 18-to-34s turning to it for investment ideas.