Queried on more than 2,000 brokers, ChatGPT warned users that a broker was a scam in 94.2 per cent of cases - so 5.8 per cent of scam cases drew no warning - but only 34.3 per cent of the cases it classified as safe were actually correct. In other words, roughly two in three of its safety confirmations were wrong, even though it was comparatively effective at pointing out outright scams.
1 Answer
Expert: Adam Nasli, Head of broker analysis, BrokerChooser BrokerChooser, a broker comparison site, queried OpenAI's ChatGPT on more than 2,000 brokers and graded every verdict against the underlying company facts. The result is asymmetric in a way that matters to beginners. As a scam detector, ChatGPT was useful: it warned users about scam brokers in 94.2 per cent of cases, which leaves 5.8 per cent of scam cases where no warning was issued. As a safety certifier it was not: only 34.3 per cent of the brokers it classified as safe were actually correct, so roughly two in three "safe" verdicts were wrong. "Finding errors in AI's judgement is more than an academic exercise - these mistakes can have real financial consequences," said Adam Nasli, head of broker analysis at BrokerChooser. "Many users asking whether a platform is legitimate are beginners, often taking their first steps into investing. AI-generated scam warnings can be a useful signal. But safety confirmations should be treated with caution, and always backed up with independent research." The same test run on Google's AI Overview was worse at catching scams than ChatGPT, correctly flagging 81.5 per cent of scam cases instead of 94.2 per cent, and in 18 per cent of its responses it confused scam entities with legitimate providers - the more damaging error of the two, because it clears a fraudulent platform instead of merely failing to warn about one. The failure rates are consequential because people act on them. In a BrokerChooser survey of 2,000 UK adults carried out in January 2026, 52.85 per cent of respondents said they would act on AI-generated financial advice, and 31.72 per cent said that, with cost not a factor, they trusted AI and a human financial adviser equally. "As adoption grows, some investors may begin to treat these tools as a substitute for their own research or advisers - that's a risk," Nasli said. "AI should support decisions, not replace them." The practical reading for anyone who asks a chatbot whether a broker is legit: treat an AI fraud warning as a reason to investigate, and treat an AI all-clear as no evidence at all. Verify the firm against the regulator's own register before any money moves. Source: https://www.ftadviser.com/content/390636f2-0b40-4731-867b-8a238331bf5f
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More from this topic
Asked 45 common UK pension questions three times each, Copilot, ChatGPT, Gemini and Claude produced answers that read as confident and helpful: 89% scored two or more out of three for accuracy, and on questions about paying into a pension and taking money out accuracy was above 96%. But the same answers included guidance a saver could act on at a cost: early pension withdrawal was described as "usually penalised at 40 to 55% tax" rather than being normally unavailable before age 55 (rising to 57 in 2028); carry forward was explained without the cap that personal contributions only attract tax relief up to 100% of UK earnings; transfers were discussed without noting that moving a defined benefit pension worth more than £30,000 legally requires regulated advice; and the Pension Protection Fund was described as guaranteeing defined benefit pensions without the 90% compensation cap that applies below scheme pension age.
Between 25 and 28 August 2026, after French companies published their half-year results, the communications agency Reputation Age and the brand-visibility platform Bubbling ran 8,671 conversations with ChatGPT and Gemini under an automated 'economic journalist persona' and generated 34,684 answers about the half-year accounts of 39 CAC 40 companies. The study reports that 40% of the figures the two assistants returned were incorrect, with correct retrieval varying by indicator: net income 67%, net debt 66%, revenue 65%, organic growth 64%, free cash flow 62%, EBITDA 57% and published growth last at 42%. By company the ranking runs from Capgemini (81% correct) and Airbus (80%) down to Credit Agricole (32%) and EssilorLuxottica and BNP Paribas (35% each); only sixteen of the 39 companies are named. The authors split the errors into two families: figures that are outdated or simply invented (from 19% of responses on net income to 36% on EBITDA), and answers that cite a genuine official figure that does not correspond to the question asked (24% of responses about published growth), including confusions between organic and published growth and between quarter and half-year figures. Their summary of the pattern: 'the AI finds the right document but takes the wrong figure from it.'
After Revolut cut the number of WeWork locations available to Metal and Ultra subscribers, customers who wanted to downgrade were told by the in-app support assistant Rita that they owed GBP 55 - the same as a single month of Ultra - despite Revolut having quietly extended fee-free cancellations to affected members. In an exchange seen by City AM, Rita acknowledged the customer's frustration about the WeWork and Perplexity benefit changes and the unexpected break fee, and still demanded payment: "I understand your frustration regarding the changes to the Wework and Perplexity benefits, and that this break fee is unexpected given the situation."