The High Cost of Digital Advice: Study Reveals AI Chatbots Frequently Fail Financial Queries

The Growing Reliance on 'Fin-AI'

In an era where artificial intelligence is integrated into almost every facet of daily life, a growing number of consumers are turning to generative AI for one of the most critical aspects of their lives: their finances. Recent data from Intuit Credit Karma suggests a significant shift in consumer behavior, revealing that 66% of Americans who have used generative AI have utilized it to seek financial advice. This trend is even more pronounced among younger generations, with the figure jumping to 82% for Gen Z and Millennials.

Finance has now become the second most common use case for generative AI, trailing only slightly behind health and wellness. However, as more people treat chatbots as virtual financial advisors, a new report warns that this trust could be a costly mistake.

The 'Artificial Authority' Report: A Reality Check

A comprehensive study titled the "Artificial Authority" report, conducted by the AI and technology firm Saturn, has cast a shadow over the reliability of AI-driven financial counsel. To test the waters, Saturn's Domain team benchmarked 18 of the world's most popular AI tools, including industry giants like ChatGPT, Google Gemini, Anthropic's Claude, and Microsoft Copilot.

The findings were sobering. Across the tested models, the average accuracy rate for financial questions was just 43%. This means that more than half the time—57% of the cases—the chatbots provided incorrect information. The situation worsened significantly when the AI was presented with complex, multi-step scenarios involving specific tax laws or precise financial figures; in these instances, accuracy plummeted to a staggering 12%, with mistakes occurring 88% of the time.

Paid vs. Free: Does a Subscription Guarantee Accuracy?

The study also explored whether paying for a premium AI subscription provides a safer experience. While paid models did perform better than their free counterparts, the margin of error remained alarmingly high. Free models exhibited a 63% failure rate, whereas paid models failed 49% of the time.

Among the free tools, Claude Haiku 4.5 was identified as one of the worst performers, providing wrong or incomplete answers 82% of the time. On the other end of the spectrum, ChatGPT-5.6 Luna (max) was the top free performer, though it still gave incorrect information 56% of the time.

Even the most successful model in the study, Anthropic's paid Claude Opus 5 (reasoning), only achieved a pass rate of 61%. Even for the 'best' AI, nearly four out of ten answers were wrong, and the failure rate climbed to 67% when dealing with complex financial queries.

Dangerous Hallucinations and Real-World Risks

The most concerning aspect of the Saturn report isn't just the percentage of errors, but the nature of those mistakes. AI 'hallucinations'—where the bot confidently presents false information as fact—can lead to severe financial penalties or missed opportunities.

The researchers highlighted two particularly dangerous examples: one AI model falsely claimed that college graduates could stop student loan payments by moving abroad, a move that could actually lead to higher monthly repayments in reality. In another instance, a Gemini model incorrectly informed a borrower that taking a mortgage payment holiday would have no effect on their credit score—advice that could devastate a user's creditworthiness.

The Bottom Line: Trust, but Verify

As AI continues to evolve from a simple productivity tool to a perceived authority, the gap between user trust and actual performance is widening. A report from EY noted that 53% of respondents prefer using an AI assistant for investment decisions, with another 14% open to fully autonomous AI. However, Saturn's findings suggest that relying on these tools for debt management, pensions, inheritance tax, and retirement planning is a high-risk gamble.

While AI can be an excellent starting point for brainstorming or organizing data, the consensus is clear: when it comes to your wallet, a human expert remains irreplaceable. For now, the safest way to use AI for finance is to treat its output as a suggestion that must be verified by a certified professional.