The High Cost of AI Advice: New Study Reveals Chatbots Frequently Fail Financial Accuracy Tests

The Allure of 'Fin-AI'

In an era of instant gratification, an increasing number of consumers are turning to generative artificial intelligence to manage their wallets. The convenience of having a 24/7 financial advisor in your pocket is undeniable, and the statistics reflect a massive shift in consumer behavior. According to data from Intuit Credit Karma, approximately 66% of Americans who have used generative AI have sought financial advice from these tools. This trend is even more pronounced among younger generations, with the number jumping to 82% for Gen Z and Millennials.

Currently, finance stands as the second most common use case for generative AI (41%), trailing only slightly behind health and wellness (44%). However, as more people trust these bots with their debt management, mortgages, and retirement planning, a sobering reality is emerging: the bots are often wrong.

The 'Artificial Authority' Report: A Reality Check

A comprehensive benchmarking study titled the "Artificial Authority" report, conducted by the technology firm Saturn, has cast a shadow over the reliability of popular AI tools. The research team tested 18 of the most prominent AI models, including industry giants like ChatGPT, Google Gemini, Anthropic Claude, and Microsoft Copilot, to determine if they could provide accurate financial counsel.

The results were alarming. On average, the AI models achieved an accuracy rate of only 43%. This means that more than half the time—57% of the time, to be exact—the chatbots provided incorrect or incomplete information. The failure rate spiked dramatically when the bots were presented with complex, multi-step financial scenarios involving specific tax rules or precise calculations. In these high-stakes cases, accuracy plummeted to a staggering 12%, with mistakes occurring in 88% of the tests.

Paid vs. Free: Does a Subscription Buy Accuracy?

The study also examined whether paid versions of these tools offered a safer bet for users. While paid models did perform better, the gap was not enough to ensure reliability. Free models showed a 63% failure rate, whereas paid models failed 49% of the time.

Among the free tools, Claude Haiku 4.5 was identified as one of the worst performers, giving wrong or incomplete answers 82% of the time. On the other end of the spectrum, ChatGPT-5.6 Luna (max) was the top free model, yet it still failed 56% of the time.

Even the best-performing model overall—the paid Claude Opus 5 (reasoning)—only achieved a 61% pass rate. This means that even with the most advanced reasoning capabilities available, the bot still failed nearly four out of every ten queries. When the questions became complex, Claude Opus 5's error rate rose to 67%.

Dangerous Hallucinations and Real-World Risks

The danger of AI financial advice isn't just in a wrong number; it's in the confidently delivered misinformation—often called 'hallucinations'—that can lead to severe financial loss. Saturn's deep dive revealed specific errors that could devastate a user's finances.

• Student Loan Misinformation: One model falsely claimed that college graduates could stop their student loan payments simply by moving abroad. In reality, such a move could actually lead to higher monthly repayments.
• Credit Score Errors: A Gemini model incorrectly informed a borrower that taking a mortgage payment holiday would have no impact on their credit score, a mistake that could lead to a significant drop in creditworthiness.

Conclusion: The Need for Human Oversight

As AI moves from being a helpful assistant to an perceived 'authority,' the risks grow. A report from EY highlighted that 53% of respondents prefer using AI for investment decisions, and 14% are even comfortable with fully autonomous AI managing their money. However, given the failure rates identified by Saturn, this level of trust is precarious.

While AI can be a powerful tool for brainstorming or organizing data, it remains an unreliable source for personal financial planning. Whether it is debt management, pensions, or inheritance tax, the consensus is clear: always verify AI-generated financial advice with a certified human professional to avoid costly mistakes.