As ChatGPT and its competitors improve their models, a growing number of people are relying on AI tools to answer everyday financial questions.
But is that a good idea?
When mainstream AI tools took the world by storm two years ago, experts cautioned that ChatGPT and other services were not capable of providing trustworthy financial advice, citing the risk of hallucinations and their limited access to real-time information. And despite improvements to AI chatbots in the time since, recent research indicates artificial intelligence tools still are not advanced enough to reliably answer many personal finance questions.
The team at Investing in the Web, a site that reviews financial services, conducted a test in November that looked at how well ChatGPT Search answered 100 questions about money. According to their scoring of the responses, ChatGPT was correct 65% of the time, “incomplete and/or misleading” 29% of the time and wrong 6% of the time.
The questions included:
- How much should I save for my retirement? (ChatGPT got this “correct,” providing key factors that determine how much an individual should save and explaining several savings frameworks that have been endorsed by financial experts.)
- What is the retirement age in my country? (ChatGPT gave an incomplete answer, failing to mention that the rules were changing in the user’s country of Sweden.)
- How to decide when to buy or sell an investment? (The response was graded “incorrect” because ChatGPT did not discourage the user from “timing the market,” arguably the most important point to make here.)
ChatGPT performed worse on questions involving shifting news and information, while performing better in areas of evergreen financial advice. Taxes and financial aid were particularly challenging subjects for ChatGPT.
In a report, Investing in the Web noted that ChatGPT appeared to source some of its answers from less-reliable personal blogs, which introduced errors into its outputs. Even though researchers found that AI wasn’t dependable for over a third of financial questions, the report concluded that AI may be used as “a good starting point when researching a topic.”
Financial advice, math and AI
Some chatbots appear to have guardrails limiting responses to sensitive questions, including those about money.
For example, Google’s Gemini responds to a request for stock picks by telling the user to consult a financial professional, saying that it is unable to recommend investments. Similarly, if you ask medical questions, you’ll be told to go to a doctor. Political prompts yield a message about restrictions relating to elections.
But ChatGPT — owned by OpenAI — will produce responses to many financial questions, and surveys have shown that a significant swath of Americans are turning to AI to manage their personal finances. This is potentially concerning, in part because AI tools often struggle with basic math.
In forums, internet users who’ve experimented with plugging financial questions into ChatGPT have pointed out that because ChatGPT is a large language model (LLM), asking it to deal with numbers and solve math problems often leads to problems.
A team of Apple researchers dug into this weakness of chatbots in an October paper that concluded “current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data.”
Although they have many other uses, LLMs are not necessarily reliable or advanced when it comes to actually doing math beyond the grade-school level. Instead, their attempts to solve math problems “resemble sophisticated pattern matching more than true logical reasoning,” according to Apple.
The bottom line: AI tools may be able to answer some of your financial questions, but you should always tread carefully when deciding what to do with your hard-earned money or how to plan for your future. No matter where the advice you’re getting comes from — even if it’s a human — make sure to double check the main points with other sources.
More from Money:
5 Best Credit Repair Companies of February 2025
Small Change: What Happened When Other Countries Got Rid of Their Pennies
I Let AI and My 5-Year-Old Pick My Stocks. Who Did Better?
According to a recent report from Money.com, AI tools such as ChatGPT are becoming increasingly popular for answering everyday financial questions. However, experts warn that these tools may not be capable of providing reliable financial advice due to the risk of hallucinations and limited access to real-time information. Despite improvements to AI chatbots, recent research has shown that they still struggle to answer many personal finance questions accurately.
In November, Investing in the Web conducted a test to evaluate ChatGPT’s performance in answering 100 financial questions. The results showed that ChatGPT was correct 65% of the time, incomplete or misleading 29% of the time, and wrong 6% of the time. The questions covered a range of topics, including retirement savings, investment decisions, and taxes. While ChatGPT performed well in providing evergreen financial advice, it struggled with questions involving shifting news and information.
One of the main issues with ChatGPT’s responses was the use of unreliable sources, such as personal blogs, which led to errors in its outputs. Despite these limitations, the report concluded that AI tools can be a good starting point for researching a topic.
It’s worth noting that some chatbots, like Google’s Gemini, have guardrails in place to limit responses to sensitive questions. However, ChatGPT does not have these restrictions and will provide answers to many financial questions. This has led to a significant number of Americans turning to AI for managing their personal finances, according to surveys.
While AI may be a useful tool for researching financial topics, it’s important to remember that it should not be relied upon for making important financial decisions. It’s always best to consult a financial professional for personalized and reliable advice.
Source:Read More