FleetCheck warns fleet managers over AI “hallucinations”

FleetCheck has warned fleet managers to treat general purpose artificial intelligence tools with a “high degree” of caution, amid growing concerns over inaccurate and misleading outputs generated by popular AI platforms.

The fleet software specialist said generative AI tools such as OpenAI’s ChatGPT and Anthropic’s Claude can deliver valuable insights from complex fleet data but can also produce responses that are misleading or entirely incorrect.

“We’re now seeing fleet managers across our user base start to use AI quite extensively and getting a good feel for the reliability of its results,” he said.

“There is no question that sometimes, its responses are excellent and rapidly creates insights from data that would be difficult to achieve through any other means. However, there are also times when its outputs lie somewhere on a line between misleading and dead wrong.”

Haymon-Collins highlighted an example where an AI tool incorrectly flagged potential commercial malpractice after analysing fuel card data and identifying rising pump prices between February and March.

“It didn’t spot the war in Iran had caused the issue, which is obviously a fundamental error,” he said.

According to Haymon-Collins, these kinds of “hallucinations” are likely to become more common as AI-generated content increasingly feeds back into the datasets used to train future models.

“We’re in an odd situation where the AI models are getting better but the data they use is getting worse because the information on which they train is now largely based on AI output,” he said.

Referencing published research from OpenAI, he added: “OpenAI’s own publicly published research shows that its latest reasoning model now creates hallucinations 48% of the time compared to just 16% in older versions.

“This is not a bug that can be solved but a feature of the technology. Especially, it is poor at admitting when it can’t produce a credible answer and tends to often present supposition as fact.”

Haymon-Collins said the only realistic safeguard was greater human oversight and scrutiny of AI-generated outputs.

“The only solution is closer human involvement and, for fleet managers, that means sense checking the output and not taking anything at face value,” he said.

“It’s worth bearing in mind that the hallucinations can often be subtle but still sufficiently wrong to cause issues.”

FleetCheck said it was taking a cautious approach to its own deployment of AI technologies, particularly in customer-facing applications.

Haymon-Collins said the company was focusing on tightly controlled use cases such as document processing, where the scope for error could be significantly reduced.

“Many fleet use cases for AI can be constructed but there has to be a very high level of reliability,” he said.

“Where we are offering products that use generative AI, such as document processing, we can limit the potential for any issues to a tiny degree.

“However, if you are a fleet manager employing AI for wider strategic insight and advice, the risks are much greater. It would certainly be possible to make decisions that are expensive, time consuming and embarrassing unless you exercise a high degree of caution.”