Before relying on an AI chatbot for a quick news summary, you might want to reconsider. A recent report from the BBC reveals significant flaws in the summaries produced by these popular chatbots.
Google Gemini Had the Highest Rate of Problematic Summaries
The study evaluated ChatGPT, Google Gemini, Microsoft Copilot, and Perplexity AI. To kick off the experiment, the BBC posed 100 news-related questions to each chatbot, asking them to refer to BBC News sources whenever possible.
Experts from the BBC then analyzed the summaries provided by these AI chatbots. Shockingly, 51% of the summaries contained some form of error, ranging from factual inaccuracies to misquotations or outdated information. Of these mistakes, 19% were factual errors, like incorrect dates. Additionally, 13% of the quotations attributed to the BBC were either altered from the original text or didn’t appear in the articles the chatbots referenced.
When looking at each individual chatbot, Google Gemini stood out as the worst performer, with over 60% of its summaries containing problematic information. Microsoft Copilot followed closely behind at 50%, while ChatGPT and Perplexity AI each had around 40% of their responses flawed.
The BBC’s conclusion highlighted that the issues present in these summaries went beyond simply providing incorrect information. The researchers noted, “This research suggests that the types of errors introduced by AI assistants are broader than mere factual inaccuracies. The AI tools we assessed often struggled to distinguish between opinion and fact, inserted editorial comments, and frequently omitted crucial context. Even when each statement in a response is factually correct, these issues can lead to misleading or biased summaries.”
I personally didn’t consider using an AI chatbot for news summaries due to concerns about the technology’s reliability, but the findings from this study are strikingly high in error rates. Clearly, AI still has a long way to go before it can be trusted as a reliable source for news.
AI Features Still In Development
AI technology, particularly chatbots, continues to advance rapidly. However, as the BBC study indicates, seeking accurate news information through AI remains a challenging endeavor.
The BBC has also raised concerns about another AI-driven feature: Apple’s notification summaries. In December 2024, a notification erroneously reported that Luigi Mangione had shot himself, referring to him as the suspected shooter of healthcare CEO Brian Thompson.
In light of complaints from the BBC and other organizations, Apple temporarily suspended the summaries for news and entertainment apps starting with iOS 18.3.
So, when you’re looking to stay updated on the news, it’s best to keep it straightforward: skip the AI summaries and read the articles yourself.
