Introduction
A recent international study by the Union of European Radio-Broadcasters (UER) and the British Broadcasting Corporation (BBC) has revealed that leading AI news assistants frequently misinterpret and distort factual news content. The study assessed the accuracy, source reliability, and ability to differentiate between facts and opinions of AI assistants across 14 languages, including ChatGPT, Copilot, Gemini, and Perplexity.
Study Findings
The research analyzed 3,000 AI-powered news assistant responses to factual news queries. In total, 45% of the AI-generated responses contained at least one significant error, with 81% exhibiting some form of issue according to the study.
Key AI companies like OpenAI and Microsoft have previously acknowledged the issue of “hallucinations” in AI models, where incorrect or misleading information is generated due to factors such as insufficient data. Perplexity, however, claims a 93.9% factual accuracy rate for one of its “Deep Research” modes on their website.
Source Errors
The study found that one-third of AI assistant responses contained severe source errors, such as nonexistent, misleading, or incorrect attributions. Gemini, Google’s AI assistant, had significant source problems in 72% of its responses, while the rest of the AI assistants had less than 25% such issues.
Precision problems were detected in 20% of all AI assistant responses, including outdated information.
- Gemini incorrectly answered questions about changes in disposable vaporizer legislation.
- ChatGPT incorrectly reported Pope Francis as the current pope several months after his death.
The study involved 22 public service media outlets from 18 countries, including France, Germany, Spain, Ukraine, the United Kingdom, and the United States.
Key Questions and Answers
- What is the main finding of the study? The study found that leading AI news assistants often misinterpret and distort factual news content, with 45% of their responses containing significant errors.
- Which AI assistants were evaluated in the study? The study assessed ChatGPT, Copilot, Gemini, and Perplexity across 14 languages.
- What types of errors were identified in AI responses? The study found issues with source reliability, precision, and the ability to distinguish facts from opinions in AI-generated responses.
- How did Gemini perform compared to other AI assistants? Gemini had significant source problems in 72% of its responses, while the other AI assistants had less than 25% such issues.
- What are hallucinations in AI models? Hallucinations refer to the generation of incorrect or misleading information by AI models, often due to insufficient data or other factors.
