Beyond Fact or Fiction: Evaluating the Advanced Fact-Checking Capabilities of Large Language Models like GPT-4

Researchers from the University of Zurich focus on the role of Large Language Models (LLMs) like GPT-4 in autonomous fact-checking, evaluating their ability to phrase queries, retrieve contextual data, and make decisions while providing explanations and citations. Results indicate that LLMs, particularly GPT-4, perform well with contextual information, but accuracy varies based on query language and claim veracity. While it shows promise in fact-checking, inconsistencies in accuracy highlight the need for further research to understand their capabilities and limitations better.

Automated fact-checking research has developed with various approaches and shared tasks over the past decade. Researchers have proposed components like claim detection and evidence extraction, often relying on large language models and sources like Wikipedia. However, ensuring explainability remains challenging, as clear explanations of fact-checking verdicts are crucial for journalistic use.

The importance of fact-checking has grown with the rise of misinformation online. Hoaxes triggered this surge during significant events like the 2016 US presidential election and the Brexit referendum. Manual fact-checking must be improved for the vast amount of online information, necessitating automated solutions. Large Language Models like GPT-4 have become vital for verifying information. More explainability in these models is a challenge in journalistic applications.

The current study assesses the use of LLMs in fact-checking, focusing on GPT-3.5 and GPT-4. The models are evaluated under two conditions: one without external information and one with access to context. Researchers introduce an original methodology using the ReAct framework to create an iterative agent for automated fact-checking. The agent autonomously decides whether to conclude a search or continue with more queries, aiming to balance accuracy and efficiency, and justifies its verdict with cited reasoning.

The proposed method assesses LLMs for autonomous fact-checking, with GPT-4 generally outperforming GPT-3.5 on the PolitiFact dataset. Contextual information significantly improves LLM performance. However, caution is advised due to varying accuracy, especially in nuanced categories like half-true and mostly false. The study calls for further research to enhance the understanding of when LLMs excel or falter in fact-checking tasks.

GPT-4 outperforms GPT-3.5 in fact-checking, especially when contextual information is incorporated. Nevertheless, accuracy varies with factors like query language and claim integrity, particularly in nuanced categories. It also stresses the importance of informed human supervision when deploying LLMs, as even a 10% error rate can have severe consequences in today’s information landscape, highlighting the irreplaceable role of human fact-checkers.

Further research is essential to comprehensively understand the conditions under which LLM agents excel or falter in fact-checking. It is a priority to investigate the inconsistent accuracy of LLMs and identify methods for enhancing their performance. Future studies can examine LLM performance across query languages and its relationship with claim veracity. Exploring diverse strategies for equipping LLMs with relevant contextual information holds the potential for improving fact-checking. Analyzing the factors influencing the models’ improved detection of false statements compared to true ones can offer valuable insights into enhancing accuracy.

Check out the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on Telegram and WhatsApp.