Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs
Ronit Singhal, Pransh Patwa, Parth Patwa, Aman Chadha, Amitava Das
2024-08-23
Summary
This paper discusses a new automated system for fact-checking online claims using advanced techniques from artificial intelligence, specifically focusing on large language models (LLMs).
What's the problem?
With the rise of misinformation on social media, it's become increasingly important to verify the accuracy of claims made online. Manually checking each claim is very difficult and time-consuming, which highlights the need for an automated system that can efficiently assess the truthfulness of information.
What's the solution?
The authors developed a system that uses a dataset called Averitec to check the validity of claims. Their approach combines a method called Retrieve and Generate (RAG) to find relevant evidence from a knowledge base and uses this evidence along with the claim to classify its truthfulness using a large language model. They also explored how well different LLMs can learn from few examples to improve their accuracy. Their system achieved a notable improvement in performance compared to previous methods.
Why it matters?
This research is significant because it addresses the urgent need for reliable fact-checking tools in an era where misinformation spreads quickly. By automating this process, the system can help ensure that people receive accurate information, which is crucial for informed decision-making and maintaining trust in online content.
Abstract
Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is highly challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset to assess the veracity of claims. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. All code will be made available on All code will be made available on https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms.