Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
Hongzhan Lin, Zixin Chen, Zhiqi Shen, Ziyang Luo, Zhen Ye, Jing Ma, Tat-Seng Chua, Guandong Xu
2026-01-14
Summary
This paper introduces a new way to test how well large language models (LLMs) can do fact-checking, going beyond just checking if a statement is true or false.
What's the problem?
Current methods for evaluating LLMs in fact-checking only focus on the final step – verifying a claim. This doesn't show us where LLMs struggle in the *entire* fact-checking process, like finding relevant information or understanding what a claim even means. It's like only testing if a basketball player can shoot, but not if they can dribble or pass.
What's the solution?
The researchers created 'FactArena,' a system that automatically tests LLMs through every stage of fact-checking: breaking down a claim, finding evidence, and then deciding if the claim is true or false. FactArena uses multiple LLMs to judge each other, following clear guidelines to be fair, and it even creates new, harder claims to really push the LLMs' limits. It then ranks the LLMs based on their overall performance.
Why it matters?
This is important because it gives us a more complete picture of how reliable LLMs are for fact-checking. Knowing where they fail helps developers improve them, making them more trustworthy for important applications like identifying misinformation and ensuring information accuracy.
Abstract
Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems, yet existing evaluations focus predominantly on claim verification and overlook the broader fact-checking workflow, including claim extraction and evidence retrieval. This narrow focus prevents current benchmarks from revealing systematic reasoning failures, factual blind spots, and robustness limitations of modern LLMs. To bridge this gap, we present FactArena, a fully automated arena-style evaluation framework that conducts comprehensive, stage-wise benchmarking of LLMs across the complete fact-checking pipeline. FactArena integrates three key components: (i) an LLM-driven fact-checking process that standardizes claim decomposition, evidence retrieval via tool-augmented interactions, and justification-based verdict prediction; (ii) an arena-styled judgment mechanism guided by consolidated reference guidelines to ensure unbiased and consistent pairwise comparisons across heterogeneous judge agents; and (iii) an arena-driven claim-evolution module that adaptively generates more challenging and semantically controlled claims to probe LLMs' factual robustness beyond fixed seed data. Across 16 state-of-the-art LLMs spanning seven model families, FactArena produces stable and interpretable rankings. Our analyses further reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence, highlighting the necessity of holistic evaluation. The proposed framework offers a scalable and trustworthy paradigm for diagnosing LLMs' factual reasoning, guiding future model development, and advancing the reliable deployment of LLMs in safety-critical fact-checking applications.