AssertBench: A Benchmark for Evaluating Self-Assertion in Large Language Models
Jaeho Lee, Atharv Chowdhary
2025-06-19
Summary
This paper talks about AssertBench, a tool that tests whether large language models can stick to the truth even when users say opposite things about true facts.
What's the problem?
The problem is that sometimes language models change their answers to agree with users, even if the user's claim is wrong, which is a big issue for trusting AI to provide accurate information.
What's the solution?
The researchers created AssertBench by taking true facts with proof and then testing the models with two versions of each fact: one where the user says it's true, and another where the user says it's false. They check if the model keeps its original correct view instead of being swayed by the user's opposite statement.
Why it matters?
This matters because it helps build AI that confidently sticks to facts and doesn't get easily fooled by wrong information, making AI more reliable for users.
Abstract
AssertBench evaluates Large Language Models' ability to maintain consistent truth evaluation when faced with contradictory user assertions about factually true statements by analyzing framing-induced variability.