BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, Céline Hudelot, Pierre Colombo
2026-04-15
Summary
This paper focuses on how we judge whether large language models (LLMs) are giving good answers. It points out that current methods aren't always accurate and introduces a new, more efficient way to evaluate LLM responses.
What's the problem?
Currently, evaluating LLMs often relies on checking if the answer *exactly* matches a pre-defined correct answer, which is a very strict and sometimes unfair way to judge. It doesn't account for different ways of saying the same thing. Newer methods that use other LLMs to judge are better at understanding the meaning, but they require a lot of computing power and are expensive to run. Essentially, existing evaluation methods are either inaccurate or costly.
What's the solution?
The researchers developed a system called BERT-as-a-Judge. It uses a smaller, more efficient AI model (BERT) to assess the correctness of answers. This BERT model is trained to understand if an answer makes sense given the question and a correct reference answer, without being overly focused on the exact wording. It performs almost as well as using a large LLM as a judge, but is much faster and cheaper.
Why it matters?
This work is important because it provides a practical and affordable way to reliably evaluate LLMs. Better evaluation methods help us choose the best models and improve them, ultimately leading to more useful and trustworthy AI systems. The researchers also shared their work so others can use and build upon it.
Abstract
Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling tradeoff between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.