reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, Marjan Ghazvininejad
2025-03-18
Summary
This paper explores how reliable AI "reward models" are, especially when the input they receive is slightly changed or reworded.
What's the problem?
Reward models, used for evaluating text and guiding AI, might be overfitting to specific wording, making them unreliable when faced with paraphrased or slightly altered inputs. This means they might not truly understand the meaning, but rather recognize patterns in the specific wording they were trained on.
What's the solution?
The researchers created reWordBench, a tool that systematically transforms the input to reward models in ways that preserve the meaning or ranking. They then trained reward models to give similar scores to paraphrases, improving their robustness to various kinds of transformations.
Why it matters?
This work matters because it improves the reliability and trustworthiness of reward models, ensuring they are evaluating based on actual meaning rather than superficial patterns. This leads to better AI alignment and higher-quality outputs.
Abstract
Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time algorithms. However, while recent reward models increase performance on standard benchmarks, this may partly be due to overfitting effects, which would confound an understanding of their true capability. In this work, we scrutinize the robustness of reward models and the extent of such overfitting. We build **reWordBench**, which systematically transforms reward model inputs in meaning- or ranking-preserving ways. We show that state-of-the-art reward models suffer from substantial performance degradation even with minor input transformations, sometimes dropping to significantly below-random accuracy, suggesting brittleness. To improve reward model robustness, we propose to explicitly train them to assign similar scores to paraphrases, and find that this approach also improves robustness to other distinct kinds of transformations. For example, our robust reward model reduces such degradation by roughly half for the Chat Hard subset in RewardBench. Furthermore, when used in alignment, our robust reward models demonstrate better utility and lead to higher-quality outputs, winning in up to 59% of instances against a standardly trained RM.