ViLBench: A Suite for Vision-Language Process Reward Modeling
Haoqin Tu, Weitao Feng, Hardy Chen, Hui Liu, Xianfeng Tang, Cihang Xie
2025-03-27
Summary
This paper is about creating a way to better evaluate how well AI models understand both images and language by giving them detailed feedback on their reasoning process.
What's the problem?
It's hard to tell if AI models are truly understanding complex tasks or just getting the right answer by chance. Current methods don't give enough insight into their reasoning process.
What's the solution?
The researchers created a new benchmark called ViLBench that requires AI models to show their work step-by-step and rewards them for each correct step, allowing for a more detailed evaluation of their reasoning abilities.
Why it matters?
This work matters because it can help researchers develop AI models that are more reliable and trustworthy by ensuring they are actually reasoning correctly, not just memorizing answers.
Abstract
Process-supervised reward models serve as a fine-grained function that provides detailed step-wise feedback to model responses, facilitating effective selection of reasoning trajectories for complex tasks. Despite its advantages, evaluation on PRMs remains less explored, especially in the multimodal domain. To address this gap, this paper first benchmarks current vision large language models (VLLMs) as two types of reward models: output reward models (ORMs) and process reward models (PRMs) on multiple vision-language benchmarks, which reveal that neither ORM nor PRM consistently outperforms across all tasks, and superior VLLMs do not necessarily yield better rewarding performance. To further advance evaluation, we introduce ViLBench, a vision-language benchmark designed to require intensive process reward signals. Notably, OpenAI's GPT-4o with Chain-of-Thought (CoT) achieves only 27.3% accuracy, indicating the benchmark's challenge for current VLLMs. Lastly, we preliminarily showcase a promising pathway towards bridging the gap between general VLLMs and reward models -- by collecting 73.6K vision-language process reward data using an enhanced tree-search algorithm, our 3B model is able to achieve an average improvement of 3.3% over standard CoT and up to 2.5% compared to its untrained counterpart on ViLBench by selecting OpenAI o1's generations. We release the implementations at https://ucsc-vlaa.github.io/ViLBench with our code, model, and data.