VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues
Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, Yi R., Fung
2025-02-24
Summary
This paper talks about VLM²-Bench, a tool designed to test how well AI models can recognize and connect visual cues, like identifying the same person in different photos, even without knowing their identity.
What's the problem?
AI models called Vision-Language Models (VLMs) are supposed to understand both images and text, but they struggle with a basic skill humans use daily: linking visual cues across different images. For example, recognizing the same object or person in different settings is difficult for these models, and existing tests don't fully measure this ability.
What's the solution?
The researchers created VLM²-Bench, a benchmark with over 3,000 test cases and nine subtasks that evaluate how well VLMs can connect visual cues. They tested eight open-source models and GPT-4o using this benchmark and found that even the best AI model was 34.80% less accurate than humans. They also explored ways to improve these models by enhancing their visual understanding and reducing biases from language-based reasoning.
Why it matters?
This matters because linking visual cues is a fundamental skill needed for tasks like facial recognition, object tracking, and video analysis. By identifying weaknesses in current AI models and suggesting improvements, VLM²-Bench helps researchers develop better systems that can handle real-world challenges more effectively.
Abstract
Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. Despite the extensive knowledge that vision-language models (VLMs) possess, it remains largely unexplored whether they are capable of performing this fundamental task. To address this, we introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can Visually Link Matching cues, with 9 subtasks and over 3,000 test cases. Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with further analysis of various language-side and vision-side prompting methods, leads to a total of eight key findings. We identify critical challenges in models' ability to link visual cues, highlighting a significant performance gap where even GPT-4o lags 34.80% behind humans. Based on these insights, we advocate for (i) enhancing core visual capabilities to improve adaptability and reduce reliance on prior knowledge, (ii) establishing clearer principles for integrating language-based reasoning in vision-centric tasks to prevent unnecessary biases, and (iii) shifting vision-text training paradigms toward fostering models' ability to independently structure and infer relationships among visual cues.