Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models

Summary

This paper investigates whether powerful vision-language models (VLMs) actually *understand* what they're looking at, or if they're just really good at remembering famous things. The research shows these models are surprisingly reliant on memorization, performing much better on well-known buildings than on ordinary ones.

What's the problem?

Current VLMs, while impressive, might be succeeding by simply recognizing and recalling information about popular subjects instead of truly understanding visual concepts. This means they might fail when presented with something they haven't 'seen' before, revealing a lack of genuine reasoning ability. There wasn't a good way to systematically test this weakness, and existing datasets weren't large or detailed enough.

What's the solution?

The researchers created a new, large dataset called YearGuessr, containing over 55,000 images of buildings from around the world. Each image is labeled with when the building was constructed, its location, and how popular it is (measured by how many times its webpage is viewed). They then used this dataset to test how well VLMs could predict a building's construction year, focusing on whether performance differed based on the building's popularity. They also developed new ways to measure accuracy that specifically account for this popularity bias.

Why it matters?

This research is important because it highlights a significant limitation of current VLMs. If these models can't generalize beyond memorized information, their usefulness in real-world applications – where they'll inevitably encounter unfamiliar situations – is limited. Understanding this bias is crucial for developing more robust and truly intelligent AI systems that can reason and understand the world around them, rather than just recognizing what they've already 'learned'.