Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs
Angela van Sprang, Laurens Samson, Ana Lucic, Erman Acar, Sennay Ghebreab, Yuki M. Asano
2025-12-10
Summary
This paper investigates a problem with advanced AI models called Multimodal Large Language Models, or MLLMs, which are designed to understand both images and text. The researchers created new tests to see if these models truly understand information the same way, no matter if it's presented as an image, as text, or a combination of both.
What's the problem?
MLLMs are built to connect how they understand images and language, essentially putting them in the same 'thinking space'. However, the paper shows that these models often give different answers to the same question depending on whether the information is given as an image, text, or both. This means they aren't consistently reasoning across different types of input, even though they should be. It's like asking someone the same question in different ways and getting different answers – it suggests they don't truly *understand* the core concept.
What's the solution?
The researchers developed two new sets of tests, called REST and REST+, specifically designed to highlight these inconsistencies. They tested 15 different MLLMs using these benchmarks, presenting the same information in images, text, and combined formats. They then analyzed how often the models gave consistent answers across these formats. They also looked at factors like text recognition accuracy (OCR) and visual details like color and resolution to see if those affected the results. They found that even perfect OCR didn't fix the problem, and visual characteristics *did* influence performance.
Why it matters?
This research is important because it reveals a fundamental weakness in current MLLMs. If these models can't consistently understand information regardless of how it's presented, it limits their reliability in real-world applications. Understanding *why* these inconsistencies happen – the paper links it to a 'gap' between how the models process text and images – is a crucial step towards building more robust and trustworthy AI systems that can truly bridge the gap between vision and language.
Abstract
We introduce two new benchmarks REST and REST+(Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.