MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models
Young-Jun Lee, Byung-Kwan Lee, Jianshu Zhang, Yechan Hwang, Byungsoo Ko, Han-Gyu Kim, Dongyu Yao, Xuankun Rong, Eojin Joo, Seung-Ho Han, Bowon Ko, Ho-Jin Choi
2025-10-21
Summary
This paper introduces a new way to test how well Vision-and-Language Models (VLMs) can handle conversations that go back and forth over multiple turns, like a real discussion. It shows that current models struggle with these kinds of complex interactions.
What's the problem?
Existing tests for VLMs mostly focus on single questions and answers. Real-world use, however, requires models to remember what was said earlier in a conversation and build upon that. Current multi-turn conversation datasets aren't diverse or challenging enough to truly measure a model’s ability to handle these complex, ongoing dialogues. Basically, the tests weren't realistic enough to show where models really needed improvement.
What's the solution?
The researchers created a new dataset called MultiVerse, which contains 647 conversations averaging four turns each. These conversations cover a wide range of tasks – from simple questions about images to more difficult things like math and coding. To judge how well the models do, they used GPT-4o to automatically check 37 different aspects of the responses, like if the model correctly understood the image, if its language was clear, and if the information it provided was accurate.
Why it matters?
This work is important because it highlights that even the best VLMs aren't very good at multi-turn conversations, only getting about half the answers right. It also shows that giving the model the full conversation history helps smaller models perform better, meaning that 'remembering' the context is crucial. MultiVerse provides a more realistic and challenging benchmark for improving these models and pushing the field forward.
Abstract
Vision-and-Language Models (VLMs) have shown impressive capabilities on single-turn benchmarks, yet real-world applications often demand more intricate multi-turn dialogues. Existing multi-turn datasets (e.g, MMDU, ConvBench) only partially capture the breadth and depth of conversational scenarios encountered by users. In this work, we introduce MultiVerse, a novel multi-turn conversation benchmark featuring 647 dialogues - each averaging four turns - derived from a diverse set of 12 popular VLM evaluation benchmarks. With 484 tasks and 484 interaction goals, MultiVerse covers a wide range of topics, from factual knowledge and perception to advanced reasoning tasks such as mathematics and coding. To facilitate robust assessment, we propose a checklist-based evaluation method that leverages GPT-4o as the automated evaluator, measuring performance across 37 key aspects, including perceptual accuracy, linguistic clarity, and factual correctness. We evaluate 18 VLMs on MultiVerse, revealing that even the strongest models (e.g., GPT-4o) achieve only a 50% success rate in complex multi-turn conversations, highlighting the dataset's challenging nature. Notably, we find that providing full dialogue context significantly enhances performance for smaller or weaker models, emphasizing the importance of in-context learning. We believe MultiVerse is a landscape of evaluating multi-turn interaction abilities for VLMs.