MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
Gailun Zeng, Ziyang Luo, Hongzhan Lin, Yuchen Tian, Kaixin Li, Ziyang Gong, Jianxiong Guo, Jing Ma
2025-11-14
Summary
This paper introduces a new way to test how well AI models that can understand both images and text – called Large Multimodal Models or LMMs – can critique their own work, similar to how a student might review and improve their own essay.
What's the problem?
AI models are getting really good at tasks like describing pictures or answering questions about them, but we haven't figured out a good way to measure if they can actually *evaluate* how good their own answers are and identify areas for improvement, which is crucial for them to become truly helpful assistants. Existing tests mostly focus on language-based AI, not these models that handle images too.
What's the solution?
The researchers created a benchmark called MM-CRITIC, which is a large collection of over 500 different tasks designed to test an LMM’s ability to critique itself in three ways: identifying basic errors, suggesting corrections, and comparing different answers. They used human experts to help create ideal answers and scoring guidelines, then used another powerful AI model, GPT-4o, to automatically grade the LMMs’ critiques, making the evaluation more consistent and reliable.
Why it matters?
This work is important because it provides a standardized way to assess and improve the self-critique abilities of LMMs. By understanding where these models struggle with self-evaluation, researchers can develop better AI assistants that are more accurate, reliable, and capable of learning from their mistakes, ultimately making them more useful in real-world applications.
Abstract
The ability of critique is vital for models to self-improve and serve as reliable AI assistants. While extensively studied in language-only settings, multimodal critique of Large Multimodal Models (LMMs) remains underexplored despite their growing capabilities in tasks like captioning and visual reasoning. In this work, we introduce MM-CRITIC, a holistic benchmark for evaluating the critique ability of LMMs across multiple dimensions: basic, correction, and comparison. Covering 8 main task types and over 500 tasks, MM-CRITIC collects responses from various LMMs with different model sizes and is composed of 4471 samples. To enhance the evaluation reliability, we integrate expert-informed ground answers into scoring rubrics that guide GPT-4o in annotating responses and generating reference critiques, which serve as anchors for trustworthy judgments. Extensive experiments validate the effectiveness of MM-CRITIC and provide a comprehensive assessment of leading LMMs' critique capabilities under multiple dimensions. Further analysis reveals some key insights, including the correlation between response quality and critique, and varying critique difficulty across evaluation dimensions. Our code is available at https://github.com/MichealZeng0420/MM-Critic.