ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Qing Jiang, Gen luo, Yuqin Yang, Yuda Xiong, Yihao Chen, Zhaoyang Zeng, Tianhe Ren, Lei Zhang
2024-11-28
Summary
This paper introduces ChatRex, a new multimodal large language model (MLLM) designed to improve how computers perceive and understand human poses by combining visual data with language processing.
What's the problem?
Many existing models struggle with accurately perceiving objects in images while also understanding language. For example, the current best model only correctly identifies objects about 43.9% of the time. This limitation makes it difficult to perform tasks that require both understanding and perception, such as answering questions about images or recognizing objects in different contexts.
What's the solution?
The authors developed ChatRex, which uses a unique design that separates how it perceives objects from how it understands language. Instead of having the model directly predict where objects are, it uses a system that retrieves information from a network that identifies potential object locations. They also created a new dataset called Rexverse-2M to help train the model on various tasks related to perception and understanding. This dual approach allows ChatRex to perform better in recognizing objects while still being good at understanding language.
Why it matters?
This research is important because it enhances the ability of AI to interact with the world in a more human-like way. By improving both perception and understanding, ChatRex can be used in many applications, such as virtual assistants, interactive storytelling, and advanced robotics, making technology more intuitive and effective.
Abstract
Perception and understanding are two pillars of computer vision. While multimodal large language models (MLLM) have demonstrated remarkable visual understanding capabilities, they arguably lack accurate perception abilities, e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on the COCO dataset, limiting many tasks requiring the combination of perception and understanding. In this work, we aim to bridge this perception gap from both model designing and data development perspectives. We first introduce ChatRex, an MLLM with a decoupled perception design. Instead of having the LLM directly predict box coordinates, we feed the output boxes from a universal proposal network into the LLM, allowing it to output the corresponding box indices to represent its detection results, turning the regression task into a retrieval-based task that LLM handles more proficiently. From the data perspective, we build a fully automated data engine and construct the Rexverse-2M dataset which possesses multiple granularities to support the joint training of perception and understanding. After standard two-stage training, ChatRex demonstrates strong perception capabilities while preserving multimodal understanding performance. The combination of these two capabilities simultaneously unlocks many attractive applications, demonstrating the complementary roles of both perception and understanding in MLLM. Code is available at https://github.com/IDEA-Research/ChatRex.