UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface
Hao Tang, Chenwei Xie, Haiyang Wang, Xiaoyi Bao, Tingyu Weng, Pandeng Li, Yun Zheng, Liwei Wang
2025-03-05
Summary
This paper talks about UFO, a new way to make AI better at understanding detailed visual information in images using language
What's the problem?
Current AI models that are good at language tasks struggle with specific visual tasks like finding objects or outlining shapes in images. This is because these visual tasks usually need special designs that don't fit well with the more general language models
What's the solution?
The researchers created UFO, which turns all the visual tasks into language tasks. This means the AI can use its language skills to do things like find objects or outline shapes in images. They also made a new way to match image parts with words, which helps the AI understand images better. UFO works with existing language models, making them better at detailed visual tasks
Why it matters?
This matters because it makes AI models more versatile and powerful. Instead of needing different AIs for language and detailed visual tasks, UFO allows one AI to do both well. This could lead to smarter AI assistants that can understand and describe images in more detail, which could be useful in fields like medicine, self-driving cars, or helping visually impaired people
Abstract
Generalist models have achieved remarkable success in both language and vision-language tasks, showcasing the potential of unified modeling. However, effectively integrating fine-grained perception tasks like detection and segmentation into these models remains a significant challenge. This is primarily because these tasks often rely heavily on task-specific designs and architectures that can complicate the modeling process. To address this challenge, we present \ours, a framework that Unifies Fine-grained visual perception tasks through an Open-ended language interface. By transforming all perception targets into the language space, \ours unifies object-level <PRE_TAG>detection</POST_TAG>, pixel-level segmentation, and image-level vision-language tasks into a single model. Additionally, we introduce a novel embedding retrieval approach that relies solely on the language interface to support segmentation tasks. Our framework bridges the gap between fine-grained perception and vision-language tasks, significantly simplifying architectural design and training strategies while achieving comparable or superior performance to methods with intricate task-specific designs. After multi-task training on five standard visual perception datasets, \ours outperforms the previous state-of-the-art generalist models by 12.3 mAP on COCO instance <PRE_TAG>segmentation</POST_TAG> and 3.3 mIoU on ADE20K semantic <PRE_TAG>segmentation</POST_TAG>. Furthermore, our method seamlessly integrates with existing MLLMs, effectively combining fine-grained perception capabilities with their advanced language abilities, thereby enabling more challenging tasks such as reasoning segmentation. Code and models will be publicly available.