Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, Jiaya Jia
2025-03-11
Summary
This paper talks about Seg-Zero, an AI tool that helps computers understand and outline objects in images by thinking through steps like a human, making it better at handling new types of images it hasn’t seen before.
What's the problem?
Current AI image tools struggle to explain their decisions and can’t adapt well to new types of pictures, often needing lots of labeled examples to learn.
What's the solution?
Seg-Zero splits the task into two parts: a reasoning model that figures out what to look for and a segmentation model that draws the outlines, using rewards to learn without needing step-by-step examples.
Why it matters?
This helps AI tools work better in real-world situations, like medical imaging or robotics, where they need to adapt quickly to new tasks and explain how they make decisions.
Abstract
Traditional methods for reasoning segmentation rely on supervised fine-tuning with categorical labels and simple descriptions, limiting its out-of-domain generalization and lacking explicit reasoning processes. To address these limitations, we propose Seg-Zero, a novel framework that demonstrates remarkable generalizability and derives explicit chain-of-thought reasoning through cognitive reinforcement. Seg-Zero introduces a decoupled architecture consisting of a reasoning model and a segmentation model. The reasoning model interprets user intentions, generates explicit reasoning chains, and produces positional prompts, which are subsequently used by the segmentation model to generate precious pixel-level masks. We design a sophisticated reward mechanism that integrates both format and accuracy rewards to effectively guide optimization directions. Trained exclusively via reinforcement learning with GRPO and without explicit reasoning data, Seg-Zero achieves robust zero-shot generalization and exhibits emergent test-time reasoning capabilities. Experiments show that Seg-Zero-7B achieves a zero-shot performance of 57.5 on the ReasonSeg benchmark, surpassing the prior LISA-7B by 18\%. This significant improvement highlights Seg-Zero's ability to generalize across domains while presenting an explicit reasoning process. Code is available at https://github.com/dvlab-research/Seg-Zero.