✦
Summary
SeC is a concept-driven segmentation framework that advances complex video object segmentation via progressive concept construction. It shifts from conventional feature matching to the progressive construction and utilization of high-level, object-centric representations. SeC employs Large Vision-Language Models (LVLMs) to integrate visual cues across diverse frames, constructing robust conceptual priors. This approach enables SeC to handle drastic visual variations, occlusions, and complex scene changes, which are challenging for current techniques.
Key Features
Concept-driven segmentation framework
Progressive construction and utilization of high-level, object-centric representations
Employment of Large Vision-Language Models (LVLMs) for robust conceptual priors
Adaptive balancing of LVLM-based semantic reasoning and enhanced feature matching
Dynamically adjusting computational efforts based on scene complexity
Robust handling of drastic visual variations, occlusions, and complex scene changes
High-quality segmentation results with computational efficiency
State-of-the-art performance on SeCVOS and other benchmarks
About SeC
SeC forms a comprehensive semantic representation of the target based on processed frames, realizing robust segmentation of follow-up frames. It adaptively balances LVLM-based semantic reasoning with enhanced feature matching, dynamically adjusting computational efforts based on scene complexity. This allows SeC to achieve high-quality segmentation results while being computationally efficient. SeC has been evaluated on various benchmarks, including the newly introduced Semantic Complex Scenarios Video Object Segmentation benchmark (SeCVOS), and has demonstrated substantial improvements over state-of-the-art approaches.
SeCVOS is a benchmark designed to challenge models with substantial appearance variations and dynamic scene transformations. It comprises 160 manually annotated multi-scenario videos, which are used to rigorously assess VOS methods in scenarios demanding high-level conceptual reasoning and robust semantic understanding. SeC has achieved an 11.8-point improvement over SAM 2.1 on SeCVOS, establishing a new state-of-the-art in concept-aware video object segmentation. This demonstrates the effectiveness of SeC in handling complex video object segmentation tasks.
