MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation
Henghui Ding, Chang Liu, Shuting He, Kaining Ying, Xudong Jiang, Chen Change Loy, Yu-Gang Jiang
2025-12-17
Summary
This paper introduces a new dataset called MeViS, designed to help computers better understand videos by focusing on how objects *move* as described in language. It's about teaching computers to not just identify *what* something is, but *how* it's moving, based on spoken or written instructions.
What's the problem?
Current datasets used to train computers to understand videos often focus on what objects look like – their color, shape, etc. – and use descriptions that could let you identify the object just by looking at a single frame. This doesn't really test if the computer understands motion, which is crucial for understanding what's happening in a video. Existing datasets don't emphasize the importance of both motion in the video *and* how that motion is described in language.
What's the solution?
The researchers created MeViS, a large dataset with over 33,000 descriptions of object motions in videos, covering over 8,000 objects across more than 2,000 videos. They then tested 15 existing computer programs on tasks related to understanding these motion descriptions, like identifying objects moving in a specific way or tracking them as they move. They found these programs struggled, so they developed a new method, LMPM++, which performs better on these tasks.
Why it matters?
This work is important because it highlights the need for datasets that specifically test a computer's ability to understand motion in videos. By providing MeViS and a new, improved method, the researchers are helping to advance the field of video understanding, making it possible for computers to better interpret complex scenes and respond to instructions about moving objects. The dataset is publicly available, allowing other researchers to build upon this work.
Abstract
This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects' motions. Existing referring video segmentation datasets often focus on salient objects and use language expressions rich in static attributes, potentially allowing the target object to be identified in a single frame. Such datasets underemphasize the role of motion in both videos and languages. To explore the feasibility of using motion expressions and motion reasoning clues for pixel-level video understanding, we introduce MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio, covering 8,171 objects in 2,006 videos of complex scenarios. We benchmark 15 existing methods across 4 tasks supported by MeViS, including 6 referring video object segmentation (RVOS) methods, 3 audio-guided video object segmentation (AVOS) methods, 2 referring multi-object tracking (RMOT) methods, and 4 video captioning methods for the newly introduced referring motion expression generation (RMEG) task. The results demonstrate weaknesses and limitations of existing methods in addressing motion expression-guided video understanding. We further analyze the challenges and propose an approach LMPM++ for RVOS/AVOS/RMOT that achieves new state-of-the-art results. Our dataset provides a platform that facilitates the development of motion expression-guided video understanding algorithms in complex video scenes. The proposed MeViS dataset and the method's source code are publicly available at https://henghuiding.com/MeViS/