OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video Diffusion Models
Dvir Samuel, Matan Levy, Nir Darshan, Gal Chechik, Rami Ben-Ari
2025-03-25
Summary
This paper is about a new way to edit videos that allows you to remove objects, extract layers, and combine them with other videos without needing to train a special AI model.
What's the problem?
Existing video editing techniques often require a lot of training data or complex optimization, making them difficult to use.
What's the solution?
The researchers developed a method called OmnimatteZero that uses pre-trained AI models to achieve real-time video editing, allowing for quick and easy object removal, layer extraction, and compositing.
Why it matters?
This work matters because it makes advanced video editing techniques more accessible and efficient, opening up new possibilities for creative video manipulation.
Abstract
Omnimatte aims to decompose a given video into semantically meaningful layers, including the background and individual objects along with their associated effects, such as shadows and reflections. Existing methods often require extensive training or costly self-supervised optimization. In this paper, we present OmnimatteZero, a training-free approach that leverages off-the-shelf pre-trained video diffusion models for omnimatte. It can remove objects from videos, extract individual object layers along with their effects, and composite those objects onto new videos. We accomplish this by adapting zero-shot image inpainting techniques for video object removal, a task they fail to handle effectively out-of-the-box. We then show that self-attention maps capture information about the object and its footprints and use them to inpaint the object's effects, leaving a clean background. Additionally, through simple latent arithmetic, object layers can be isolated and recombined seamlessly with new video layers to produce new videos. Evaluations show that OmnimatteZero not only achieves superior performance in terms of background reconstruction but also sets a new record for the fastest Omnimatte approach, achieving real-time performance with minimal frame runtime.