Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
Feng-Lin Liu, Shi-Yang Li, Yan-Pei Cao, Hongbo Fu, Lin Gao
2025-08-25
Summary
This paper introduces a new method called Sketch3DVE for editing the actual structure of objects within videos, not just changing how things look. It allows you to make detailed changes to 3D scenes in videos, even when the camera is moving around a lot.
What's the problem?
Editing videos to change what's *in* the scene, like moving objects or changing their shape, is really hard. Existing methods struggle when the camera angle changes significantly – think big rotations or zooms. It’s difficult to create new views of the scene that look realistic and consistent, keep parts of the video untouched, and translate simple edits (like a sketch) into a full 3D video.
What's the solution?
Sketch3DVE lets you edit videos by essentially sketching what you want to change. The system takes that sketch and figures out how to realistically integrate it into the 3D world of the video. It first edits the first frame, then spreads those changes to the rest of the video. To understand the 3D structure, it creates a 3D model of the scene using multiple views. Then, it carefully merges the edited parts with the original video, making sure everything looks seamless and natural, using a technique that considers the 3D space.
Why it matters?
This research is important because it makes video editing much more powerful and flexible. Instead of just changing colors or styles, you can now actually reshape the 3D world within a video, even with complex camera movements. This could be useful for special effects, creating virtual environments, or even just making simple edits to home videos.
Abstract
Recent video editing methods achieve attractive results in style transfer or appearance modification. However, editing the structural content of 3D scenes in videos remains challenging, particularly when dealing with significant viewpoint changes, such as large camera rotations or zooms. Key challenges include generating novel view content that remains consistent with the original video, preserving unedited regions, and translating sparse 2D inputs into realistic 3D video outputs. To address these issues, we propose Sketch3DVE, a sketch-based 3D-aware video editing method to enable detailed local manipulation of videos with significant viewpoint changes. To solve the challenge posed by sparse inputs, we employ image editing methods to generate edited results for the first frame, which are then propagated to the remaining frames of the video. We utilize sketching as an interaction tool for precise geometry control, while other mask-based image editing methods are also supported. To handle viewpoint changes, we perform a detailed analysis and manipulation of the 3D information in the video. Specifically, we utilize a dense stereo method to estimate a point cloud and the camera parameters of the input video. We then propose a point cloud editing approach that uses depth maps to represent the 3D geometry of newly edited components, aligning them effectively with the original 3D scene. To seamlessly merge the newly edited content with the original video while preserving the features of unedited regions, we introduce a 3D-aware mask propagation strategy and employ a video diffusion model to produce realistic edited videos. Extensive experiments demonstrate the superiority of Sketch3DVE in video editing. Homepage and code: http://http://geometrylearning.com/Sketch3DVE/