ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler
Serin Yang, Taesung Kwon, Jong Chul Ye
2024-10-10
Summary
This paper presents ViBiDSampler, a new method for improving video interpolation by using a bidirectional diffusion sampler to create smoother and more coherent frames between two keyframes.
What's the problem?
Generating intermediate frames between two existing video frames (keyframes) can be difficult. Current methods often struggle with creating smooth transitions and may produce visual artifacts, which means the generated frames don't look realistic. These methods also require multiple steps of re-noising, making the process slow and inefficient.
What's the solution?
ViBiDSampler addresses these issues by using a bidirectional sampling strategy that works in both forward and backward directions from the keyframes. This means it can better understand the flow of motion in the video. The method includes advanced guidance techniques to enhance the quality of the generated frames without needing extensive re-noising or adjustments. As a result, it can produce high-quality videos quickly, achieving impressive performance metrics.
Why it matters?
This research is significant because it improves how we generate videos, making it easier to create smooth transitions between keyframes. This has practical applications in areas like video editing, animation, and creating slow-motion effects, ultimately enhancing the quality of visual content in various media.
Abstract
Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation. However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start & end) conditioned generation, which is essential for effective bounded interpolation. Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps. In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames. Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process. By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes. On a single 3090 GPU, our method can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.