Imagine360: Immersive 360 Video Generation from Perspective Anchor
Jing Tan, Shuai Yang, Tong Wu, Jingwen He, Yuwei Guo, Ziwei Liu, Dahua Lin
2024-12-05
Summary
This paper introduces Imagine360, a new framework that transforms standard perspective videos into immersive 360-degree videos, allowing viewers to explore dynamic scenes from all angles.
What's the problem?
Creating engaging 360-degree videos can be challenging because traditional videos only show a limited view. Viewers miss out on the full experience if they can't see everything happening around them. Existing methods for generating 360-degree videos often lack the ability to create rich motion patterns and can be difficult to use, making it hard for content creators to produce high-quality immersive videos.
What's the solution?
Imagine360 solves this problem by using advanced techniques to generate high-quality 360-degree videos from standard perspective footage. It employs a dual-branch design that combines local and global information from both perspective and panoramic videos. This allows the system to learn detailed visual and motion patterns. Additionally, it uses an antipodal mask to capture long-range motion between opposite points in the video and elevation-aware designs to adapt to changes in height across frames. These features help create smooth and coherent motion in the final 360-degree video.
Why it matters?
This research is important because it enhances the ability of creators to produce immersive content that fully engages viewers. By making it easier to generate high-quality 360-degree videos, Imagine360 can be used in various applications such as virtual reality experiences, gaming, and educational content, ultimately improving how audiences interact with visual media.
Abstract
360^circ videos offer a hyper-immersive experience that allows the viewers to explore a dynamic scene from full 360 degrees. To achieve more user-friendly and personalized content creation in 360^circ video format, we seek to lift standard perspective videos into 360^circ equirectangular videos. To this end, we introduce Imagine360, the first perspective-to-360^circ video generation framework that creates high-quality 360^circ videos with rich and diverse motion patterns from video anchors. Imagine360 learns fine-grained spherical visual and motion patterns from limited 360^circ video data with several key designs. 1) Firstly we adopt the dual-branch design, including a perspective and a panorama video denoising branch to provide local and global constraints for 360^circ video generation, with motion module and spatial LoRA layers fine-tuned on extended web 360^circ videos. 2) Additionally, an antipodal mask is devised to capture long-range motion dependencies, enhancing the reversed camera motion between antipodal pixels across hemispheres. 3) To handle diverse perspective video inputs, we propose elevation-aware designs that adapt to varying video masking due to changing elevations across frames. Extensive experiments show Imagine360 achieves superior graphics quality and motion coherence among state-of-the-art 360^circ video generation methods. We believe Imagine360 holds promise for advancing personalized, immersive 360^circ video creation.