Novel View Extrapolation with Video Diffusion Priors
Kunhao Liu, Ling Shao, Shijian Lu
2024-11-25
Summary
This paper introduces ViewExtrapolator, a new method for creating realistic images from new viewpoints in videos, using advanced techniques to improve the quality of these generated views.
What's the problem?
Many existing methods for generating new views from videos work well when the new views are similar to the original ones but struggle when the new views are very different. This is known as novel view extrapolation, and it can lead to poor quality images that don't accurately represent the scene. Traditional methods often create artifacts or unclear images when trying to generate these far-off views.
What's the solution?
ViewExtrapolator addresses this problem by using a technique called Stable Video Diffusion (SVD) to enhance the quality of the generated images. It redesigns how SVD processes video data to fix issues that cause artifacts in the images. The method allows for generating realistic views even from just one video or image, without needing extensive retraining. This makes it more efficient and capable of working with various types of 3D rendering.
Why it matters?
This research is important because it improves how we can generate images from videos, making it easier to create high-quality visuals for applications like virtual reality, gaming, and film production. By enhancing the ability to extrapolate new views, ViewExtrapolator opens up new possibilities for how we visualize and interact with video content.
Abstract
The field of novel view synthesis has made significant strides thanks to the development of radiance field methods. However, most radiance field techniques are far better at novel view interpolation than novel view extrapolation where the synthesis novel views are far beyond the observed training views. We design ViewExtrapolator, a novel view synthesis approach that leverages the generative priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation. By redesigning the SVD denoising process, ViewExtrapolator refines the artifact-prone views rendered by radiance fields, greatly enhancing the clarity and realism of the synthesized novel views. ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available. Additionally, ViewExtrapolator requires no fine-tuning of SVD, making it both data-efficient and computation-efficient. Extensive experiments demonstrate the superiority of ViewExtrapolator in novel view extrapolation. Project page: https://kunhao-liu.github.io/ViewExtrapolator/.