Noise Calibration: Plug-and-play Content-Preserving Video Enhancement using Pre-trained Video Diffusion Models
Qinyu Yang, Haoxin Chen, Yong Zhang, Menghan Xia, Xiaodong Cun, Zhixun Su, Ying Shan
2024-07-16
Summary
This paper presents a new method called Noise Calibration that enhances video quality while keeping the original content intact, using pre-trained video diffusion models.
What's the problem?
Improving the quality of videos usually requires a lot of training and can lead to changes in the original content, which might not be desirable. Existing methods often focus on either enhancing visual quality or maintaining content consistency, but they struggle to do both effectively without significant costs.
What's the solution?
The authors propose Noise Calibration, a method that balances visual quality and content consistency. It uses a special loss function to ensure that the structure of the original video is preserved while enhancing its quality through a denoising process. This method allows for quick adjustments to initial random noise, refining it over just a few iterations to keep the video's original content while improving its overall appearance.
Why it matters?
This research is important because it provides an efficient way to enhance video quality without losing the essence of the original footage. By allowing users to improve videos easily and effectively, this method could be useful in various applications, such as restoring old films, enhancing security camera footage, or improving low-quality videos for better viewing experiences.
Abstract
In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation that considers both visual quality and consistency of content. Consistency of content is ensured by a proposed loss function that maintains the structure of the input, while visual quality is improved by utilizing the denoising process of pretrained diffusion models. To address the formulated optimization problem, we have developed a plug-and-play noise optimization strategy, referred to as Noise Calibration. By refining the initial random noise through a few iterations, the content of original video can be largely preserved, and the enhancement effect demonstrates a notable improvement. Extensive experiments have demonstrated the effectiveness of the proposed method.