DreamCinema: Cinematic Transfer with Free Camera and 3D Character
Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, Yueqi Duan
2024-08-23
Summary
This paper introduces DreamCinema, a new framework that allows users to create films using advanced AI technology to generate realistic characters and cinematic effects.
What's the problem?
Creating high-quality films typically requires a lot of technical skills and resources, which can be difficult for everyday people. Current methods for cinematic transfer still need manual adjustments for characters and scenes, making it complicated and expensive.
What's the solution?
The authors developed DreamCinema, which simplifies the filmmaking process by using AI to automatically generate 3D characters and optimize camera movements. They extract important cinematic elements and use a character generator to create high-quality characters. Additionally, they implement a motion transfer strategy that smoothly integrates these characters into the film, making it easier for users to produce professional-looking videos.
Why it matters?
This research is significant because it democratizes film production, allowing more people to express their creativity without needing extensive technical knowledge or resources. By making filmmaking more accessible, it opens up new opportunities for storytelling in digital media.
Abstract
We are living in a flourishing era of digital media, where everyone has the potential to become a personal filmmaker. Current research on cinematic transfer empowers filmmakers to reproduce and manipulate the visual elements (e.g., cinematography and character behaviors) from classic shots. However, characters in the reimagined films still rely on manual crafting, which involves significant technical complexity and high costs, making it unattainable for ordinary users. Furthermore, their estimated cinematography lacks smoothness due to inadequate capturing of inter-frame motion and modeling of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC has opened up the possibility of efficiently generating characters tailored to users' needs, diversifying cinematography. In this paper, we propose DreamCinema, a novel cinematic transfer framework that pioneers generative AI into the film production paradigm, aiming at facilitating user-friendly film creation. Specifically, we first extract cinematic elements (i.e., human and camera pose) and optimize the camera trajectory. Then, we apply a character generator to efficiently create 3D high-quality characters with a human structure prior. Finally, we develop a structure-guided motion transfer strategy to incorporate generated characters into film creation and transfer it via 3D graphics engines smoothly. Extensive experiments demonstrate the effectiveness of our method for creating high-quality films with free camera and 3D characters.