DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction
Xiaobiao Du, Haiyang Sun, Ming Lu, Tianqing Zhu, Xin Yu
2024-07-25
Summary
This paper introduces DreamCar, a new method for creating detailed 3D models of cars from images, even if only one image is available. It addresses the challenges of existing methods that struggle with real-world images and aims to improve the efficiency and quality of car reconstruction.
What's the problem?
Creating high-quality 3D models of cars is usually done by professional artists, which is very expensive. Existing methods rely on datasets that often only show one side of a car, making it hard to reconstruct complete models. Additionally, these methods struggle with real-world images due to errors in camera positioning and lack of diverse training data, leading to poor quality and misaligned textures in the reconstructed models.
What's the solution?
The authors developed DreamCar, which uses a large dataset called Car360 containing over 5,600 car images to improve the model's understanding of car shapes. DreamCar employs techniques like Score Distillation Sampling to guide the reconstruction process and uses the symmetry of cars to generate additional training views. It also includes a pose optimization method to correct camera positioning errors, allowing for better texture alignment and overall model quality. This approach enables high-quality 3D car reconstruction from just a few images or even a single image.
Why it matters?
This research is important because it reduces the cost and effort needed to create realistic 3D car models, which can be used in self-driving technology, video games, and virtual reality. By improving how we can reconstruct cars from limited data, DreamCar opens up new possibilities for realistic simulations and applications in various industries.
Abstract
Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. https://xiaobiaodu.github.io/dreamcar-project/{Our code is available.}