SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs
Leheng Li, Weichao Qiu, Yingjie Cai, Xu Yan, Qing Lian, Bingbing Liu, Ying-Cong Chen
2024-10-02
Summary
This paper introduces SyntheOcc, a new method for creating high-quality street view images that accurately represent 3D spaces, which is crucial for improving autonomous driving technology.
What's the problem?
Autonomous driving systems need detailed and accurate data to understand their surroundings, especially for predicting where objects are located in 3D space. However, creating these datasets requires a lot of manual effort to label and annotate the data, which is time-consuming and expensive.
What's the solution?
SyntheOcc addresses this issue by using a diffusion model to generate photorealistic images that are conditioned on occupancy labels, which describe where objects are in a 3D environment. It employs a technique called 3D semantic multi-plane images (MPIs) to provide detailed and spatially aligned descriptions of scenes. This allows SyntheOcc to create a virtually unlimited number of diverse and annotated images that can be used for training perception models in autonomous vehicles.
Why it matters?
This research is important because it significantly reduces the need for manual annotation in creating datasets for autonomous driving. By generating high-quality synthetic data, SyntheOcc can help improve the performance of AI systems in understanding their environment, leading to safer and more reliable autonomous vehicles.
Abstract
The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric-controlled images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. As a result, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.