AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos
Yuze He, Wang Zhao, Shaohui Liu, Yubin Hu, Yushi Bai, Yu-Hui Wen, Yong-Jin Liu
2024-12-02
Summary
This paper introduces AlphaTablets, a new method for representing and reconstructing 3D planes from videos using a unique approach that combines the benefits of both 2D and 3D representations.
What's the problem?
When trying to create 3D models from videos, existing methods often struggle with accurately capturing the shapes and boundaries of planes. Traditional approaches can be either too rigid or not detailed enough, leading to poor-quality reconstructions that lack precision and clarity.
What's the solution?
AlphaTablets solves this problem by representing 3D planes as rectangles with alpha channels, which allows for smooth surfaces and clear edges. The method uses a process called differentiable rasterization to convert these representations into images effectively. It starts with 2D segments from the video and uses geometric information to initialize 3D planes as AlphaTablets. The model then refines these planes through a process of optimization and merging, resulting in accurate and complete 3D representations. Extensive testing on the ScanNet dataset shows that AlphaTablets performs better than existing methods.
Why it matters?
This research is important because it provides a more effective way to reconstruct 3D planes from videos, which can be applied in various fields such as virtual reality, gaming, and architectural modeling. By improving the accuracy and quality of these reconstructions, AlphaTablets can enhance how we create and interact with 3D environments.
Abstract
We introduce AlphaTablets, a novel and generic representation of 3D planes that features continuous 3D surface and precise boundary delineation. By representing 3D planes as rectangles with alpha channels, AlphaTablets combine the advantages of current 2D and 3D plane representations, enabling accurate, consistent and flexible modeling of 3D planes. We derive differentiable rasterization on top of AlphaTablets to efficiently render 3D planes into images, and propose a novel bottom-up pipeline for 3D planar reconstruction from monocular videos. Starting with 2D superpixels and geometric cues from pre-trained models, we initialize 3D planes as AlphaTablets and optimize them via differentiable rendering. An effective merging scheme is introduced to facilitate the growth and refinement of AlphaTablets. Through iterative optimization and merging, we reconstruct complete and accurate 3D planes with solid surfaces and clear boundaries. Extensive experiments on the ScanNet dataset demonstrate state-of-the-art performance in 3D planar reconstruction, underscoring the great potential of AlphaTablets as a generic 3D plane representation for various applications. Project page is available at: https://hyzcluster.github.io/alphatablets