TreeMeshGPT: Artistic Mesh Generation with Autoregressive Tree Sequencing
Stefan Lionar, Jiabin Liang, Gim Hee Lee
2025-03-17
Summary
This paper introduces TreeMeshGPT, a new AI model that generates 3D artistic shapes (meshes) from point clouds, which are sets of 3D points. It's like sculpting a digital object based on a 3D scan.
What's the problem?
Generating high-quality 3D meshes that match a given point cloud is difficult. Traditional methods often struggle to create detailed and accurate shapes, and they can produce meshes with flipped normals, which are like having the surface facing the wrong way.
What's the solution?
TreeMeshGPT uses a novel approach called Autoregressive Tree Sequencing. Instead of predicting the next shape element in a fixed order, it builds a tree-like structure based on how the triangles in the mesh connect. This allows the model to extend the mesh locally, creating more detailed and accurate shapes. The method also compresses the data, making it more efficient.
Why it matters?
This work matters because it enables the creation of high-quality 3D artistic meshes from point clouds, which can be used in various applications, such as creating 3D models for games, movies, or product design.
Abstract
We introduce TreeMeshGPT, an autoregressive Transformer designed to generate high-quality artistic meshes aligned with input point clouds. Instead of the conventional next-token prediction in autoregressive Transformer, we propose a novel Autoregressive Tree Sequencing where the next input token is retrieved from a dynamically growing tree structure that is built upon the triangle adjacency of faces within the mesh. Our sequencing enables the mesh to extend locally from the last generated triangular face at each step, and therefore reduces training difficulty and improves mesh quality. Our approach represents each triangular face with two tokens, achieving a compression rate of approximately 22% compared to the naive face tokenization. This efficient tokenization enables our model to generate highly detailed artistic meshes with strong point cloud conditioning, surpassing previous methods in both capacity and fidelity. Furthermore, our method generates mesh with strong normal orientation constraints, minimizing flipped normals commonly encountered in previous methods. Our experiments show that TreeMeshGPT enhances the mesh generation quality with refined details and normal orientation consistency.