DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning
Ruowen Zhao, Junliang Ye, Zhengyi Wang, Guangce Liu, Yiwen Chen, Yikai Wang, Jun Zhu
2025-03-20
Summary
This paper discusses a new method for creating 3D models automatically using AI. It focuses on creating models made of triangles, which are useful for things like video games and 3D design.
What's the problem?
Existing methods for creating 3D models automatically often struggle to create detailed models with the right shape and can be limited in the complexity of the models they produce.
What's the solution?
The researchers developed a system called DeepMesh that uses a new way to break down 3D shapes into smaller parts and reinforcement learning to improve the quality and detail of the models.
Why it matters?
This work matters because it can make it easier and faster to create high-quality 3D models for various applications, like games, movies, and engineering.
Abstract
Triangle meshes play a crucial role in 3D applications for efficient manipulation and rendering. While auto-regressive methods generate structured meshes by predicting discrete vertex tokens, they are often constrained by limited face counts and mesh incompleteness. To address these challenges, we propose DeepMesh, a framework that optimizes mesh generation through two key innovations: (1) an efficient pre-training strategy incorporating a novel tokenization algorithm, along with improvements in data curation and processing, and (2) the introduction of Reinforcement Learning (RL) into 3D mesh generation to achieve human preference alignment via Direct Preference Optimization (DPO). We design a scoring standard that combines human evaluation with 3D metrics to collect preference pairs for DPO, ensuring both visual appeal and geometric accuracy. Conditioned on point clouds and images, DeepMesh generates meshes with intricate details and precise topology, outperforming state-of-the-art methods in both precision and quality. Project page: https://zhaorw02.github.io/DeepMesh/