GSTAR: Gaussian Surface Tracking and Reconstruction
Chengwei Zheng, Lixin Xue, Juan Zarate, Jie Song
2025-01-24
Summary
This paper talks about GSTAR, a new method for creating realistic 3D videos of moving objects. It's like a super advanced way to make 3D animations that look incredibly real and can handle objects changing shape or new objects appearing.
What's the problem?
Current 3D animation techniques are great for still scenes, but they struggle when things start moving and changing. It's like trying to animate a person taking off a jacket - the current methods have a hard time dealing with the jacket suddenly becoming a separate object from the person. They also find it tricky to keep track of where everything is as objects move around.
What's the solution?
The researchers created GSTAR, which uses something called Gaussians (think of them as tiny 3D building blocks) attached to a flexible 3D mesh (like a bendable skeleton). When an object's shape stays the same, GSTAR keeps the Gaussians attached to the mesh and moves everything together. But when an object's shape changes dramatically, like the jacket coming off, GSTAR can detach some Gaussians and create new surfaces. They also came up with a clever way to guess how things will move between video frames, making it easier to keep track of everything.
Why it matters?
This matters because it could revolutionize how we create and use 3D videos. Imagine being able to walk around inside a video game or movie scene that looks completely real, or using virtual reality for ultra-realistic training simulations. It could also help robots better understand and interact with the real world as it changes. By making 3D animations that can handle complex, changing scenes, GSTAR opens up exciting possibilities for entertainment, education, and even how we interact with technology in our daily lives.
Abstract
3D Gaussian Splatting techniques have enabled efficient photo-realistic rendering of static scenes. Recent works have extended these approaches to support surface reconstruction and tracking. However, tracking dynamic surfaces with 3D Gaussians remains challenging due to complex topology changes, such as surfaces appearing, disappearing, or splitting. To address these challenges, we propose GSTAR, a novel method that achieves photo-realistic rendering, accurate surface reconstruction, and reliable 3D tracking for general dynamic scenes with changing topology. Given multi-view captures as input, GSTAR binds Gaussians to mesh faces to represent dynamic objects. For surfaces with consistent topology, GSTAR maintains the mesh topology and tracks the meshes using Gaussians. In regions where topology changes, GSTAR adaptively unbinds Gaussians from the mesh, enabling accurate registration and the generation of new surfaces based on these optimized Gaussians. Additionally, we introduce a surface-based scene flow method that provides robust initialization for tracking between frames. Experiments demonstrate that our method effectively tracks and reconstructs dynamic surfaces, enabling a range of applications. Our project page with the code release is available at https://eth-ait.github.io/GSTAR/.