Fast Encoder-Based 3D from Casual Videos via Point Track Processing
Yoni Kasten, Wuyue Lu, Haggai Maron
2025-02-03
Summary
This paper talks about a new method called TracksTo4D that can quickly create 3D models from everyday videos. It's designed to work with regular videos that have moving objects, which has been a tough challenge for existing 3D reconstruction techniques.
What's the problem?
Creating 3D models from videos with moving objects is really hard. Current methods either don't work well with normal videos taken by regular cameras, or they take a very long time to process the video and create the 3D model. This makes it difficult for average people to turn their videos into 3D scenes quickly and easily.
What's the solution?
The researchers developed TracksTo4D, which is like a smart computer program that can understand the movement of points in a video and turn that into a 3D model. It works by looking at how certain points in the video move across different frames, and then uses this information to figure out the 3D structure of the scene and where the camera was positioned. TracksTo4D is trained on lots of everyday videos without needing any pre-existing 3D information, which helps it work well on all kinds of videos.
Why it matters?
This matters because it makes creating 3D models from videos much faster and more accessible. TracksTo4D can create 3D models almost as good as the best current methods, but it does it up to 95% faster. This could be really useful for things like virtual reality, video games, or even helping robots understand the world around them. It also works on videos it hasn't seen before, even if they show completely new types of objects or scenes, which makes it very flexible and useful in the real world.
Abstract
This paper addresses the long-standing challenge of reconstructing 3D structures from videos with dynamic content. Current approaches to this problem were not designed to operate on casual videos recorded by standard cameras or require a long optimization time. Aiming to significantly improve the efficiency of previous approaches, we present TracksTo4D, a learning-based approach that enables inferring 3D structure and camera positions from dynamic content originating from casual videos using a single efficient feed-forward pass. To achieve this, we propose operating directly over 2D point tracks as input and designing an architecture tailored for processing 2D point tracks. Our proposed architecture is designed with two key principles in mind: (1) it takes into account the inherent symmetries present in the input point tracks data, and (2) it assumes that the movement patterns can be effectively represented using a low-rank approximation. TracksTo4D is trained in an unsupervised way on a dataset of casual videos utilizing only the 2D point tracks extracted from the videos, without any 3D supervision. Our experiments show that TracksTo4D can reconstruct a temporal point cloud and camera positions of the underlying video with accuracy comparable to state-of-the-art methods, while drastically reducing runtime by up to 95\%. We further show that TracksTo4D generalizes well to unseen videos of unseen semantic categories at inference time.