StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams

Summary

This paper talks about StreamSplat, a new AI system that can build detailed 3D models of moving scenes in real time using regular video streams without needing the cameras to be specially set up or calibrated. It creates these 3D models by transforming the video frames into a special 3D representation called Gaussian Splatting, which captures both the shape and motion of objects dynamically and accurately.

What's the problem?

The problem is that it is very hard for AI to quickly and accurately create 3D models from videos where the camera settings are unknown or changing, especially when the scenes are dynamic with objects moving. Existing methods either need lots of processing after the videos are captured, can't handle long videos well, or struggle to keep the 3D models stable over time without errors accumulating.

What's the solution?

The solution was to design StreamSplat as a fully feed-forward system that processes each video frame as it comes in, without heavy extra computation. It uses a probabilistic sampling technique in the part that encodes static scene details and a special bidirectional deformation field in the part modeling motion. This approach allows the model to handle movement smoothly, combine information over time, and keep the reconstructed 3D scenes stable even for long video streams, while working in near real-time.

Why it matters?

This matters because being able to quickly and reliably build 3D models of changing scenes from ordinary videos opens up many practical uses, like improving virtual reality experiences, helping robots understand their environment better, enabling safer autonomous driving, and making video editing or generation more powerful. StreamSplat's ability to work online with uncalibrated videos makes it very flexible and useful for real-world applications.