What If : Understanding Motion Through Sparse Interactions
Stefan Andreas Baumann, Nick Stracke, Timy Phan, Björn Ommer
2025-10-15
Summary
This paper introduces a new way to predict how things will move in a scene, focusing on understanding all the *possible* ways things could change, not just one specific outcome.
What's the problem?
Traditionally, computer models struggle to predict the full range of possible motions in a dynamic scene. They often only give one predicted movement, and don't easily show how uncertain that prediction is or how changes to the scene (like a 'poke' or interaction) would affect things. Existing methods also don't clearly show *why* a scene is moving in a certain way.
What's the solution?
The researchers developed something called the Flow Poke Transformer, or FPT. This system predicts a *distribution* of possible movements, meaning it shows all the likely ways a scene could evolve. It does this by looking at how 'pokes' – small interactions – affect the scene. FPT creates a clear, understandable representation of how motion depends on these interactions and acknowledges the inherent uncertainty in predicting physical movements. It's also designed to be adaptable and can be improved with new data.
Why it matters?
This work is important because it allows for more realistic and flexible simulations of physical scenes. The FPT model performs well on tasks like generating realistic facial movements and estimating how objects will move, even in situations it wasn't specifically trained for. Being able to predict a range of possible motions, and understand *why* those motions occur, is crucial for applications like robotics, animation, and understanding how the physical world works.
Abstract
Understanding the dynamics of a physical scene involves reasoning about the diverse ways it can potentially change, especially as a result of local interactions. We present the Flow Poke Transformer (FPT), a novel framework for directly predicting the distribution of local motion, conditioned on sparse interactions termed "pokes". Unlike traditional methods that typically only enable dense sampling of a single realization of scene dynamics, FPT provides an interpretable directly accessible representation of multi-modal scene motion, its dependency on physical interactions and the inherent uncertainties of scene dynamics. We also evaluate our model on several downstream tasks to enable comparisons with prior methods and highlight the flexibility of our approach. On dense face motion generation, our generic pre-trained model surpasses specialized baselines. FPT can be fine-tuned in strongly out-of-distribution tasks such as synthetic datasets to enable significant improvements over in-domain methods in articulated object motion estimation. Additionally, predicting explicit motion distributions directly enables our method to achieve competitive performance on tasks like moving part segmentation from pokes which further demonstrates the versatility of our FPT. Code and models are publicly available at https://compvis.github.io/flow-poke-transformer.