DragMesh: Interactive 3D Generation Made Easy
Tianshan Zhang, Zeyu Zhang, Hao Tang
2025-12-12
Summary
This paper introduces DragMesh, a new system for realistically manipulating 3D objects in real-time. It focuses on making objects move in a natural way when a user interacts with them, like dragging a virtual arm or leg.
What's the problem?
Currently, making 3D objects move convincingly is difficult. Existing methods either prioritize physical accuracy, which is slow and not practical for immediate interaction, or they generate movement quickly but don't respect the natural limits of how joints work – think of an elbow bending backwards. There's a trade-off between speed and realism, and neither side is fully satisfied.
What's the solution?
The researchers tackled this by separating how the system *understands* what the user wants to do from how it *creates* the actual motion. First, they predict the type of joint involved (like a hinge or a ball joint) and where it's located. Then, they use a special mathematical tool called 'dual quaternions' to create smooth, realistic movements that avoid awkward positions. They feed the user’s drag input and their joint predictions into a system that generates the motion, constantly guiding it to stay within realistic boundaries at every step. This combination allows for fast, believable movement.
Why it matters?
DragMesh is important because it brings us closer to truly interactive 3D environments. It allows for real-time manipulation of 3D objects without needing to retrain the system for each new object, making it a practical step towards creating more intelligent and responsive 3D experiences. This could be useful for things like video games, virtual reality, or even designing products in 3D.
Abstract
While generative models have excelled at creating static 3D content, the pursuit of systems that understand how objects move and respond to interactions remains a fundamental challenge. Current methods for articulated motion lie at a crossroads: they are either physically consistent but too slow for real-time use, or generative but violate basic kinematic constraints. We present DragMesh, a robust framework for real-time interactive 3D articulation built around a lightweight motion generation core. Our core contribution is a novel decoupled kinematic reasoning and motion generation framework. First, we infer the latent joint parameters by decoupling semantic intent reasoning (which determines the joint type) from geometric regression (which determines the axis and origin using our Kinematics Prediction Network (KPP-Net)). Second, to leverage the compact, continuous, and singularity-free properties of dual quaternions for representing rigid body motion, we develop a novel Dual Quaternion VAE (DQ-VAE). This DQ-VAE receives these predicted priors, along with the original user drag, to generate a complete, plausible motion trajectory. To ensure strict adherence to kinematics, we inject the joint priors at every layer of the DQ-VAE's non-autoregressive Transformer decoder using FiLM (Feature-wise Linear Modulation) conditioning. This persistent, multi-scale guidance is complemented by a numerically-stable cross-product loss to guarantee axis alignment. This decoupled design allows DragMesh to achieve real-time performance and enables plausible, generative articulation on novel objects without retraining, offering a practical step toward generative 3D intelligence. Code: https://github.com/AIGeeksGroup/DragMesh. Website: https://aigeeksgroup.github.io/DragMesh.