DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model
Xueyi Liu, He Wang, Li Yi
2025-10-10
Summary
This paper focuses on teaching robots to rotate objects in their hand, a surprisingly difficult task, and making those skills work in the real world, not just in computer simulations.
What's the problem?
Robots struggle to take skills learned in simulated environments and apply them to real-world situations when it comes to manipulating objects. This is because the way objects interact with a robotic hand – the forces and movements – are very complex and hard to perfectly recreate in a simulation. Previous attempts at robotic in-hand rotation were limited to simple objects, specific hand positions, or required a lot of customization, meaning they weren't very versatile.
What's the solution?
The researchers developed a system that learns how a robotic hand interacts with objects in the real world using a relatively small amount of real-world data. Instead of trying to model the entire hand and object interaction at once, they broke it down, looking at each joint in the hand separately and figuring out how it moves based on its own dynamics. They also used a smart way to collect data automatically, so the robot could learn from a variety of interactions without a person constantly guiding it. This allows a single 'policy' (set of instructions) trained in simulation to work with many different objects.
Why it matters?
This work is important because it represents a big step towards robots being able to reliably manipulate a wide range of objects in everyday situations. Being able to rotate objects in hand is a fundamental skill for many tasks, like assembling things, organizing objects, or even just handling tools. This research makes robots more adaptable and useful in the real world, moving beyond just working with simple, pre-defined scenarios.
Abstract
Achieving generalized in-hand object rotation remains a significant challenge in robotics, largely due to the difficulty of transferring policies from simulation to the real world. The complex, contact-rich dynamics of dexterous manipulation create a "reality gap" that has limited prior work to constrained scenarios involving simple geometries, limited object sizes and aspect ratios, constrained wrist poses, or customized hands. We address this sim-to-real challenge with a novel framework that enables a single policy, trained in simulation, to generalize to a wide variety of objects and conditions in the real world. The core of our method is a joint-wise dynamics model that learns to bridge the reality gap by effectively fitting limited amount of real-world collected data and then adapting the sim policy's actions accordingly. The model is highly data-efficient and generalizable across different whole-hand interaction distributions by factorizing dynamics across joints, compressing system-wide influences into low-dimensional variables, and learning each joint's evolution from its own dynamic profile, implicitly capturing these net effects. We pair this with a fully autonomous data collection strategy that gathers diverse, real-world interaction data with minimal human intervention. Our complete pipeline demonstrates unprecedented generality: a single policy successfully rotates challenging objects with complex shapes (e.g., animals), high aspect ratios (up to 5.33), and small sizes, all while handling diverse wrist orientations and rotation axes. Comprehensive real-world evaluations and a teleoperation application for complex tasks validate the effectiveness and robustness of our approach. Website: https://meowuu7.github.io/DexNDM/