SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation
Kushal Kedia, Tyler Ga Wei Lum, Jeannette Bohg, C. Karen Liu
2026-02-24
Summary
This paper focuses on teaching robots how to use tools, which is a difficult skill because it requires precise movements and the ability to handle objects in different ways.
What's the problem?
Traditionally, getting a robot to use a tool involves a lot of manual work. Researchers need to carefully model each tool and create specific instructions (reward functions) for the robot to follow. This is time-consuming and doesn't easily allow the robot to adapt to new tools or tasks. Also, collecting real-world data to train these robots is hard because it's difficult and potentially unsafe to have a robot learn through trial and error with actual tools.
What's the solution?
The researchers developed a system called SimToolReal. Instead of teaching the robot how to use each tool individually, they created a virtual environment where the robot practices with many different, randomly generated tool-like objects. The robot is given a simple goal: manipulate each object to a random location. By training on this variety of objects in simulation, the robot learns general skills that allow it to use new tools in the real world without any additional training for those specific tools.
Why it matters?
This work is important because it makes robots much more versatile. Instead of needing to be specifically programmed for each task, a robot trained with SimToolReal can pick up and use a wide range of tools 'out of the box'. The system performs better than previous methods and shows strong results when tested with real-world tools in various scenarios, bringing us closer to robots that can help with everyday tasks.
Abstract
The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.