Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding
Joshua Jones, Oier Mees, Carmelo Sferrazza, Kyle Stachowicz, Pieter Abbeel, Sergey Levine
2025-01-16
Summary
This paper talks about FuSe, a new way to teach robots to use multiple senses like touch and hearing, not just vision, when interacting with the world. It's like teaching a robot to use all its senses, just like humans do, to understand and interact with its environment better.
What's the problem?
Current advanced robots are mainly trained to use vision to understand the world and decide what to do. This is like trying to navigate a dark room using only your eyes - it doesn't work well. In real life, we use all our senses together, especially when we can't see clearly. Robots need to learn to do this too, but it's hard because there isn't much training data available for senses other than vision.
What's the solution?
The researchers created FuSe, which is a clever way to teach robots to use multiple senses. They use language as a bridge to connect different senses. For example, they might teach the robot that both the visual image of a soft toy and the feeling of squishiness when touching it are related to the word 'soft'. This helps the robot understand how different senses relate to each other and to language descriptions. They tested this method on different types of robot systems and found it worked well across the board.
Why it matters?
This matters because it could make robots much more capable and flexible in the real world. Imagine a robot that can not only see but also feel and hear, just like humans. This could lead to robots that can help in more complex situations, like assisting in healthcare where touch is important, or in search and rescue operations where robots might need to listen for survivors. It's a big step towards making robots that can truly understand and interact with the world in a human-like way.
Abstract
Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch, and audio -- to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.