Embodied Red Teaming for Auditing Robotic Foundation Models
Sathwik Karnik, Zhang-Wei Hong, Nishant Abhangi, Yen-Chen Lin, Tsun-Hsuan Wang, Christophe Dupuy, Rahul Gupta, Pulkit Agrawal
2025-02-11
Summary
This paper talks about Embodied Red Teaming (ERT), a new way to test how well robots can understand and follow spoken instructions while staying safe.
What's the problem?
Current methods for testing robots that follow voice commands have two big issues. First, they only use a small set of instructions that humans come up with, which doesn't cover all the tricky situations a robot might face. Second, these tests only check if the robot can do the task, not whether it can do it safely without causing damage.
What's the solution?
The researchers created ERT, which uses AI to come up with a wide range of challenging instructions for robots. This system uses special AI models that understand both language and images to create realistic, difficult commands. They then tested top-of-the-line robots with these instructions to see how well they performed.
Why it matters?
This matters because as robots become more common in our daily lives, we need to make sure they're safe and reliable. ERT shows that even the best robots today can fail or act unsafely when given tricky instructions, which means we need better ways to test and improve them. By finding these problems now, we can make robots safer and more useful in the real world.
Abstract
Language-conditioned robot models have the potential to enable robots to perform a wide range of tasks based on natural language instructions. However, assessing their safety and effectiveness remains challenging because it is difficult to test all the different ways a single task can be phrased. Current benchmarks have two key limitations: they rely on a limited set of human-generated instructions, missing many challenging cases, and focus only on task performance without assessing safety, such as avoiding damage. To address these gaps, we introduce Embodied Red Teaming (ERT), a new evaluation method that generates diverse and challenging instructions to test these models. ERT uses automated red teaming techniques with Vision Language Models (VLMs) to create contextually grounded, difficult instructions. Experimental results show that state-of-the-art language-conditioned robot models fail or behave unsafely on ERT-generated instructions, underscoring the shortcomings of current benchmarks in evaluating real-world performance and safety. Code and videos are available at: https://s-karnik.github.io/embodied-red-team-project-page.