A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning
Zixin Zhang, Kanghao Chen, Hanqing Wang, Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Litao Guo, Ying-Cong Chen
2025-12-17
Summary
This paper introduces a new way for robots or AI agents to figure out how to interact with objects, based on what a person tells them to do.
What's the problem?
Currently, most AI systems that try to understand how to interact with objects are trained specifically for certain objects and environments. This means they struggle when faced with something new or a different situation. They combine the 'thinking' part of understanding instructions with the 'doing' part of actually interacting with the object all in one step, making it hard to improve each part separately and limiting their ability to generalize.
What's the solution?
The researchers created a system called A4-Agent that breaks down the problem into three separate steps. First, a 'Dreamer' imagines what the interaction would look like. Then, a 'Thinker' uses its understanding of language and vision to decide *where* on the object to interact with. Finally, a 'Spotter' precisely locates that interaction area. Importantly, they didn't train this system specifically for this task; instead, they combined already-trained AI models, each good at its own part, and let them work together.
Why it matters?
This is important because it allows AI agents to interact with objects they've never seen before in environments they haven't been trained in. By using pre-existing AI models and a modular approach, the system is much more flexible and adaptable than previous methods, bringing us closer to robots that can truly understand and respond to our instructions in the real world.
Abstract
Affordance prediction, which identifies interaction regions on objects based on language instructions, is critical for embodied AI. Prevailing end-to-end models couple high-level reasoning and low-level grounding into a single monolithic pipeline and rely on training over annotated datasets, which leads to poor generalization on novel objects and unseen environments. In this paper, we move beyond this paradigm by proposing A4-Agent, a training-free agentic framework that decouples affordance prediction into a three-stage pipeline. Our framework coordinates specialized foundation models at test time: (1) a Dreamer that employs generative models to visualize how an interaction would look; (2) a Thinker that utilizes large vision-language models to decide what object part to interact with; and (3) a Spotter that orchestrates vision foundation models to precisely locate where the interaction area is. By leveraging the complementary strengths of pre-trained models without any task-specific fine-tuning, our zero-shot framework significantly outperforms state-of-the-art supervised methods across multiple benchmarks and demonstrates robust generalization to real-world settings.