Magma: A Foundation Model for Multimodal AI Agents
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao
2025-02-19
Summary
This paper talks about Magma, a new AI model that can understand and act in both digital and physical environments. It's like a super-smart computer program that can see, understand, and interact with things on screens as well as control robots in the real world.
What's the problem?
Current AI models are usually good at either understanding what they see or doing specific tasks, but not both. They struggle to work across different types of environments, like navigating a website and controlling a robot arm, which limits how useful they can be in real-world situations.
What's the solution?
The researchers created Magma, which learns from a wide variety of data including images, videos, and information from robots. They used special techniques called Set-of-Mark and Trace-of-Mark to teach Magma how to identify important objects it can interact with and how to plan movements. This helps Magma understand both what it sees and how to act on that information, whether it's clicking buttons on a screen or moving objects with a robot arm.
Why it matters?
This matters because Magma represents a big step towards more versatile AI that can help with many different tasks in our daily lives. It could lead to better virtual assistants that can navigate complex websites for us, or robots that can understand and carry out more complicated instructions in homes or workplaces. By bridging the gap between understanding and action in both digital and physical worlds, Magma opens up new possibilities for AI to assist humans in more meaningful and practical ways.
Abstract
We present Magma, a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets. We make our model and code public for reproducibility at https://microsoft.github.io/Magma.