Phi-Ground Tech Report: Advancing Perception in GUI Grounding
Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, Qi Dai, Kai Qiu, Yifan Yang, Chong Luo, Tianyi Chen, Justin Wagle, Tim Franklin, Baining Guo
2025-08-01
Summary
This paper talks about Phi-Ground, a new model family designed to make AI better at understanding and interacting with Graphical User Interfaces (GUIs), like clicking the right buttons on a computer screen.
What's the problem?
The problem is that current AI systems struggle to precisely identify where to click on screens, which is a big challenge because even small mistakes can cause serious problems, and existing models don't perform well enough on this task.
What's the solution?
Phi-Ground solves this by using a two-step process: first, a powerful AI model analyzes the screen and instructions to create a detailed description of where to click; then, a specialized Phi-Ground model uses that description and the screen image to pinpoint the exact coordinates for the click. The model was trained on over 40 million examples from many different sources and uses smart data and training techniques to improve accuracy.
Why it matters?
This matters because better GUI grounding helps create smarter AI assistants that can control computers more like humans do, making technologies like digital helpers and automation more reliable and effective.
Abstract
The Phi-Ground model family achieves state-of-the-art performance in GUI grounding for multimodal reasoning models, improving accuracy across various benchmarks.