ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou
2024-11-27
Summary
This paper introduces ShowUI, a new model designed to improve how computer programs interact with graphical user interfaces (GUIs) by combining vision, language, and action capabilities.
What's the problem?
Most existing GUI assistants rely heavily on language and struggle to understand visual elements like buttons and icons. This makes it difficult for them to perform tasks that require recognizing and interacting with different parts of a screen, limiting their effectiveness in helping users.
What's the solution?
ShowUI addresses this problem by using a vision-language-action model that can analyze screenshots of GUIs. It includes innovations such as UI-Guided Visual Token Selection, which helps the model focus only on important visual elements, and Interleaved Vision-Language-Action Streaming, which allows the model to manage the history of actions and visual inputs efficiently. This means it can better understand what to do based on both what it sees on the screen and what users ask it to do.
Why it matters?
This research is important because it enhances the capabilities of GUI assistants, making them more efficient and user-friendly. By improving how these models understand and interact with visual interfaces, ShowUI can significantly boost productivity in various digital tasks, from web browsing to software applications.
Abstract
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (e.g., HTML or accessibility tree), they show limitations in perceiving UI visuals as humans do, highlighting the need for GUI visual agents. In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances. With above components, ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection further reduces 33% of redundant visual tokens during training and speeds up the performance by 1.4x. Navigation experiments across web Mind2Web, mobile AITW, and online MiniWob environments further underscore the effectiveness and potential of our model in advancing GUI visual agents. The models are available at https://github.com/showlab/ShowUI.