Towards Trustworthy GUI Agents: A Survey
Yucheng Shi, Wenhao Yu, Wenlin Yao, Wenhu Chen, Ninghao Liu
2025-04-02
Summary
This paper is about making sure AI 'agents' that control computer programs are safe, secure, and reliable.
What's the problem?
As AI agents become more common, there are worries about their potential security holes, privacy risks, and how well they work in changing situations.
What's the solution?
This paper examines the important aspects of making these agents trustworthy, like fixing security problems, making them reliable, and ensuring they are transparent and ethical.
Why it matters?
This work matters because it's essential to create standards and practices for developing AI agents responsibly, so they can be used safely and effectively in the real world.
Abstract
GUI agents, powered by large foundation models, can interact with digital interfaces, enabling various applications in web automation, mobile navigation, and software testing. However, their increasing autonomy has raised critical concerns about their security, privacy, and safety. This survey examines the trustworthiness of GUI agents in five critical dimensions: security vulnerabilities, reliability in dynamic environments, transparency and explainability, ethical considerations, and evaluation methodologies. We also identify major challenges such as vulnerability to adversarial attacks, cascading failure modes in sequential decision-making, and a lack of realistic evaluation benchmarks. These issues not only hinder real-world deployment but also call for comprehensive mitigation strategies beyond task success. As GUI agents become more widespread, establishing robust safety standards and responsible development practices is essential. This survey provides a foundation for advancing trustworthy GUI agents through systematic understanding and future research.