xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluations
Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, Chen Sun, Han Hou, Hui Yang, James Pan, Jianan Lou, Jiayi Mao, Jizheng Liu, Jinpeng Li, Kangyi Liu, Kenkun Liu, Rui Wang, Run Li
2025-06-18
Summary
This paper talks about xbench, a new way to test AI agents by making their evaluations match real-world professional tasks, such as in recruitment and marketing, to better understand how useful these agents really are in business settings.
What's the problem?
The problem is that most current tests for AI focus only on small technical skills and don’t show how well AI agents perform in actual jobs, which means they don’t provide a clear picture of the economic value these agents can deliver to companies.
What's the solution?
The researchers created xbench to use tasks from real professional workflows, working with experts to design tests that reflect true business needs. They focus on tasks important to real industries and measure how well AI helps with actual productivity. xbench also updates its evaluations over time to keep up with AI improvements and changing industry demands.
Why it matters?
This matters because it helps businesses and developers see which AI systems will actually be valuable in the real world, allowing better decisions on using AI to improve productivity and meet real professional challenges.
Abstract
We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity. While existing benchmarks often focus on isolated technical skills, they may not accurately reflect the economic value agents deliver in professional settings. To address this, xbench targets commercially significant domains with evaluation tasks defined by industry professionals. Our framework creates metrics that strongly correlate with productivity value, enables prediction of Technology-Market Fit (TMF), and facilitates tracking of product capabilities over time. As our initial implementations, we present two benchmarks: Recruitment and Marketing. For Recruitment, we collect 50 tasks from real-world headhunting business scenarios to evaluate agents' abilities in company mapping, information retrieval, and talent sourcing. For Marketing, we assess agents' ability to match influencers with advertiser needs, evaluating their performance across 50 advertiser requirements using a curated pool of 836 candidate influencers. We present initial evaluation results for leading contemporary agents, establishing a baseline for these professional domains. Our continuously updated evalsets and evaluations are available at https://xbench.org.