OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

Summary

This paper introduces a really tough test, called OfficeQA Pro, designed to see how well AI systems can understand and reason with a huge collection of real-world documents – specifically, almost a century’s worth of U.S. Treasury reports.

What's the problem?

Current AI models, even the most advanced ones like Claude, GPT, and Gemini, struggle with tasks that require them to deeply understand complex documents and pull out specific information. They often rely on what they’ve already been trained on, which isn’t enough when dealing with specialized or constantly updated information. Even when given access to the internet, they still aren’t very accurate when asked questions based on these detailed reports, and they have a hard time with both the text *and* the tables within the documents.

What's the solution?

The researchers created OfficeQA Pro, a set of 133 challenging questions based on the Treasury reports. They then tested different AI models on these questions, both with and without access to the reports themselves. They also experimented with giving the AI a more organized version of the documents, created by a tool called ai_parse_document, to see if that helped. They also looked at how different choices about the AI model, how tables are represented, and how information is retrieved affected the results.

Why it matters?

This work is important because it shows that even the best AI systems aren’t yet reliable enough to handle complex, real-world reasoning tasks that require understanding large amounts of detailed information. It highlights the need for better ways to help AI systems process and understand documents, which is crucial for things like financial analysis, legal research, and other important applications where accuracy is essential.