BIG-Bench Extra Hard
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat
2025-02-27
Summary
This paper talks about a new test called BIG-Bench Extra Hard (BBEH) that challenges AI language models to show how well they can think and reason across many different skills, not just math and coding.
What's the problem?
The current tests for AI language models, like BIG-Bench and BIG-Bench Hard, have become too easy. The best AI models are getting almost perfect scores, which means these tests aren't useful anymore for seeing how smart the AI really is or how it can improve.
What's the solution?
The researchers created BBEH by taking each task from the old test (BIG-Bench Hard) and making a new, much harder version of it. These new tasks test the same thinking skills but are way more difficult. They then used BBEH to test different AI models and found that even the best ones struggled, showing there's still a lot of room for improvement.
Why it matters?
This matters because as AI becomes more common in our everyday lives, we need to make sure it can think and reason well about all sorts of things, not just math problems. BBEH helps researchers see where AI still needs to get better at general thinking skills. It also gives them a way to measure progress as they work on making smarter AI that can handle more complex real-world tasks.
Abstract
Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general <PRE_TAG>reasoning capabilities</POST_TAG> and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.