What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
Alexis Audran-Reiss, Jordi Armengol Estapé, Karen Hambardzumyan, Amar Budhiraja, Martin Josifoski, Edan Toledo, Rishi Hazra, Despoina Magka, Michael Shvartsman, Parth Pathak, Justine T Kao, Lucia Cipolina-Kun, Bhavul Gauri, Jean-Christophe Gagnon-Audet, Emanuel Tewolde, Jenny Zhang, Taco Cohen, Yossi Adi, Tatiana Shavrina, Yoram Bachrach
2025-11-20
Summary
This paper investigates how creative AI agents are when trying to solve machine learning problems, and whether being more creative helps them succeed.
What's the problem?
AI is being developed to *do* AI research – essentially, to automatically build and improve machine learning models. However, we don't really understand what makes these 'AI research agents' good at their job. Specifically, it's unclear if the variety of ideas an agent explores during its work impacts how well it ultimately performs.
What's the solution?
The researchers looked at how different AI agents approached a standard challenge called MLE-bench. They noticed that agents that tried a wider range of approaches – showing more 'ideation diversity' – tended to do better. To confirm this, they even tweaked some agents to be more or less creative and saw that the more creative agents consistently outperformed the others, even when looking at different ways to measure success beyond just winning the challenge.
Why it matters?
This research is important because it highlights that simply making AI agents 'smarter' isn't enough. Encouraging them to explore a diverse set of ideas is crucial for making progress in machine learning and potentially accelerating scientific discovery. It suggests that building AI to be creatively exploratory is a key ingredient for success.
Abstract
AI research agents offer the promise to accelerate scientific progress by automating the design, implementation, and training of machine learning models. However, the field is still in its infancy, and the key factors driving the success or failure of agent trajectories are not fully understood. We examine the role that ideation diversity plays in agent performance. First, we analyse agent trajectories on MLE-bench, a well-known benchmark to evaluate AI research agents, across different models and agent scaffolds. Our analysis reveals that different models and agent scaffolds yield varying degrees of ideation diversity, and that higher-performing agents tend to have increased ideation diversity. Further, we run a controlled experiment where we modify the degree of ideation diversity, demonstrating that higher ideation diversity results in stronger performance. Finally, we strengthen our results by examining additional evaluation metrics beyond the standard medal-based scoring of MLE-bench, showing that our findings still hold across other agent performance metrics.