Agentless: Demystifying LLM-based Software Engineering Agents
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang
2024-07-03
Summary
This paper talks about a new method called Agentless that simplifies how software development problems are solved using large language models (LLMs) without relying on complex autonomous agents.
What's the problem?
The main problem is that current methods for automating software development often use complicated agents that can be hard to manage and understand. These agents need to make decisions, use various tools, and respond to feedback, which can lead to inefficiencies and higher costs.
What's the solution?
To address this issue, the authors developed Agentless, which uses a straightforward two-step process: first, it identifies where the problems are in the code (localization), and then it generates fixes for those problems (repair). This approach avoids letting the LLM make independent decisions or use complicated tools. The results showed that Agentless performed better than existing methods, fixing 27.33% of issues at a low cost of $0.34 per problem.
Why it matters?
This research is important because it demonstrates that a simpler approach can be more effective than complex systems in software development. By showing that Agentless can achieve high performance with lower costs, it suggests a new direction for future tools in software engineering, making automation more accessible and efficient.
Abstract
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic two-phase process of localization followed by repair, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (27.33%) and lowest cost (\$0.34) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.