CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Wenmeng Zhou, Fei Wang, Michael Shieh
2024-08-08
Summary
This paper introduces CodexGraph, a new system that helps large language models (LLMs) work better with code repositories by using graph databases to organize and access code more efficiently.
What's the problem?
While LLMs are great at performing specific coding tasks, they struggle when it comes to understanding and navigating entire code repositories. Current methods for helping LLMs interact with code often rely on comparing similar pieces of code or using manual tools, which can be inefficient and require expert knowledge. This makes it hard for LLMs to generalize their skills across different coding tasks and real-world applications.
What's the solution?
CodexGraph addresses these issues by integrating LLMs with graph databases that store information about the code in a structured way. This allows the LLM to create and execute queries that can retrieve relevant code snippets based on their structure and relationships. The system was tested using three benchmarks and demonstrated strong performance, showing that it can effectively help LLMs understand and navigate complex codebases.
Why it matters?
This research is significant because it enhances how AI models can interact with large sets of code, making them more useful for software development. By improving the ability of LLMs to understand and retrieve information from code repositories, CodexGraph could lead to more efficient coding tools and better support for developers in various programming tasks.
Abstract
Large Language Models (LLMs) excel in stand-alone code tasks like HumanEval and MBPP, but struggle with handling entire code repositories. This challenge has prompted research on enhancing LLM-codebase interaction at a repository scale. Current solutions rely on similarity-based retrieval or manual tools and APIs, each with notable drawbacks. Similarity-based retrieval often has low recall in complex tasks, while manual tools and APIs are typically task-specific and require expert knowledge, reducing their generalizability across diverse code tasks and real-world applications. To mitigate these limitations, we introduce \framework, a system that integrates LLM agents with graph database interfaces extracted from code repositories. By leveraging the structural properties of graph databases and the flexibility of the graph query language, \framework enables the LLM agent to construct and execute queries, allowing for precise, code structure-aware context retrieval and code navigation. We assess \framework using three benchmarks: CrossCodeEval, SWE-bench, and EvoCodeBench. Additionally, we develop five real-world coding applications. With a unified graph database schema, \framework demonstrates competitive performance and potential in both academic and real-world environments, showcasing its versatility and efficacy in software engineering. Our application demo: https://github.com/modelscope/modelscope-agent/tree/master/apps/codexgraph_agent.