Statically Contextualizing Large Language Models with Typed Holes
Andrew Blinn, Xiang Li, June Hyung Kim, Cyrus Omar
2024-09-06
Summary
This paper talks about a new method called Statically Contextualizing Large Language Models with Typed Holes, which helps improve code completion by providing better context for large language models (LLMs).
What's the problem?
Current code completion systems using LLMs often produce incorrect or broken code because they don't have enough context about the programming environment or the specific definitions being used. This is especially problematic when the definitions are not in the training data or are far from where the model is trying to generate code.
What's the solution?
To solve this problem, the authors propose integrating LLMs with a programming environment called Hazel, which provides detailed information about types and bindings in the code. This integration allows the model to access relevant information from the entire codebase, not just what is immediately next to the cursor. They also developed a dataset called MVUBench to test their methods, showing that using type information significantly improves the model's performance in generating accurate code.
Why it matters?
This research is important because it enhances how AI can assist programmers by making code completion more reliable and context-aware. By improving these systems, developers can write code faster and with fewer errors, ultimately making programming more efficient and accessible.
Abstract
Large language models (LLMs) have reshaped the landscape of program synthesis. However, contemporary LLM-based code completion systems often hallucinate broken code because they lack appropriate context, particularly when working with definitions not in the training data nor near the cursor. This paper demonstrates that tight integration with the type and binding structure of a language, as exposed by its language server, can address this contextualization problem in a token-efficient manner. In short, we contend that AIs need IDEs, too! In particular, we integrate LLM code generation into the Hazel live program sketching environment. The Hazel Language Server identifies the type and typing context of the hole being filled, even in the presence of errors, ensuring that a meaningful program sketch is always available. This allows prompting with codebase-wide contextual information not lexically local to the cursor, nor necessarily in the same file, but that is likely to be semantically local to the developer's goal. Completions synthesized by the LLM are then iteratively refined via further dialog with the language server. To evaluate these techniques, we introduce MVUBench, a dataset of model-view-update (MVU) web applications. These applications serve as challenge problems due to their reliance on application-specific data structures. We find that contextualization with type definitions is particularly impactful. After introducing our ideas in the context of Hazel we duplicate our techniques and port MVUBench to TypeScript in order to validate the applicability of these methods to higher-resource languages. Finally, we outline ChatLSP, a conservative extension to the Language Server Protocol (LSP) that language servers can implement to expose capabilities that AI code completion systems of various designs can use to incorporate static context when generating prompts for an LLM.