UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models
Jiajun Wu, Jian Yang, Wei Zhang, Lin Jing, Yuqing Ma, Ensheng Shi, Yuchi Ma, Zhoujun Li, Xianglong Liu
2025-12-23
Summary
This paper explores a new way to teach large language models (LLMs) how to write code, without needing huge amounts of example code to learn from.
What's the problem?
Currently, getting LLMs to generate good code requires a lot of training data – either examples of questions and answers related to code, or massive collections of code itself. Creating and managing these datasets is expensive and time-consuming, limiting who can build powerful coding LLMs.
What's the solution?
The researchers developed a technique called IPC, which stands for Internal Probing of LLMs for Code generation. Instead of relying on external data, IPC looks *inside* the LLM itself to understand what it already knows about code. It does this by asking the model different kinds of questions about code, checking its confidence, and then using that information to improve its coding abilities. They then used this process to train a new model called UCoder, which learns to write code without needing labeled examples.
Why it matters?
This research is important because it shows that LLMs already contain a lot of useful knowledge about code, and we can unlock that knowledge without needing massive datasets or a lot of computing power. This means more people and organizations can develop and use powerful code-generating LLMs, even with limited resources, and it opens up new possibilities for how we train these models in the future.
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, their effectiveness heavily relies on supervised training with extensive labeled (e.g., question-answering pairs) or unlabeled datasets (e.g., code snippets), which are often expensive and difficult to obtain at scale. To address this limitation, this paper introduces a method IPC, an unsupervised framework that leverages Internal Probing of LLMs for Code generation without any external corpus, even unlabeled code snippets. We introduce the problem space probing, test understanding probing, solution space probing, and knowledge consolidation and reinforcement to probe the internal knowledge and confidence patterns existing in LLMs. Further, IPC identifies reliable code candidates through self-consistency mechanisms and representation-based quality estimation to train UCoder (coder with unsupervised learning). We validate the proposed approach across multiple code benchmarks, demonstrating that unsupervised methods can achieve competitive performance compared to supervised approaches while significantly reducing the dependency on labeled data and computational resources. Analytic experiments reveal that internal model states contain rich signals about code quality and correctness, and that properly harnessing these signals enables effective unsupervised learning for code generation tasks, opening new directions for training code LLMs in resource-constrained scenarios.