C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling

Summary

This paper introduces C2LLM, which stands for Contrastive Code Large Language Models, a new set of models designed to understand and represent code. They come in two sizes, 0.5 billion and 7 billion parameters, and are built using existing code-focused language models.

What's the problem?

Existing methods for creating code embeddings, which are numerical representations of code used for tasks like searching and understanding, often struggle to capture the full meaning of a code snippet. Many rely on just looking at the very end of the code (the 'end-of-sequence' token) which creates a bottleneck – important information from earlier in the code can be lost. Also, it's often difficult to adjust the size of these embeddings to fit different needs.

What's the solution?

The researchers built C2LLM on top of a strong existing code model called Qwen-2.5-Coder. The key innovation is a new module called 'Pooling by Multihead Attention' or PMA. PMA cleverly uses the model's existing ability to understand code relationships to combine information from *all* parts of the code snippet, not just the end. This avoids the information bottleneck and allows for creating embeddings of different sizes without needing complex adjustments.

Why it matters?

C2LLM achieves state-of-the-art results on a benchmark called MTEB-Code, meaning it's currently the best performing model of its size at understanding code. This is important because better code embeddings can lead to improvements in many areas, like code search, code completion, and detecting bugs. The flexibility in embedding size also makes it more adaptable to various applications.