TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar

Summary

This paper investigates how the way code is broken down into smaller pieces, called tokens, affects how well large language models (LLMs) understand and work with code.

What's the problem?

LLMs for code use a method of splitting code into tokens based on how often certain sequences of characters appear together, similar to how words are formed. This method doesn't understand the actual rules of programming languages, meaning code that *means* the same thing but is formatted differently – like having different spaces or variable names – can be tokenized in completely different ways. This inconsistency can confuse the LLM and lead to unpredictable results.

What's the solution?

The researchers created a tool called TokDrift that automatically changes code in ways that don't alter its meaning but *do* change its tokenization. They then tested nine different code LLMs, including some very large ones, with these slightly altered code versions. They found that even small changes in formatting could significantly change how the models behaved, and pinpointed that the issue starts very early in the model's processing, when the code is first converted into numerical representations.

Why it matters?

This research shows that the way code is tokenized is a hidden problem that impacts the reliability of code LLMs. It suggests that future LLMs designed for code should use methods that are aware of programming language grammar, rather than relying solely on statistical patterns, to ensure they consistently understand and generate code correctly.