State over Tokens: Characterizing the Role of Reasoning Tokens
Mosh Levy, Zohar Elyoseph, Shauli Ravfogel, Yoav Goldberg
2025-12-16
Summary
This paper explores how large language models, or LLMs, appear to 'think' when solving problems, and argues that what looks like reasoning isn't necessarily what's actually happening inside the model.
What's the problem?
LLMs often generate a series of steps or 'reasoning tokens' before giving a final answer, which seems similar to how humans explain their thought process. However, the paper points out that these tokens don't actually reflect the true way the model is reaching its conclusion; they're more like a byproduct of the computation than a genuine explanation of it. This creates a disconnect between how the model *seems* to reason and how it *actually* reasons.
What's the solution?
The researchers propose a new way of thinking about these reasoning tokens, called 'State over Tokens' or SoT. Instead of viewing them as a narrative or explanation, they suggest seeing them as a temporary record of the model's internal 'state' during the problem-solving process. Think of it like the model is writing down intermediate calculations – those calculations help it get to the answer, but reading them back doesn't fully explain *why* it chose those specific calculations. The tokens are just a way to carry information from one step to the next.
Why it matters?
This is important because it changes how we should study LLMs. Instead of trying to understand their reasoning by simply reading the tokens they generate, we need to focus on figuring out what those tokens *represent* in terms of the model's internal workings. This shift in perspective could lead to a better understanding of how LLMs solve problems and how to improve their performance and reliability.
Abstract
Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the model's actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state -- the sole persistent information carrier across the model's stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.