Characterizing the role of reasoning tokens in large language models as state over tokens

Abstract

Large language models increasingly rely on generating reasoning tokens that appear to describe a reasoning process performed by the model to arrive at the answer. However, empirical evidence reveals that when read as text, these tokens are not a faithful explanation of the model’s actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state—the mechanism that persists the computation across the separate generation cycles. The SoT framing explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We discuss how it dispels two common misconceptions: that the text records the complete process, and that the model uses the words with the meanings a human reader ascribes to them. We argue that this framing offers a fruitful way to think about these tokens, one that captures their mechanics accurately while remaining simple and intuitive, and that adopting it opens research directions. To better understand the mechanism of large-language-model reasoning, research should move beyond reading the tokens as text and also focus on decoding them as state. When large language models ‘reason’ about a problem, they might still solve it differently from what their reasoning transcript suggest. In this Perspective, Levy et al. describe how one can start investigating the role of these reasoning tokens.