Compressed Chain of Thought: Efficient Reasoning Through Dense Representations
Jeffrey Cheng, Benjamin Van Durme
2024-12-18
Summary
This paper talks about Compressed Chain of Thought (CCoT), a new method that helps language models reason more efficiently by using compressed representations of their thought processes.
What's the problem?
Language models, like those used in AI, can struggle with reasoning tasks because they often take a long time to generate answers. This delay happens because they need to think through many steps, which can slow down their performance. Current methods for helping these models think (called chain-of-thought decoding) can be slow and inefficient, making it hard for them to provide quick answers.
What's the solution?
CCoT introduces a way to create 'contemplation tokens,' which are compressed versions of the reasoning steps that the model takes. Instead of using long sequences of reasoning, CCoT allows the model to generate shorter, more efficient representations that still capture the essential parts of its thought process. This means the model can reason faster while still improving its accuracy on complex tasks. The framework can be applied to existing language models without needing major changes.
Why it matters?
This research is important because it makes AI models more efficient at reasoning tasks, which can lead to faster and more accurate responses in real-world applications like education, customer service, and problem-solving. By improving how these models think through problems, CCoT could enhance their usability and effectiveness in various fields.
Abstract
Chain-of-thought (CoT) decoding enables language models to improve reasoning performance at the cost of high generation latency in decoding. Recent proposals have explored variants of contemplation tokens, a term we introduce that refers to special tokens used during inference to allow for extra computation. Prior work has considered fixed-length sequences drawn from a discrete set of embeddings as contemplation tokens. Here we propose Compressed Chain-of-Thought (CCoT), a framework to generate contentful and continuous contemplation tokens of variable sequence length. The generated contemplation tokens are compressed representations of explicit reasoning chains, and our method can be applied to off-the-shelf decoder language models. Through experiments, we illustrate how CCoT enables additional reasoning over dense contentful representations to achieve corresponding improvements in accuracy. Moreover, the reasoning improvements can be adaptively modified on demand by controlling the number of contemplation tokens generated.