Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, Mikhail Burtsev
2025-02-19
Summary
This paper talks about a new way to compress large amounts of text information into a single vector, pushing the limits of how much data can be packed into a small space. The researchers managed to squeeze 1568 tokens (words or parts of words) into one vector, which is much more than previous methods could achieve.
What's the problem?
Current methods for compressing text data for language models can only reduce the size by about 10 times. This is surprising because, in theory, the numbers used to represent this data should be able to hold much more information. This limitation makes it harder to make language models more efficient and able to handle longer texts.
What's the solution?
Instead of using typical AI models to compress the data, the researchers used a special optimization process for each sample of text. This allowed them to create vectors that could compress text up to 1500 times smaller than the original. They also found that the limit on compression depends more on how unpredictable the text is, rather than how long it is.
Why it matters?
This research matters because it shows there's a lot of room for improvement in how we compress and use data in language models. By demonstrating that much higher compression is possible, it suggests that we could make AI language models much more efficient and capable of handling longer texts without needing as much computing power. This could lead to faster, more powerful AI systems that can work with larger amounts of text at once.
Abstract
A range of recent works addresses the problem of compression of sequence of tokens into a shorter sequence of real-valued vectors to be used as inputs instead of token embeddings or key-value cache. These approaches allow to reduce the amount of compute in existing language models. Despite relying on powerful models as encoders, the maximum attainable lossless <PRE_TAG>compression ratio</POST_TAG> is typically not higher than x10. This fact is highly intriguing because, in theory, the maximum information capacity of large real-valued vectors is far beyond the presented rates even for 16-bit precision and a modest vector size. In this work, we explore the limits of compression by replacing the encoder with a per-sample optimization procedure. We show that vectors with compression ratios up to x1500 exist, which highlights two orders of magnitude gap between existing and practically attainable solutions. Furthermore, we empirically show that the compression limits are determined not by the length of the input but by the amount of uncertainty to be reduced, namely, the cross-entropy loss on this sequence without any conditioning. The obtained limits highlight the substantial gap between the theoretical capacity of input embeddings and their practical utilization, suggesting significant room for optimization in model design.