✦
Summary
Rotorquant is an open-source KV cache compression method for large language model inference. It uses block-diagonal rotation to reduce cache cost while preserving quality, targeting the memory and throughput bottlenecks that appear during long-context generation. The repository reports better perplexity than TurboQuant, faster decode, faster prefill, and far fewer parameters, making it useful for practical inference optimization.
Key Features
Compresses LLM KV caches with block-diagonal rotations.
Improves decode speed for long-context inference workloads.
Improves prefill speed compared with reported baselines.
Reduces parameter overhead for cache compression.
Targets drop-in integration with llama.cpp.
Helps reduce memory pressure during autoregressive generation.
Preserves quality through rotation-based cache representation.
Provides public source code for inference-system experimentation.
About Rotorquant
Technically, Rotorquant compresses key-value cache representations through block-diagonal rotational transforms. KV cache compression matters because autoregressive decoding stores attention states across tokens, and those states become expensive as context length grows. By integrating as a drop-in llama.cpp path, Rotorquant targets real inference stacks rather than only theoretical compression benchmarks.
Rotorquant is valuable for developers running local or hosted LLM inference where memory bandwidth, prefill speed, and decode latency matter. It can help make longer contexts more practical on constrained hardware while keeping output quality closer to the uncompressed baseline.