Memory Bank Compression for Continual Adaptation of Large Language Models
Thomas Katraouras, Dimitrios Rafailidis
2026-01-16
Summary
This paper focuses on how to keep large language models, like the ones powering chatbots, up-to-date with new information without making them forget what they already know.
What's the problem?
Large language models are great, but their knowledge gets stale quickly as new information emerges. Updating them with new data is hard because simply retraining them from scratch is very resource-intensive and can cause them to 'forget' previous knowledge. One approach is to give the model an external 'memory bank' to store new info, but this memory bank grows constantly with new data, becoming unmanageable over time.
What's the solution?
The researchers developed a method called MBC that compresses the memory bank using a clever technique called 'codebook optimization'. Think of it like creating a shorthand for frequently used information. They also added a 'resetting mechanism' to prevent the shorthand from becoming too simplified and losing important details. Finally, they made changes to how the model pays attention to information, allowing it to efficiently use the compressed memory.
Why it matters?
This work is important because it allows language models to continuously learn and adapt to new information in a practical way. MBC dramatically reduces the amount of memory needed to store new knowledge – shrinking it to just 0.3% of the size of other methods – while still maintaining accuracy, making continual learning more feasible for real-world applications.
Abstract
Large Language Models (LLMs) have become a mainstay for many everyday applications. However, as data evolve their knowledge quickly becomes outdated. Continual learning aims to update LLMs with new information without erasing previously acquired knowledge. Although methods such as full fine-tuning can incorporate new data, they are computationally expensive and prone to catastrophic forgetting, where prior knowledge is overwritten. Memory-augmented approaches address this by equipping LLMs with a memory bank, that is an external memory module which stores information for future use. However, these methods face a critical limitation, in particular, the memory bank constantly grows in the real-world scenario when large-scale data streams arrive. In this paper, we propose MBC, a model that compresses the memory bank through a codebook optimization strategy during online adaptation learning. To ensure stable learning, we also introduce an online resetting mechanism that prevents codebook collapse. In addition, we employ Key-Value Low-Rank Adaptation in the attention layers of the LLM, enabling efficient utilization of the compressed memory representations. Experiments with benchmark question-answering datasets demonstrate that MBC reduces the memory bank size to 0.3% when compared against the most competitive baseline, while maintaining high retention accuracy during online adaptation learning. Our code is publicly available at https://github.com/Thomkat/MBC.