KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Augmentations and Constraints
Kailin Jiang, Hongbo Jiang, Ning Jiang, Zhi Gao, Jinhe Bi, Yuchen Ren, Bin Li, Yuntao Du, Lei Liu, Qing Li
2025-10-23
Summary
This paper focuses on how to update the knowledge of large AI models, called Large Multimodal Models (LMMs), with new information without them forgetting what they already know.
What's the problem?
LMMs are trained with a lot of information, but that information becomes outdated quickly as the world changes. Simply retraining them with new data causes them to 'forget' previously learned facts – this is called catastrophic forgetting. Existing methods struggle to effectively add new knowledge *and* keep the old knowledge intact, making it hard for these models to stay current.
What's the solution?
The researchers developed a method called KORE, which stands for KnOwledge-oRientEd augmentations and constraints. KORE works in two main ways: first, it transforms new facts into a structured format the model can easily understand. Second, it carefully updates the model’s internal settings so that learning new things doesn’t disrupt the existing knowledge. It does this by finding a 'safe space' within the model’s structure where new information can be added without interfering with the old.
Why it matters?
This research is important because it addresses a key limitation of current AI models. By allowing LMMs to continuously learn and adapt without forgetting, we can create more reliable and useful AI systems that stay up-to-date with the latest information and can be applied to a wider range of real-world problems.
Abstract
Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address this, we propose KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints for injecting new knowledge into large multimodal models while preserving old knowledge. Unlike general text or image data augmentation, KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B, LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting.