REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration
Yisu Wang, Ming Wang, Haoyuan Song, Wenjie Huang, Chaozheng Wang, Yi Xie, Xuming Ran
2025-10-06
Summary
This paper introduces REPAIR, a new method for updating large language models (LLMs) after they've been initially trained, focusing on making those updates accurate and efficient without messing up what the model already knows.
What's the problem?
Large language models are really good, but fixing mistakes or adding new information after they're built is hard and expensive. Simply retraining the whole model is costly, and making small changes can sometimes accidentally cause the model to forget things it already learned or create new, unintended problems. Existing methods don't always account for how changes in one area of the model can affect other areas.
What's the solution?
REPAIR tackles this by using a system of careful, step-by-step updates. It's like editing a document – you don't rewrite the whole thing for one small change. REPAIR uses a feedback loop to check its work and a dynamic memory system to manage changes effectively. It also frequently combines new knowledge with existing knowledge and makes sure changes stay focused on the specific area being updated, preventing widespread disruptions. Essentially, it's a controlled and precise way to edit the model's knowledge.
Why it matters?
This research is important because it provides a way to build more reliable and adaptable LLMs. Instead of constantly needing to rebuild models from scratch, we can continually improve them with targeted updates, making them more accurate and useful over time. This is a big step towards creating AI systems that can learn and evolve without becoming unstable or forgetting crucial information.
Abstract
Post-training for large language models (LLMs) is constrained by the high cost of acquiring new knowledge or correcting errors and by the unintended side effects that frequently arise from retraining. To address these issues, we introduce REPAIR (Robust Editing via Progressive Adaptive Intervention and Reintegration), a lifelong editing framework designed to support precise and low-cost model updates while preserving non-target knowledge. REPAIR mitigates the instability and conflicts of large-scale sequential edits through a closed-loop feedback mechanism coupled with dynamic memory management. Furthermore, by incorporating frequent knowledge fusion and enforcing strong locality guards, REPAIR effectively addresses the shortcomings of traditional distribution-agnostic approaches that often overlook unintended ripple effects. Our experiments demonstrate that REPAIR boosts editing accuracy by 10%-30% across multiple model families and significantly reduces knowledge forgetting. This work introduces a robust framework for developing reliable, scalable, and continually evolving LLMs.