How to Teach Large Multimodal Models New Skills
Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem
2025-10-13
Summary
This paper investigates how to teach large AI models, which can understand both text and images, new skills without making them forget what they already know.
What's the problem?
When you try to teach these large models a new task by further training them, they often get better at the new task but worse at things they were already good at – this is called 'forgetting'. The researchers wanted to understand *why* this happens and if there's a way to prevent it. They noticed this 'forgetting' isn't always permanent and can sometimes recover with more training, but they wanted to figure out what causes the initial drop in performance.
What's the solution?
The researchers discovered that this 'forgetting' is linked to how the model changes the probability of choosing certain words when generating text. They developed a way to measure this shift in word choices, and found it directly relates to how much the model 'forgets'. Based on this, they found that focusing the training process on specific parts of the model – either the parts that pay attention to relationships between words, or the parts that control how information flows – while leaving other parts untouched, significantly reduced forgetting while still allowing the model to learn the new skill effectively.
Why it matters?
This research is important because it provides a practical way to continuously update large AI models with new abilities without sacrificing their existing knowledge. This is crucial for building AI systems that can adapt and improve over time, making them more useful and reliable in real-world applications. It helps us move towards AI that doesn't need to be completely retrained from scratch every time we want it to do something new.
Abstract
How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. We observe that apparent "forgetting" on held-out tasks after narrow fine-tuning can partly recover at later stages. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that co-varies with forgetting. Guided by this picture, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers, and (ii) updating only the MLP Gate&Up while freezing the Down projection. Across models and tasks, these choices deliver strong target gains while largely preserving held-out performance. Code is available at https://github.com/jessemelpolio/LMM_CL