How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training
Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, Huajun Chen
2025-02-18
Summary
This paper talks about how large language models (LLMs) learn new information during training by studying something called 'knowledge circuits,' which are like the pathways in the AI's brain that store and process knowledge.
What's the problem?
Even though LLMs are great at tasks that require a lot of knowledge, we don't fully understand how they take in and organize new information. This makes it hard to improve their training methods and ensure they learn efficiently.
What's the solution?
The researchers studied how LLMs acquire knowledge by looking at 'knowledge circuits,' which are specific parts of the AI's neural network that handle storing and using information. They found that learning happens in phases: first, the circuits form, and then they get optimized. They also discovered that new knowledge is easier to learn if it's related to what the model already knows. The study used advanced techniques to map these circuits and track how they evolve during training.
Why it matters?
This matters because understanding how LLMs learn can help scientists make these models smarter and more efficient. By improving how LLMs are trained, we can create AI systems that are better at learning new information and adapting to different tasks, which could make them more useful in real-world applications.
Abstract
Despite exceptional capabilities in knowledge-intensive tasks, Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge, particularly how to structurally embed acquired knowledge in their neural computations. We address this issue through the lens of knowledge circuit evolution, identifying computational subgraphs that facilitate knowledge storage and processing. Our systematic analysis of circuit evolution throughout continual pre-training reveals several key findings: (1) the acquisition of new knowledge is influenced by its relevance to pre-existing knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase shift from formation to optimization; (3) the evolution of knowledge circuits follows a deep-to-shallow pattern. These insights not only advance our theoretical understanding of the mechanisms of new knowledge acquisition in LLMs, but also provide potential implications for improving continual pre-training strategies to enhance model performance. Code and data will be available at https://github.com/zjunlp/DynamicKnowledgeCircuits.