ACE: Attribution-Controlled Knowledge Editing for Multi-hop Factual Recall
Jiayu Yang, Yuxuan Fan, Songning Lai, Shengen Wu, Jiaqi Tang, Chun Kang, Zhijiang Guo, Yutao Yue
2025-10-13
Summary
This paper investigates how to accurately update information within large language models, specifically when the update requires the model to remember and use multiple pieces of information in a chain of reasoning.
What's the problem?
Large language models struggle to correctly recall facts when those facts are connected and require multiple steps of reasoning to use. When you try to change a fact within a complex reasoning process, the model often forgets earlier parts of the chain, leading to incorrect answers. The issue arises because current methods don't fully understand how the model stores and uses information during these multi-step thought processes at a very detailed level – specifically, how individual neurons contribute.
What's the solution?
The researchers discovered that during reasoning, certain neurons act as 'query' neurons, asking for specific information, and other neurons act as 'value' neurons, providing that information. These query neurons activate value neurons across different layers of the model, building up the answer step-by-step. They developed a new method called ACE, which carefully identifies and edits these crucial query-value pathways at the neuron level. This ensures that when a fact is updated, the model doesn't disrupt the entire reasoning chain, leading to more accurate results.
Why it matters?
This work is important because it provides a deeper understanding of how large language models actually *think* when reasoning. By focusing on the neuron-level interactions, ACE offers a more reliable way to update the knowledge within these models, improving their accuracy and making them more trustworthy for tasks that require complex reasoning and factual recall. It also opens up new avenues for improving these models by understanding their internal mechanisms.
Abstract
Large Language Models (LLMs) require efficient knowledge editing (KE) to update factual information, yet existing methods exhibit significant performance decay in multi-hop factual recall. This failure is particularly acute when edits involve intermediate implicit subjects within reasoning chains. Through causal analysis, we reveal that this limitation stems from an oversight of how chained knowledge is dynamically represented and utilized at the neuron level. We discover that during multi hop reasoning, implicit subjects function as query neurons, which sequentially activate corresponding value neurons across transformer layers to accumulate information toward the final answer, a dynamic prior KE work has overlooked. Guided by this insight, we propose ACE: Attribution-Controlled Knowledge Editing for Multi-hop Factual Recall, a framework that leverages neuron-level attribution to identify and edit these critical query-value (Q-V) pathways. ACE provides a mechanistically grounded solution for multi-hop KE, empirically outperforming state-of-the-art methods by 9.44% on GPT-J and 37.46% on Qwen3-8B. Our analysis further reveals more fine-grained activation patterns in Qwen3 and demonstrates that the semantic interpretability of value neurons is orchestrated by query-driven accumulation. These findings establish a new pathway for advancing KE capabilities based on the principled understanding of internal reasoning mechanisms.