On Relation-Specific Neurons in Large Language Models
Yihong Liu, Runsheng Chen, Lea Hirlimann, Ahmad Dawar Hakimi, Mingyang Wang, Amir Hossein Kargaran, Sascha Rothe, François Yvon, Hinrich Schütze
2025-02-28
Summary
This paper talks about special neurons in large language models (LLMs) that focus on understanding relationships between things, rather than just memorizing facts. The researchers studied these neurons in the Llama-2 AI model to see how they work and what happens when you turn them off.
What's the problem?
In AI language models, it's hard to understand how they store and use information about relationships between things. Scientists weren't sure if there were specific parts of the AI (neurons) that handled these relationships independently of the specific things involved.
What's the solution?
The researchers studied the Llama-2 AI model, looking for neurons that seemed to focus on relationships. They found these special neurons and then did experiments where they turned off some of these neurons to see what happened. They discovered three important things: 1) turning off more of these neurons made the AI worse at understanding that relationship, 2) some neurons could handle multiple types of relationships and even work across different languages, and 3) turning off neurons for one relationship sometimes made the AI better at handling other relationships.
Why it matters?
This matters because it helps us understand how AI thinks and processes information. By knowing that there are specific parts of AI models dedicated to understanding relationships, we can potentially make better, more efficient AI systems. It could also help us figure out why AI sometimes makes mistakes or has biases, and maybe even find ways to fix these problems. This research is a step towards making AI that thinks more like humans do, which could lead to more useful and trustworthy AI assistants in the future.
Abstract
In large language models (LLMs), certain neurons can store distinct pieces of knowledge learned during pretraining. While knowledge typically appears as a combination of relations and entities, it remains unclear whether some neurons focus on a relation itself -- independent of any entity. We hypothesize such neurons detect a relation in the input text and guide generation involving such a relation. To investigate this, we study the Llama-2 family on a chosen set of relations with a statistics-based method. Our experiments demonstrate the existence of relation-specific neurons. We measure the effect of selectively deactivating candidate neurons specific to relation r on the LLM's ability to handle (1) facts whose relation is r and (2) facts whose relation is a different relation r' neq r. With respect to their capacity for encoding relation information, we give evidence for the following three properties of relation-specific neurons. (i) Neuron cumulativity. The neurons for r present a cumulative effect so that deactivating a larger portion of them results in the degradation of more facts in r. (ii) Neuron versatility. Neurons can be shared across multiple closely related as well as less related relations. Some relation neurons transfer across languages. (iii) Neuron interference. Deactivating neurons specific to one relation can improve LLM generation performance for facts of other relations. We will make our code publicly available at https://github.com/cisnlp/relation-specific-neurons.