The Curious Case of Analogies: Investigating Analogical Reasoning in Large Language Models
Taewhoo Lee, Minju Song, Chanwoong Yoon, Jungwoo Park, Jaewoo Kang
2025-12-03
Summary
This research investigates how well large language models (LLMs) can perform analogical reasoning, which is a key part of how humans think and learn. It looks at whether LLMs can understand relationships between things and apply those relationships to new situations, like solving analogies you might see on a test.
What's the problem?
While LLMs are good at recognizing patterns and basic concepts, it wasn't clear if they could grasp more complex, high-level relationships and *use* them to solve analogies. The question is whether they just memorize things or actually understand the underlying logic of how things relate to each other. Specifically, the researchers wanted to know if LLMs could handle analogies that require understanding relationships, not just surface-level similarities.
What's the solution?
The researchers tested LLMs using proportional and story analogies. They examined what was happening *inside* the LLM when it got analogies right or wrong. They found that when the LLM succeeded, the relevant relationship information was present in the middle layers of the model. When it failed, that information was missing. They also discovered that LLMs sometimes struggle to apply known relationships to *new* things, even if they understand the relationship itself. They tried 'patching' the LLM's internal representation to help it transfer information, with some success. Finally, they found that successful analogies relied on a strong structural similarity between the parts of the analogy, while failures showed a breakdown in that structure.
Why it matters?
This work shows that LLMs are starting to develop the ability to understand and use relationships, but they aren't quite at the level of human reasoning. It highlights both what LLMs can do and where they still fall short, which is important for improving their ability to think and solve problems more like humans do. Understanding these limitations can help researchers build better AI systems that can truly reason and learn.
Abstract
Analogical reasoning is at the core of human cognition, serving as an important foundation for a variety of intellectual activities. While prior work has shown that LLMs can represent task patterns and surface-level concepts, it remains unclear whether these models can encode high-level relational concepts and apply them to novel situations through structured comparisons. In this work, we explore this fundamental aspect using proportional and story analogies, and identify three key findings. First, LLMs effectively encode the underlying relationships between analogous entities; both attributive and relational information propagate through mid-upper layers in correct cases, whereas reasoning failures reflect missing relational information within these layers. Second, unlike humans, LLMs often struggle not only when relational information is missing, but also when attempting to apply it to new entities. In such cases, strategically patching hidden representations at critical token positions can facilitate information transfer to a certain extent. Lastly, successful analogical reasoning in LLMs is marked by strong structural alignment between analogous situations, whereas failures often reflect degraded or misplaced alignment. Overall, our findings reveal that LLMs exhibit emerging but limited capabilities in encoding and applying high-level relational concepts, highlighting both parallels and gaps with human cognition.