Improving Multi-step RAG with Hypergraph-based Memory for Long-Context Complex Relational Modeling
Chulun Zhou, Chunkang Zhang, Guoxin Yu, Fandong Meng, Jie Zhou, Wai Lam, Mo Yu
2026-01-02
Summary
This paper introduces a new way to help large language models (LLMs) perform better on complex tasks that require understanding a lot of information and reasoning through multiple steps, specifically focusing on improving how these models use information they've already 'remembered'.
What's the problem?
Current systems that help LLMs remember information, called 'working memory', mostly just store facts as they're found. They treat these facts as separate pieces and don't really connect them to see how they relate to each other. This limits the model's ability to truly understand the bigger picture and reason effectively, leading to incomplete or fragmented thinking when dealing with long and complicated topics.
What's the solution?
The researchers developed a new memory system called HGMem, which uses a 'hypergraph' to represent memory. Think of it like connecting facts and ideas not just in a simple list, but in a network where relationships between them are clearly shown. This allows the model to build a more dynamic and interconnected understanding, recognizing how different pieces of information work together to solve a problem and evolve its knowledge as it goes.
Why it matters?
This research is important because it addresses a key weakness in current LLM systems – their struggle with complex reasoning and understanding large amounts of information. By improving the way models remember and connect information, HGMem can lead to more accurate, insightful, and comprehensive answers, especially when dealing with challenging tasks that require a 'global' understanding of the topic.
Abstract
Multi-step retrieval-augmented generation (RAG) has become a widely adopted strategy for enhancing large language models (LLMs) on tasks that demand global comprehension and intensive reasoning. Many RAG systems incorporate a working memory module to consolidate retrieved information. However, existing memory designs function primarily as passive storage that accumulates isolated facts for the purpose of condensing the lengthy inputs and generating new sub-queries through deduction. This static nature overlooks the crucial high-order correlations among primitive facts, the compositions of which can often provide stronger guidance for subsequent steps. Therefore, their representational strength and impact on multi-step reasoning and knowledge evolution are limited, resulting in fragmented reasoning and weak global sense-making capacity in extended contexts. We introduce HGMem, a hypergraph-based memory mechanism that extends the concept of memory beyond simple storage into a dynamic, expressive structure for complex reasoning and global understanding. In our approach, memory is represented as a hypergraph whose hyperedges correspond to distinct memory units, enabling the progressive formation of higher-order interactions within memory. This mechanism connects facts and thoughts around the focal problem, evolving into an integrated and situated knowledge structure that provides strong propositions for deeper reasoning in subsequent steps. We evaluate HGMem on several challenging datasets designed for global sense-making. Extensive experiments and in-depth analyses show that our method consistently improves multi-step RAG and substantially outperforms strong baseline systems across diverse tasks.