Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception
Jihao Zhao, Zhiyuan Ji, Pengnian Qi, Simin Niu, Bo Tang, Feiyu Xiong, Zhiyu Li
2024-10-22
Summary
This paper introduces Meta-Chunking, a new method for improving how text is segmented in Retrieval-Augmented Generation (RAG) systems, focusing on the logical connections between sentences.
What's the problem?
In RAG systems, effectively breaking down text into manageable pieces, or chunks, is crucial for tasks that require deep understanding, like answering questions. However, traditional methods often fail to capture the subtle relationships between sentences, which can lead to poor performance in knowledge-based tasks. This lack of effective chunking can make it harder for models to retrieve and generate accurate information.
What's the solution?
To solve this problem, the authors developed Meta-Chunking, which groups sentences based on their logical connections rather than just following strict sentence or paragraph boundaries. They created two strategies: Margin Sampling Chunking, which uses a model to decide whether to split sentences based on their context, and Perplexity Chunking, which identifies chunk boundaries by analyzing how predictable the text is. Additionally, they introduced a dynamic merging strategy that balances detailed and broader chunking approaches. Their experiments showed that Meta-Chunking significantly improves performance in question-answering tasks while reducing processing time.
Why it matters?
This research is important because it enhances the efficiency and effectiveness of AI systems that rely on understanding and generating text. By improving how text is segmented, Meta-Chunking can lead to better performance in applications like search engines, chatbots, and educational tools, ultimately making AI more useful in real-world scenarios.
Abstract
Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline, which impacts the quality of knowledge-intensive tasks. This paper introduces the concept of Meta-Chunking, which refers to a granularity between sentences and paragraphs, consisting of a collection of sentences within a paragraph that have deep linguistic logical connections. To implement Meta-Chunking, we designed two strategies based on LLMs: Margin Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform binary classification on whether consecutive sentences need to be segmented, making decisions based on the probability difference obtained from margin sampling. The latter precisely identifies text chunk boundaries by analyzing the characteristics of perplexity distribution. Additionally, considering the inherent complexity of different texts, we propose a strategy that combines Meta-Chunking with dynamic merging to achieve a balance between fine-grained and coarse-grained text chunking. Experiments conducted on eleven datasets demonstrate that Meta-Chunking can more efficiently improve the performance of single-hop and multi-hop question answering based on RAG. For instance, on the 2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only consuming 45.8% of the time. Our code is available at https://github.com/IAAR-Shanghai/Meta-Chunking.