SKETCH: Structured Knowledge Enhanced Text Comprehension for Holistic Retrieval
Aakash Mahalingam, Vinesh Kumar Gande, Aman Chadha, Vinija Jain, Divya Chaudhary
2024-12-25
Summary
This paper talks about SKETCH, a new method that improves how AI systems retrieve and understand information by combining semantic text retrieval with knowledge graphs for better context comprehension.
What's the problem?
Retrieval-Augmented Generation (RAG) systems are designed to help AI generate accurate responses by pulling in relevant information from large datasets. However, these systems often struggle to effectively process and retrieve this information while keeping a clear understanding of the context, leading to less accurate answers.
What's the solution?
To tackle this issue, the authors introduce SKETCH, which enhances the RAG retrieval process by integrating structured data (from knowledge graphs) with unstructured data (like text). This combination helps the AI maintain a better understanding of the context when generating responses. The paper shows that SKETCH significantly improves retrieval performance and context accuracy compared to traditional methods across several datasets, achieving high scores in answer relevancy and context precision.
Why it matters?
This research is important because it sets new standards for how AI systems can retrieve and comprehend information. By improving the accuracy and relevance of responses, SKETCH can enhance applications like chatbots, search engines, and other tools that rely on understanding complex information, ultimately making them more useful and reliable.
Abstract
Retrieval-Augmented Generation (RAG) systems have become pivotal in leveraging vast corpora to generate informed and contextually relevant responses, notably reducing hallucinations in Large Language Models. Despite significant advancements, these systems struggle to efficiently process and retrieve information from large datasets while maintaining a comprehensive understanding of the context. This paper introduces SKETCH, a novel methodology that enhances the RAG retrieval process by integrating semantic text retrieval with knowledge graphs, thereby merging structured and unstructured data for a more holistic comprehension. SKETCH, demonstrates substantial improvements in retrieval performance and maintains superior context integrity compared to traditional methods. Evaluated across four diverse datasets: QuALITY, QASPER, NarrativeQA, and Italian Cuisine-SKETCH consistently outperforms baseline approaches on key RAGAS metrics such as answer_relevancy, faithfulness, context_precision and context_recall. Notably, on the Italian Cuisine dataset, SKETCH achieved an answer relevancy of 0.94 and a context precision of 0.99, representing the highest performance across all evaluated metrics. These results highlight SKETCH's capability in delivering more accurate and contextually relevant responses, setting new benchmarks for future retrieval systems.