SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation
Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Weichuan Liu, Lei Hou, Juanzi Li
2024-06-28
Summary
This paper talks about SeaKR, a new system designed to improve how large language models (LLMs) retrieve and use information by making them aware of their own uncertainty. This helps the models decide when to look for more information and how to use it effectively.
What's the problem?
Large language models often struggle with providing accurate answers because they can be unsure about the information they have. When they don't know something, they might give incorrect or misleading responses. Existing methods for retrieving additional information don't take this uncertainty into account, which can lead to inefficiencies and errors in the answers produced by the models.
What's the solution?
To solve this problem, the authors developed SeaKR, which allows LLMs to recognize when they are uncertain about an answer. When the model feels uncertain, it activates a retrieval process to gather more information. SeaKR also re-ranks the retrieved information based on how much it helps reduce the model's uncertainty. This means that the model can choose the best pieces of information to use when answering questions. Additionally, SeaKR supports different reasoning strategies to tackle complex tasks that may require multiple pieces of information.
Why it matters?
This research is important because it enhances the reliability of language models by allowing them to better manage their uncertainty. By improving how these models retrieve and integrate knowledge, SeaKR can lead to more accurate and trustworthy answers in applications like education, customer service, and any field where accurate information is crucial.
Abstract
This paper introduces Self-aware Knowledge Retrieval (SeaKR), a novel adaptive RAG model that extracts self-aware uncertainty of LLMs from their internal states. SeaKR activates retrieval when the LLMs present high self-aware uncertainty for generation. To effectively integrate retrieved knowledge snippets, SeaKR re-ranks them based on LLM's self-aware uncertainty to preserve the snippet that reduces their uncertainty to the utmost. To facilitate solving complex tasks that require multiple retrievals, SeaKR utilizes their self-aware uncertainty to choose among different reasoning strategies. Our experiments on both complex and simple Question Answering datasets show that SeaKR outperforms existing adaptive RAG methods. We release our code at https://github.com/THU-KEG/SeaKR.