Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, Ji-Rong Wen
2024-06-28
Summary
This paper talks about DPA-RAG, a new framework designed to improve how retrieval-augmented generation (RAG) systems work with large language models (LLMs). It focuses on aligning the way these models retrieve information based on their different knowledge preferences to reduce errors and improve accuracy.
What's the problem?
Large language models can sometimes produce incorrect or misleading information, a problem known as 'hallucination.' RAG systems aim to reduce this issue by using external information to support the model's responses. However, aligning the retrieval process with the diverse ways that different LLMs understand and prefer information is challenging. This misalignment can lead to ineffective retrieval and less reliable results.
What's the solution?
To tackle this challenge, the authors developed DPA-RAG, which includes a process for constructing preference knowledge and five new strategies for enhancing queries. This framework allows for both external and internal alignment of preferences: it helps the retriever find the right information based on how different LLMs think (external alignment) and prepares LLMs to better understand this information before they are fine-tuned (internal alignment). The authors tested DPA-RAG on various datasets and found that it outperformed existing methods, showing that it effectively integrates different types of LLMs.
Why it matters?
This research is important because it enhances the reliability of RAG systems by ensuring that the retrieval process aligns with how different LLMs prefer to receive information. By improving this alignment, DPA-RAG can help create more accurate and trustworthy AI systems, which is crucial for applications like question-answering, chatbots, and any technology that relies on generating reliable information.
Abstract
Retrieval-augmented generation (RAG) has demonstrated effectiveness in mitigating the hallucination problem of large language models (LLMs). However, the difficulty of aligning the retriever with the diverse LLMs' knowledge preferences inevitably poses an inevitable challenge in developing a reliable RAG system. To address this issue, we propose DPA-RAG, a universal framework designed to align diverse knowledge preferences within RAG systems. Specifically, we initially introduce a preference knowledge construction pipline and incorporate five novel query augmentation strategies to alleviate preference data scarcity. Based on preference data, DPA-RAG accomplishes both external and internal preference alignment: 1) It jointly integrate pair-wise, point-wise, and contrastive preference alignment abilities into the reranker, achieving external preference alignment among RAG components. 2) It further introduces a pre-aligned stage before vanilla Supervised Fine-tuning (SFT), enabling LLMs to implicitly capture knowledge aligned with their reasoning preferences, achieving LLMs' internal alignment. Experimental results across four knowledge-intensive QA datasets demonstrate that DPA-RAG outperforms all baselines and seamlessly integrates both black-box and open-sourced LLM readers. Further qualitative analysis and discussions also provide empirical guidance for achieving reliable RAG systems. Our code is publicly available at https://github.com/dongguanting/DPA-RAG.