Zero-Shot Dense Retrieval with Embeddings from Relevance Feedback
Nour Jedidi, Yung-Sung Chuang, Leslie Shing, James Glass
2024-10-30
Summary
This paper presents a new approach called ReDE-RF, which improves how large language models (LLMs) can retrieve relevant documents without needing a lot of specific training data.
What's the problem?
Retrieving the right documents from a database can be challenging, especially when there isn't enough labeled data to guide the process. Traditional methods often rely on generating hypothetical documents using LLMs, but this can be inefficient and requires the model to have specific knowledge about the topic, which isn't always practical.
What's the solution?
The authors propose Real Document Embeddings from Relevance Feedback (ReDE-RF), which changes the way document retrieval is approached. Instead of generating many hypothetical documents, ReDE-RF focuses on estimating relevance by having the LLM select which existing documents are most relevant to a query. This means the model only needs to determine relevance rather than generate new content, which speeds up the search process and improves accuracy.
Why it matters?
This research is significant because it provides a more efficient way to retrieve documents in situations where labeled data is scarce. By improving retrieval accuracy and reducing the time it takes to find relevant information, ReDE-RF could enhance various applications, such as search engines and information management systems, making it easier for users to access the information they need.
Abstract
Building effective dense retrieval systems remains difficult when relevance supervision is not available. Recent work has looked to overcome this challenge by using a Large Language Model (LLM) to generate hypothetical documents that can be used to find the closest real document. However, this approach relies solely on the LLM to have domain-specific knowledge relevant to the query, which may not be practical. Furthermore, generating hypothetical documents can be inefficient as it requires the LLM to generate a large number of tokens for each query. To address these challenges, we introduce Real Document Embeddings from Relevance Feedback (ReDE-RF). Inspired by relevance feedback, ReDE-RF proposes to re-frame hypothetical document generation as a relevance estimation task, using an LLM to select which documents should be used for nearest neighbor search. Through this re-framing, the LLM no longer needs domain-specific knowledge but only needs to judge what is relevant. Additionally, relevance estimation only requires the LLM to output a single token, thereby improving search latency. Our experiments show that ReDE-RF consistently surpasses state-of-the-art zero-shot dense retrieval methods across a wide range of low-resource retrieval datasets while also making significant improvements in latency per-query.