RARe: Retrieval Augmented Retrieval with In-Context Examples
Atula Tejaswi, Yoonsang Lee, Sujay Sanghavi, Eunsol Choi
2024-10-30
Summary
This paper introduces RARe, which stands for Retrieval Augmented Retrieval with In-Context Examples. It explores how using examples of similar queries can improve the performance of models that retrieve information from large datasets.
What's the problem?
Retrieval tasks often rely on models to find relevant information based on user queries. However, simply adding examples of previous queries and their corresponding documents to the current query doesn't work effectively for these models. This limitation can lead to less accurate results when trying to retrieve information.
What's the solution?
To solve this issue, the authors developed RARe, which fine-tunes pre-trained models using in-context examples that are semantically similar to the target query. This means that instead of just adding examples, the model learns from them in a way that improves its ability to retrieve relevant information. The authors tested RARe on various retrieval tasks and found that it consistently improved performance by up to 2.72% in accuracy compared to previous methods.
Why it matters?
This research is important because it enhances how retrieval models work, making them more effective at finding accurate information. By improving the performance of these models, RARe can help in many applications, such as search engines and information retrieval systems, leading to better user experiences and more reliable results.
Abstract
We investigate whether in-context examples, widely used in decoder-only language models (LLMs), can improve embedding model performance in retrieval tasks. Unlike in LLMs, naively prepending in-context examples (query-document pairs) to the target query at inference time does not work out of the box. We introduce a simple approach to enable retrievers to use in-context examples. Our approach, RARe, finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This can be applied to adapt various base architectures (i.e., decoder-only language models, retriever models) and consistently achieves performance gains of up to +2.72% nDCG across various open-domain retrieval datasets (BeIR, RAR-b). In particular, we find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. We further provide analysis on the design choices of in-context example augmentation and lay the foundation for future work in this space.