Rank1: Test-Time Compute for Reranking in Information Retrieval
Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, Benjamin Van Durme
2025-02-27
Summary
This paper talks about Rank1, a new way to make search engines better by using AI that can think and reason like humans do
What's the problem?
Current search engines struggle to understand complex questions or give detailed explanations for their answers. They also use a lot of computer power, which can be expensive and slow
What's the solution?
The researchers created Rank1, which uses AI models that can think through problems step-by-step, like OpenAI's o1. They trained Rank1 on over 600,000 examples of how these thinking AIs solve problems. This helps Rank1 understand questions better, explain its answers, and even work well with questions it wasn't specifically trained on
Why it matters?
This matters because it could make search engines much smarter and more helpful. Rank1 can give better answers to tricky questions, explain its thinking, and do all this without needing super powerful computers. This could lead to search engines that feel more like talking to a smart human helper, making it easier for everyone to find the information they need
Abstract
We introduce Rank1, the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAI's o1, Deepseek's R1, etc.) for distillation in order to rapidly improve the performance of a smaller model. We gather and open-source a dataset of more than 600,000 examples of R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due to the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or RAG-based systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/memory. Overall, Rank1 shows that test-time compute allows for a fundamentally new type of explainable and performant reranker model for search.