Hypencoder: Hypernetworks for Information Retrieval
Julian Killingback, Hansi Zeng, Hamed Zamani
2025-02-12
Summary
This paper talks about Hypencoder, a new way to make computer searches more accurate and efficient by using a special type of artificial intelligence called hypernetworks.
What's the problem?
Current search methods use a simple math trick to match queries with documents, which limits how well they can understand the true meaning behind searches. This means searches often miss important results or give irrelevant ones.
What's the solution?
The researchers created Hypencoder, which uses a small AI network to figure out how relevant a document is to a search query. This AI is created on the fly for each search using a bigger AI called a hypernetwork. Hypencoder was tested on various search tasks and performed better than existing methods, even on tricky searches like when you can't quite remember a word or need to follow complex instructions.
Why it matters?
This matters because it could make internet searches and other information retrieval tasks much more accurate and useful. Hypencoder can search through millions of documents super fast while understanding searches better than current methods. This could lead to big improvements in how we find information online, in databases, or in any large collection of documents.
Abstract
The vast majority of retrieval models depend on vector inner products to produce a relevance score between a query and a document. This naturally limits the expressiveness of the relevance score that can be employed. We propose a new paradigm, instead of producing a vector to represent the query we produce a small <PRE_TAG>neural network</POST_TAG> which acts as a learned relevance function. This small neural network takes in a representation of the document, in this paper we use a single vector, and produces a scalar relevance score. To produce the little neural network we use a hypernetwork, a network that produce the weights of other networks, as our query encoder or as we call it a Hypencoder. Experiments on in-domain search tasks show that Hypencoder is able to significantly outperform strong dense retrieval models and has higher metrics then reranking models and models an order of magnitude larger. Hypencoder is also shown to generalize well to out-of-domain search tasks. To assess the extent of Hypencoder's capabilities, we evaluate on a set of hard retrieval tasks including tip-of-the-tongue retrieval and instruction-following retrieval tasks and find that the performance gap widens substantially compared to standard retrieval tasks. Furthermore, to demonstrate the practicality of our method we implement an approximate search algorithm and show that our model is able to search 8.8M documents in under 60ms.