Can this Model Also Recognize Dogs? Zero-Shot Model Search from Weights
Jonathan Kahana, Or Nathan, Eliahu Horwitz, Yedid Hoshen
2025-02-14
Summary
This paper talks about ProbeLog, a new method that helps find AI models capable of recognizing specific concepts, like 'dogs,' even if the models were not trained for that exact purpose. It uses a special way of analyzing how models respond to certain inputs to make searching for them more accurate and efficient.
What's the problem?
There are many AI models available online, but finding the right one for a specific task is difficult because current search methods rely on basic text searches in documentation. This doesn't work well if the model's metadata or training data isn't available, making it hard to identify models that can perform specific tasks.
What's the solution?
The researchers developed ProbeLog, which creates a unique 'descriptor' for each output of an AI model by observing how it reacts to a set of test inputs. This allows users to search for models based on their actual capabilities rather than relying on incomplete metadata. ProbeLog also supports zero-shot searches, meaning users can search for concepts like 'dogs' without needing prior examples. To make this process faster and less expensive, they introduced a collaborative filtering technique that reduces the computational cost of creating these descriptors.
Why it matters?
This matters because it makes it easier to find and use AI models for specific tasks without needing detailed information about their training. By improving how we search for models, ProbeLog could save time and resources while enabling more effective use of existing AI technologies across various fields, such as healthcare, education, or business.
Abstract
With the increasing numbers of publicly available models, there are probably pretrained, online models for most tasks users require. However, current model search methods are rudimentary, essentially a text-based search in the documentation, thus users cannot find the relevant models. This paper presents ProbeLog, a method for retrieving classification models that can recognize a target concept, such as "Dog", without access to model metadata or training data. Differently from previous probing methods, ProbeLog computes a descriptor for each output dimension (logit) of each model, by observing its responses on a fixed set of inputs (probes). Our method supports both logit-based retrieval ("find more logits like this") and zero-shot, text-based retrieval ("find all logits corresponding to dogs"). As probing-based representations require multiple costly feedforward passes through the model, we develop a method, based on collaborative filtering, that reduces the cost of encoding repositories by 3x. We demonstrate that ProbeLog achieves high retrieval accuracy, both in real-world and fine-grained search tasks and is scalable to full-size repositories.