One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework
Lorenzo Bianchi, Giacomo Pacini, Fabio Carrara, Nicola Messina, Giuseppe Amato, Fabrizio Falchi
2025-10-13
Summary
This paper introduces a new way to automatically write descriptions for parts of images, not just the whole image, without needing specific training data for those parts.
What's the problem?
Existing automatic image captioning systems typically focus on describing the entire image as a whole. They don't easily allow you to describe specific regions or objects *within* an image without being specifically trained to do so. This limits their usefulness when you need detailed descriptions of particular areas.
What's the solution?
The researchers developed a system that breaks down images into smaller pieces, called 'patches'. Instead of trying to understand the whole image at once, it generates descriptions for each patch individually and then combines those descriptions to explain any region you want, from a single patch to the entire image. They found that using image analysis techniques that create detailed and meaningful representations of these patches, like DINO, is crucial for good results.
Why it matters?
This is important because it allows for more flexible and detailed image descriptions. Instead of just getting 'a cat on a couch', you could get 'a fluffy orange cat on the left side of a blue couch'. This is useful for applications like helping visually impaired people understand images, or for more precise image searching and analysis.
Abstract
Zero-shot captioners are recently proposed models that utilize common-space vision-language representations to caption images without relying on paired image-text data. To caption an image, they proceed by textually decoding a text-aligned image feature, but they limit their scope to global representations and whole-image captions. We present , a unified framework for zero-shot captioning that shifts from an image-centric to a patch-centric paradigm, enabling the captioning of arbitrary regions without the need of region-level supervision. Instead of relying on global image representations, we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions, from single patches to non-contiguous areas and entire images. We analyze the key ingredients that enable current latent captioners to work in our novel proposed framework. Experiments demonstrate that backbones producing meaningful, dense visual features, such as DINO, are key to achieving state-of-the-art performance in multiple region-based captioning tasks. Compared to other baselines and state-of-the-art competitors, our models achieve better performance on zero-shot dense, region-set, and a newly introduced trace captioning task, highlighting the effectiveness of patch-wise semantic representations for scalable caption generation. Project page at https://paciosoft.com/Patch-ioner/ .