GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
M. Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh, Wei Lin, Paul Gavrikov, Michael Dorkenwald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, Yuki Mitsufuji, Horst Possegger, Rogerio Feris, Leonid Karlinsky, James Glass
2024-10-13
Summary
This paper introduces GLOV, a new method that allows large language models (LLMs) to help improve the performance of vision-language models (VLMs) by optimizing how they generate prompts for tasks like image recognition.
What's the problem?
Vision-language models are designed to understand and generate content that involves both images and text, but they often struggle with specific tasks. Current methods for improving these models can be inefficient and do not fully utilize the capabilities of LLMs, which can lead to subpar performance in tasks that require understanding complex visual and textual information.
What's the solution?
To address this issue, the authors developed GLOV, which uses LLMs as implicit optimizers for VLMs. GLOV works by first describing the task to the LLM and then asking it to suggest prompts that would help the VLM perform better. These prompts are ranked based on how effective they are for the task. The LLM learns from previous successes and failures in generating prompts, adjusting its approach each time to improve performance. This method allows for better task-specific prompt generation without needing to change the VLM directly.
Why it matters?
This research is significant because it enhances the ability of AI systems to perform complex vision-language tasks more effectively. By leveraging LLMs in this way, GLOV can lead to improvements in applications such as image captioning, visual question answering, and other areas where understanding both text and images is crucial. This could result in more accurate and efficient AI systems across various fields.
Abstract
In this work, we propose a novel method (GLOV) enabling Large Language Models (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP). These prompts are ranked according to a purity measure obtained through a fitness function. In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of text prompts preferred by the downstream VLM. Furthermore, we also explicitly steer the LLM generation process in each optimization step by specifically adding an offset difference vector of the embeddings from the positive and negative solutions found by the LLM, in previous optimization steps, to the intermediate layer of the network for the next generation step. This offset vector steers the LLM generation toward the type of language preferred by the downstream VLM, resulting in enhanced performance on the downstream vision tasks. We comprehensively evaluate our GLOV on 16 diverse datasets using two families of VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models -- showing that the discovered solutions can enhance the recognition performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these models.