Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Yang Shen, Xiu-Shen Wei, Yifan Sun, Yuxin Song, Tao Yuan, Jian Jin, Heyang Xu, Yazhou Yao, Errui Ding
2024-12-31
Summary
This paper talks about Explanatory Instructions, a new approach to help computer vision (CV) systems understand and perform tasks without needing specific training for each task, similar to how language models operate.
What's the problem?
Computer vision systems struggle to generalize their learning to new tasks without having seen examples of those tasks before. Unlike natural language processing, where models can often understand new tasks through context and descriptions, CV systems tend to rely on strict definitions of tasks (like 'image segmentation'), which limits their ability to adapt to new situations. This makes it difficult for these systems to perform well in real-world applications where they encounter unfamiliar tasks.
What's the solution?
To address this issue, the authors introduce Explanatory Instructions, which provide a way to define CV tasks using detailed descriptions that explain what the system should do with an image. They created a large dataset with 12 million examples that include images, instructions, and expected outputs. By training a vision-language model on this dataset, the system learns to follow these instructions and can perform well on both familiar and new tasks without needing additional training for each one.
Why it matters?
This research is important because it enhances the flexibility and capability of computer vision systems. By enabling these systems to understand and adapt to new tasks using explanatory instructions, it can improve their performance in various applications such as robotics, healthcare, and autonomous vehicles. This approach could lead to more intelligent AI systems that can operate effectively in dynamic environments.
Abstract
Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input to explanatory instruction to output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.