Task Me Anything
Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, Ranjay Krishna
2024-06-18
Summary
This paper introduces Task-Me-Anything, a tool designed to create customized benchmarks for evaluating large multimodal language models (MLMs). It helps developers find the right model for their specific tasks by generating tailored assessments based on user needs.
What's the problem?
Currently, there are many benchmarks available to evaluate MLMs, but they often assess general capabilities rather than specific tasks. This makes it difficult for developers to choose the right benchmark that accurately reflects their application's requirements. As a result, they may feel overwhelmed by the options and unsure which model will perform best for their needs.
What's the solution?
Task-Me-Anything addresses this problem by allowing users to create their own benchmarks tailored to their specific queries. It has a large collection of visual assets, including images, videos, and 3D objects, which can be used to generate a vast number of task instances. The system can produce up to 750 million question-answering pairs that test how well MLMs understand and respond to visual information. Additionally, it provides insights into the strengths and weaknesses of different models, helping users make informed decisions.
Why it matters?
This research is important because it simplifies the process of evaluating and selecting multimodal language models. By providing a way to create customized benchmarks, Task-Me-Anything enhances the ability of developers to assess model performance accurately. This can lead to better applications in various fields, such as AI development, education, and content creation, where understanding both text and images is crucial.
Abstract
Benchmarks for large multimodal language models (MLMs) now serve to simultaneously assess the general capabilities of models instead of evaluating for a specific capability. As a result, when a developer wants to identify which models to use for their application, they are overwhelmed by the number of benchmarks and remain uncertain about which benchmark's results are most reflective of their specific use case. This paper introduces Task-Me-Anything, a benchmark generation engine which produces a benchmark tailored to a user's needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and can programmatically generate a vast number of task instances. Additionally, it algorithmically addresses user queries regarding MLM performance efficiently within a computational budget. It contains 113K images, 10K videos, 2K 3D object assets, over 365 object categories, 655 attributes, and 335 relationships. It can generate 750M image/video question-answering pairs, which focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals critical insights: open-source MLMs excel in object and attribute recognition but lack spatial and temporal understanding; each model exhibits unique strengths and weaknesses; larger models generally perform better, though exceptions exist; and GPT4o demonstrates challenges in recognizing rotating/moving objects and distinguishing colors.