LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
Nimrod Shabtay, Felipe Maia Polo, Sivan Doveh, Wei Lin, M. Jehanzeb Mirza, Leshem Chosen, Mikhail Yurochkin, Yuekai Sun, Assaf Arbelle, Leonid Karlinsky, Raja Giryes
2024-10-15
Summary
This paper introduces LiveXiv, a new benchmark designed to evaluate large multimodal models (LMMs) using scientific papers from ArXiv, allowing for real-time testing of these models' abilities.
What's the problem?
As AI models are trained on data from the web, there is a risk that the benchmarks used to test their abilities may become contaminated or outdated. This makes it hard to accurately assess how well these models can perform in real-world situations, especially when they need to understand and generate content that includes both text and images.
What's the solution?
LiveXiv addresses this issue by automatically generating visual question-answer pairs from current scientific papers without needing human input. It collects domain-specific manuscripts and uses a multi-modal model to create questions based on the visual content in those papers, like graphs and charts. The benchmark includes an efficient evaluation method that allows for testing only a small subset of models to estimate overall performance, significantly reducing costs.
Why it matters?
This research is important because it provides a scalable and up-to-date way to evaluate AI models in scientific fields. By using real scientific literature as a data source, LiveXiv helps ensure that the evaluations are relevant and accurate, which can lead to better AI systems that understand complex multimodal information.
Abstract
The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal (<2.5%). Our dataset is available online on HuggingFace, and our code will be available here.