BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks
Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats Leon Richter, Saverio Vadacchino, Shubbam Agarwal, Sanket Biswas, Sara Shanian, Ying Zhang
2024-12-09
Summary
This paper talks about BigDocs, a new open-access dataset designed to improve how AI models understand and work with documents and code by providing a large collection of multimodal data.
What's the problem?
AI models that work with documents often struggle because they lack access to high-quality training data. Most existing datasets are either too limited or have restrictive licenses, making it hard for researchers and developers to use them effectively. This limits the ability of AI to process complex documents like receipts, reports, and code outputs.
What's the solution?
The authors created BigDocs-7.5M, which includes 7.5 million multimodal documents across 30 different tasks. This dataset is open-access and has been carefully curated to ensure high quality and permissive licensing. They also developed BigDocs-Bench, a benchmark suite with 10 new tasks that reflect real-world scenarios, such as reasoning over graphical user interfaces and generating code from images. Their experiments showed that using BigDocs significantly improves the performance of AI models on various tasks compared to existing closed-source models.
Why it matters?
This research is important because it provides a valuable resource for the AI community, enabling better training of models that can understand and interact with complex documents. By making this dataset available, BigDocs encourages innovation in multimodal AI applications, helping to enhance capabilities in fields like finance, education, and software development.
Abstract
Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure our data is high-quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench, a benchmark suite with 10 novel tasks where we create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning. The project is hosted at https://bigdocs.github.io .