GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training
Renqiu Xia, Mingsheng Li, Hancheng Ye, Wenjie Wu, Hongbin Zhou, Jiakang Yuan, Tianshuo Peng, Xinyu Cai, Xiangchao Yan, Bin Wang, Conghui He, Botian Shi, Tao Chen, Junchi Yan, Bo Zhang
2024-12-17
Summary
This paper talks about GeoX, a new AI model designed to solve geometric problems by combining visual and language understanding, making it better at interpreting diagrams and symbols.
What's the problem?
Many existing AI models struggle with solving geometry problems because they are mainly trained on regular images and text. This means they have difficulty understanding complex diagrams and performing the reasoning needed for geometry tasks. Additionally, specialized models for geometry often focus on specific problems, making them less effective for a wider range of geometric challenges.
What's the solution?
GeoX addresses these issues by using a multi-modal approach that includes both visual and language components. It incorporates a diagram encoder to help the model understand geometric images and a symbol decoder to interpret the symbols used in geometry. The training process involves three stages: first, it learns to recognize diagrams; second, it aligns geometric concepts with language; and third, it fine-tunes its ability to solve problems based on visual instructions. This comprehensive training allows GeoX to generate accurate solutions for various geometric problems.
Why it matters?
This research is important because it enhances the capabilities of AI in solving complex geometric problems, which can be useful in education, engineering, and computer graphics. By improving how AI understands and processes geometry, GeoX can help students learn better and assist professionals in fields that require precise geometric calculations.
Abstract
Despite their proficiency in general tasks, Multi-modal Large Language Models (MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands understanding diagrams, interpreting symbols, and performing complex reasoning. This limitation arises from their pre-training on natural images and texts, along with the lack of automated verification in the problem-solving process. Besides, current geometric specialists are limited by their task-specific designs, making them less effective for broader geometric problems. To this end, we present GeoX, a multi-modal large model focusing on geometric understanding and reasoning tasks. Given the significant differences between geometric diagram-symbol and natural image-text, we introduce unimodal pre-training to develop a diagram encoder and symbol decoder, enhancing the understanding of geometric images and corpora. Furthermore, we introduce geometry-language alignment, an effective pre-training paradigm that bridges the modality gap between unimodal geometric experts. We propose a Generator-And-Sampler Transformer (GS-Former) to generate discriminative queries and eliminate uninformative representations from unevenly distributed geometric signals. Finally, GeoX benefits from visual instruction tuning, empowering it to take geometric images and questions as input and generate verifiable solutions. Experiments show that GeoX outperforms both generalists and geometric specialists on publicly recognized benchmarks, such as GeoQA, UniGeo, Geometry3K, and PGPS9k.