ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution
Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt
2025-02-07
Summary
This paper talks about ChartCitor, a new system that helps AI models answer questions about charts more accurately by identifying specific parts of the chart that support the answers.
What's the problem?
When AI models answer questions about charts, they often make mistakes or give answers that aren't backed up by the actual data in the chart. Existing methods struggle to connect the answers to specific parts of the chart because of challenges like aligning text and visuals or handling complex layouts.
What's the solution?
The researchers created ChartCitor, which uses multiple AI agents to break down and analyze charts. It extracts data from charts, reformulates answers, and maps specific parts of the chart (like bars or lines) to the AI's responses. This system provides clear visual evidence for each answer, making it easier to verify the accuracy of the AI's outputs.
Why it matters?
This research is important because it makes AI-generated answers about charts more trustworthy and easier to check. By improving how AI handles chart data, ChartCitor could save time for professionals in fields like business and research, where analyzing charts is a common task.
Abstract
Large Language Models (LLMs) can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground responses in source charts due to limited visual-semantic context, complex visual-text alignment requirements, and difficulties in bounding box prediction across complex layouts. We present ChartCitor, a multi-agent framework that provides fine-grained bounding box citations by identifying supporting evidence within chart images. The system orchestrates LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval through pre-filtering and re-ranking, and table-to-chart mapping. ChartCitor outperforms existing baselines across different chart types. Qualitative user studies show that ChartCitor helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted chart QA and enables professionals to be more productive.