PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback
Kanika Goswami, Puneet Mathur, Ryan Rossi, Franck Dernoncourt
2025-02-07
Summary
This paper talks about PlotGen, a new AI system that helps create accurate scientific charts and graphs automatically. It uses multiple AI agents working together to understand what users want, generate the right code, and improve the visuals through feedback.
What's the problem?
Making good scientific charts is hard, especially for beginners. It's tricky to choose the right tools and learn how to use them properly. While AI language models can help write code, they often make mistakes that need fixing.
What's the solution?
The researchers created PlotGen, which uses several AI agents to tackle different parts of making a chart. One agent breaks down the user's request into steps, another writes the code, and three more check and improve the chart's numbers, labels, and overall look. These agents work together and learn from their mistakes to make better charts.
Why it matters?
This matters because it makes creating scientific charts easier and more accurate, even for people who aren't experts. It helps scientists and students spend less time fixing errors in their charts and more time understanding their data. By improving how we visualize scientific information, PlotGen could help speed up research and make it easier to share scientific findings with others.
Abstract
Scientific data visualization is pivotal for transforming raw data into comprehensible visual representations, enabling pattern recognition, forecasting, and the presentation of data-driven insights. However, novice users often face difficulties due to the complexity of selecting appropriate tools and mastering visualization techniques. Large Language Models (LLMs) have recently demonstrated potential in assisting code generation, though they struggle with accuracy and require iterative debugging. In this paper, we propose PlotGen, a novel multi-agent framework aimed at automating the creation of precise scientific visualizations. PlotGen orchestrates multiple LLM-based agents, including a Query Planning Agent that breaks down complex user requests into executable steps, a Code Generation Agent that converts pseudocode into executable Python code, and three retrieval feedback agents - a Numeric Feedback Agent, a Lexical Feedback Agent, and a Visual Feedback Agent - that leverage multimodal LLMs to iteratively refine the data accuracy, textual labels, and visual correctness of generated plots via self-reflection. Extensive experiments show that PlotGen outperforms strong baselines, achieving a 4-6 percent improvement on the MatPlotBench dataset, leading to enhanced user trust in LLM-generated visualizations and improved novice productivity due to a reduction in debugging time needed for plot errors.