MATATA: a weak-supervised MAthematical Tool-Assisted reasoning for Tabular Applications
Vishnou Vinayagame, Gregory Senay, Luis MartÃ
2024-12-02
Summary
This paper introduces MATATA, a new method for improving mathematical reasoning in language models that work with tabular data, using weak supervision and tool assistance.
What's the problem?
Many existing methods for mathematical reasoning rely on large, closed-source models or require a lot of manual effort to create prompts. This can be problematic, especially in sensitive business contexts where data privacy is important. Additionally, these methods often need extensive external data and may not be efficient in their use of resources.
What's the solution?
MATATA addresses these challenges by using small language models (SLMs) that can be hosted locally and trained with minimal external data. It employs a self-improvement approach that allows the model to learn from its own performance over time. The framework uses weak supervision to guide the model's reasoning process and incorporates flexible tools that can be reused across different datasets. This results in strong performance on tasks like FinQA and TAT-QA without needing large models or extensive prompt engineering.
Why it matters?
This research is important because it provides a cost-effective way to enhance the capabilities of language models in handling complex mathematical reasoning tasks, particularly with tabular data. By reducing reliance on large models and ensuring data privacy, MATATA can be especially useful in business applications where sensitive information is involved. This approach also opens the door for more accessible AI tools that can perform sophisticated reasoning without requiring massive computational resources.
Abstract
Mathematical reasoning capabilities are increasing with tool-augmented language agents, but methods often rely either on closed-source or large models, external data, or extensive prompt engineering. This work introduces MATATA, a novel cost-effective method to train LLM agents for tabular data problems through reasoning, planning, and tool use. With a progressive self-improvement paradigm and an iterative weak supervision, it empowers 3.8B/8B Small Language Models (SLMs), particularly suited for local hosting and sensitive business contexts where data privacy is crucial. By employing a flexible and reusable tools across different datasets, it achieves robust performance with effective scalability across shared tasks. Experiments show that MATATA reaches state-of-the-art performances on FinQA and TAT-QA among reasoning frameworks based on open-source models. Moreover, MATATA models compete with GPT-4 based frameworks on TabMWP, while being SLMs.