HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
William Watson, Nicole Cho, Tucker Balch, Manuela Veloso
2024-06-18
Summary
This paper introduces HiddenTables and PyQTax, a new cooperative game and dataset designed to improve how large language models (LLMs) handle question-answering tasks related to tables while ensuring data privacy.
What's the problem?
Many LLMs struggle with analyzing tables for question-answering tasks due to several challenges. These include limitations on how much information they can process at once (finite context windows), difficulties in understanding the structure of tables (like how to interpret cell boundaries), and concerns about keeping data confidential when using external models. These issues make it hard for LLMs to provide accurate answers based on table data.
What's the solution?
To address these problems, the authors created a cooperative game called HiddenTables. In this game, one LLM acts as a 'Solver' that generates code to answer questions about tables, while another part, called the 'Oracle,' evaluates how well the Solver performs. This setup allows for testing LLMs' abilities in a secure way, without exposing sensitive data. The authors also developed a new dataset named PyQTax, which contains over 116,000 question-table-answer triplets, providing detailed labels for different types of questions. This combination helps improve the efficiency and accuracy of LLMs in handling table-related queries.
Why it matters?
This research is important because it offers a new approach to help AI models better understand and answer questions about tables while protecting sensitive information. By creating a structured way for LLMs to interact with complex datasets, HiddenTables and PyQTax could enhance applications in areas like data analysis, business intelligence, and any field that relies on interpreting tabular data. This advancement helps ensure that AI technologies can be used effectively and responsibly.
Abstract
A myriad of different Large Language Models (LLMs) face a common challenge in contextually analyzing table question-answering tasks. These challenges are engendered from (1) finite context windows for large tables, (2) multi-faceted discrepancies amongst tokenization patterns against cell boundaries, and (3) various limitations stemming from data confidentiality in the process of using external models such as gpt-3.5-turbo. We propose a cooperative game dubbed "HiddenTables" as a potential resolution to this challenge. In essence, "HiddenTables" is played between the code-generating LLM "Solver" and the "Oracle" which evaluates the ability of the LLM agents to solve Table QA tasks. This game is based on natural language schemas and importantly, ensures the security of the underlying data. We provide evidential experiments on a diverse set of tables that demonstrate an LLM's collective inability to generalize and perform on complex queries, handle compositional dependencies, and align natural language to programmatic commands when concrete table schemas are provided. Unlike encoder-based models, we have pushed the boundaries of "HiddenTables" to not be limited by the number of rows - therefore we exhibit improved efficiency in prompt and completion tokens. Our infrastructure has spawned a new dataset "PyQTax" that spans across 116,671 question-table-answer triplets and provides additional fine-grained breakdowns & labels for varying question taxonomies. Therefore, in tandem with our academic contributions regarding LLMs' deficiency in TableQA tasks, "HiddenTables" is a tactile manifestation of how LLMs can interact with massive datasets while ensuring data security and minimizing generation costs.