MisSynth: Improving MISSCI Logical Fallacies Classification with Synthetic Data
Mykhailo Poliakov, Nadiya Shvai
2025-11-03
Summary
This paper explores how to make computer programs better at spotting misleading or false information about health, specifically when that misinformation twists real scientific research.
What's the problem?
It's really hard for computers – and even people – to identify health misinformation, especially when it sounds plausible because it's based on a distorted version of actual science. There isn't a lot of clearly labeled data to *teach* computers what these fallacies look like, making it difficult to build reliable detection systems.
What's the solution?
The researchers created a system called MisSynth. It uses a technique where the computer first finds relevant information, then *generates* new examples of flawed arguments about health topics. Think of it like the computer creating practice problems for itself. They then used these computer-generated examples to slightly adjust an existing powerful language model, called LLaMA, to improve its ability to recognize these fallacies. This adjustment process is called 'fine-tuning' and doesn't require a huge amount of computing power.
Why it matters?
This work is important because it shows that we can significantly improve the ability of AI to detect health misinformation, even when we don't have a massive amount of human-labeled data. This is crucial for helping people access accurate health information and avoiding potentially harmful advice, and it can be done without needing supercomputers.
Abstract
Health-related misinformation is very prevalent and potentially harmful. It is difficult to identify, especially when claims distort or misinterpret scientific findings. We investigate the impact of synthetic data generation and lightweight fine-tuning techniques on the ability of large language models (LLMs) to recognize fallacious arguments using the MISSCI dataset and framework. In this work, we propose MisSynth, a pipeline that applies retrieval-augmented generation (RAG) to produce synthetic fallacy samples, which are then used to fine-tune an LLM model. Our results show substantial accuracy gains with fine-tuned models compared to vanilla baselines. For instance, the LLaMA 3.1 8B fine-tuned model achieved an over 35% F1-score absolute improvement on the MISSCI test split over its vanilla baseline. We demonstrate that introducing synthetic fallacy data to augment limited annotated resources can significantly enhance zero-shot LLM classification performance on real-world scientific misinformation tasks, even with limited computational resources. The code and synthetic dataset are available on https://github.com/mxpoliakov/MisSynth.