Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning
Yujian Liu, Shiyu Chang, Tommi Jaakkola, Yang Zhang
2024-10-28
Summary
This paper discusses a new method called Prereq-Tune, which aims to improve the accuracy of large language models (LLMs) by addressing inconsistencies in knowledge during their training process.
What's the problem?
Large language models can sometimes produce incorrect or misleading information, a phenomenon known as 'hallucination.' This often happens because the data used to fine-tune these models can differ significantly from the data they were originally trained on, leading to confusion. When the model encounters unfamiliar information during fine-tuning, it may generate plausible but incorrect outputs.
What's the solution?
To tackle this issue, the authors propose Prereq-Tune, a strategy that separates the learning of skills from the learning of knowledge. This involves adding a new stage called prerequisite learning, where the model learns necessary background knowledge before it focuses on specific tasks. Additionally, they suggest using fictitious synthetic data to help ground the model's outputs in its internal knowledge. Their experiments show that Prereq-Tune leads to better performance in generating accurate responses compared to traditional methods.
Why it matters?
This research is important because it helps improve the reliability of language models, making them more accurate and trustworthy in providing information. By reducing hallucinations and enhancing factuality, this method can lead to better applications of AI in areas like education, healthcare, and customer service, where accurate information is crucial.
Abstract
Recent studies have identified one aggravating factor of LLM hallucinations as the knowledge inconsistency between pre-training and fine-tuning, where unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong outputs. In this paper, we propose a novel fine-tuning strategy called Prereq-Tune to address this knowledge inconsistency and reduce hallucinations. Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge, so the model learns only the task skills without being impacted by the knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional prerequisite learning stage to learn the necessary knowledge for SFT, allowing subsequent SFT to focus only on task skills. Prereq-Tune can also be combined with fictitious synthetic data to enhance the grounding of LLM outputs to their internal knowledge. Experiments show that Prereq-Tune outperforms existing baselines in improving LLM's factuality across short QA and long-form generation tasks. It also opens new possibilities for knowledge-controlled generation in LLMs. Our code is available at https://github.com/UCSB-NLP-Chang/Prereq_tune.git.