RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response
Junyu Luo, Xiao Luo, Kaize Ding, Jingyang Yuan, Zhiping Xiao, Ming Zhang
2024-12-24
Summary
This paper talks about RobustFT, a new framework designed to improve the fine-tuning of large language models (LLMs) by making them more effective at handling noisy data during training.
What's the problem?
When training LLMs for specific tasks, the data used often contains errors or 'noise' that can confuse the model and reduce its performance. This noise can come from mistakes in human annotations or inaccuracies from other models. As a result, there is a need for a method that helps LLMs learn better despite this noisy data.
What's the solution?
RobustFT addresses this problem by introducing a two-step process: first, it identifies and detects noise in the training data using a system of multiple expert models. Then, it cleans up this data by relabeling it based on reliable information. The framework also includes a mechanism to select only high-quality samples for fine-tuning, ensuring that the model learns from the best examples available. Extensive experiments show that RobustFT significantly improves model performance even when dealing with noisy data.
Why it matters?
This research is important because it enhances how LLMs can be trained in real-world situations where data is often imperfect. By improving the ability of these models to learn from noisy information, RobustFT can lead to better performance in various applications, making AI systems more reliable and effective.
Abstract
Supervised fine-tuning (SFT) plays a crucial role in adapting large language models (LLMs) to specific domains or tasks. However, as demonstrated by empirical experiments, the collected data inevitably contains noise in practical applications, which poses significant challenges to model performance on downstream tasks. Therefore, there is an urgent need for a noise-robust SFT framework to enhance model capabilities in downstream tasks. To address this challenge, we introduce a robust SFT framework (RobustFT) that performs noise detection and relabeling on downstream task data. For noise identification, our approach employs a multi-expert collaborative system with inference-enhanced models to achieve superior noise detection. In the denoising phase, we utilize a context-enhanced strategy, which incorporates the most relevant and confident knowledge followed by careful assessment to generate reliable annotations. Additionally, we introduce an effective data selection mechanism based on response entropy, ensuring only high-quality samples are retained for fine-tuning. Extensive experiments conducted on multiple LLMs across five datasets demonstrate RobustFT's exceptional performance in noisy scenarios.