Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinement
Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, Kai Chen
2025-01-22
Summary
This paper talks about a new way to create high-quality training data for AI language models called Condor. It's like teaching a smart computer to have better conversations by giving it specially crafted practice examples.
What's the problem?
As AI language models get smarter, they need more and better training data to improve their conversation skills. Getting this data from real people is becoming too slow and expensive. So, researchers need to find a way to create good training data artificially.
What's the solution?
The researchers created Condor, which is like a two-step recipe for making artificial training data. First, it uses something called a World Knowledge Tree to organize information. Then, it has a Self-Reflection Refinement step where the AI looks at its own work and tries to make it better. They tested Condor by training an AI model with just 20,000 examples made by Condor, and it worked really well.
Why it matters?
This matters because it could make AI language models a lot better without needing tons of human-made training data. It's like giving AI a supercharged study guide that helps it learn faster and better. The researchers also found that there's still a lot of room for improvement, which means we might be able to make AI even smarter in the future using this method. This could lead to AI assistants that are much better at understanding and talking to people in natural ways.
Abstract
The quality of Supervised Fine-Tuning (SFT) data plays a critical role in enhancing the conversational capabilities of Large Language Models (LLMs). However, as LLMs become more advanced, the availability of high-quality human-annotated SFT data has become a significant bottleneck, necessitating a greater reliance on synthetic training data. In this work, we introduce Condor, a novel two-stage synthetic data generation framework that incorporates World Knowledge Tree and Self-Reflection Refinement to produce high-quality SFT data at scale. Our experimental results demonstrate that a base model fine-tuned on only 20K Condor-generated samples achieves superior performance compared to counterparts. The additional refinement stage in Condor further enables iterative self-improvement for LLMs at various scales (up to 72B), validating the effectiveness of our approach. Furthermore, our investigation into the scaling for synthetic data in post-training reveals substantial unexplored potential for performance improvements, opening promising avenues for future research.