Better Alignment with Instruction Back-and-Forth Translation
Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, Xian Li
2024-08-09
Summary
This paper introduces a new method called instruction back-and-forth translation, which helps improve how well large language models (LLMs) understand and follow instructions by creating high-quality synthetic data.
What's the problem?
As large language models become more common, it’s important that they can accurately understand and respond to instructions. However, existing methods for training these models often use datasets that are not very effective, leading to poor performance. This is especially problematic in situations where precise instruction following is critical, like in education or automated tasks.
What's the solution?
The authors developed a technique called instruction back-and-forth translation. This involves translating original instructions into another language and then translating them back to the original language. This process helps create clearer and more natural instructions for the models. The researchers then fine-tune the LLMs using these improved instruction-response pairs, resulting in better performance compared to other training methods. They found that their approach produced higher quality instructions and more diverse responses than traditional methods.
Why it matters?
This research is significant because it enhances the ability of AI systems to understand complex instructions better. By improving how LLMs are trained, this method can lead to more reliable applications in various fields, such as virtual assistants, educational tools, and other automated systems where clear communication is essential.
Abstract
We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). Given documents from a web corpus, we generate and curate synthetic instructions using the backtranslation approach proposed by Li et al.(2023a), and rewrite the responses to improve their quality further based on the initial documents. Fine-tuning with the resulting (backtranslated instruction, rewritten response) pairs yields higher win rates on AlpacaEval than using other common instruction datasets such as Humpback, ShareGPT, Open Orca, Alpaca-GPT4 and Self-instruct. We also demonstrate that rewriting the responses with an LLM outperforms direct distillation, and the two generated text distributions exhibit significant distinction in embedding space. Further analysis shows that our backtranslated instructions are of higher quality than other sources of synthetic instructions, while our responses are more diverse and complex than those obtained from distillation. Overall we find that instruction back-and-forth translation combines the best of both worlds -- making use of the information diversity and quantity found on the web, while ensuring the quality of the responses which is necessary for effective alignment.