LOGO -- Long cOntext aliGnment via efficient preference Optimization
Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, Min Zhang
2024-10-25
Summary
This paper introduces LOGO, a new training strategy designed to improve the performance of long-context models (LCMs) in generating accurate responses by optimizing how they process long sequences of information.
What's the problem?
Long-context models can handle very large amounts of text, but they often produce incorrect or misaligned answers, known as hallucinations. This happens because existing methods for training these models either require too much data or are not efficient enough, leading to subpar performance.
What's the solution?
LOGO uses a method called preference optimization to help LCMs better align their outputs with the correct information. It does this by breaking down the training process into smaller parts that are easier to manage, which helps avoid memory issues that can occur with long sequences. By training on a smaller dataset of 0.3 billion tokens for just 16 hours on a powerful GPU, LOGO allows the model to perform comparably to state-of-the-art models like GPT-4 while still being effective in other tasks.
Why it matters?
This research is important because it enhances how AI models understand and generate long texts, making them more reliable and accurate. Improving LCMs can lead to better applications in fields like natural language processing, where understanding context over long passages is crucial.
Abstract
Long-context models(LCMs) have shown great potential in processing long input sequences(even more than 100M tokens) conveniently and effectively. With significant progress, recent research has pointed out that LCMs can accurately locate token-level salient information within the context. Yet, the generation performance of these LCMs is far from satisfactory and might result in misaligned responses, such as hallucinations. To enhance the generation capability of LCMs, existing works have investigated the effects of data size and quality for both pre-training and instruction tuning. Though achieving meaningful improvement, previous methods fall short in either effectiveness or efficiency. In this paper, we introduce LOGO(Long cOntext aliGnment via efficient preference Optimization), a training strategy that first introduces preference optimization for long-context alignment. To overcome the GPU memory-bound issue caused by the long sequence, LOGO employs a reference-free preference optimization strategy and adopts a position synthesis method to construct the training data. By training with only 0.3B data on a single 8timesA800 GPU machine for 16 hours, LOGO allows the Llama-3-8B-Instruct-80K model to achieve comparable performance with GPT-4 in real-world long-context tasks while preserving the model's original capabilities on other tasks, e.g., language modeling and MMLU. Moreover, LOGO can extend the model's context window size while enhancing its generation performance.