Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
Hippolyte Gisserot-Boukhlef, Ricardo Rei, Emmanuel Malherbe, Céline Hudelot, Pierre Colombo, Nuno M. Guerreiro
2024-10-03
Summary
This paper investigates whether aligning machine translation models with user preferences is always the best way to improve their performance, focusing on a method called Contrastive Preference Optimization (CPO).
What's the problem?
Machine translation (MT) models need to be evaluated on how well they translate text, and while newer neural metrics have improved this process, there is still a debate on the best way to optimize these models. Preference alignment techniques, which adjust the model based on user preferences, have gained popularity, but it’s unclear if they consistently lead to better translation quality across different situations.
What's the solution?
The researchers conducted experiments using CPO to see how it impacts translation quality compared to traditional methods like Supervised Fine-Tuning (SFT). They found that while CPO performed better on high-quality data regarding preference alignment, it sometimes caused inconsistencies when evaluated with different metrics. They also discovered that using just the base model for generating translations could achieve results similar to those produced by more complex systems, ensuring better consistency across various evaluation methods.
Why it matters?
This research is important because it helps clarify the effectiveness of preference alignment in improving machine translation models. By understanding the strengths and weaknesses of different optimization methods, developers can create more reliable and accurate translation systems, which are crucial for effective communication in our increasingly globalized world.
Abstract
Neural metrics for machine translation (MT) evaluation have become increasingly prominent due to their superior correlation with human judgments compared to traditional lexical metrics. Researchers have therefore utilized neural metrics through quality-informed decoding strategies, achieving better results than likelihood-based methods. With the rise of Large Language Models (LLMs), preference-based alignment techniques have gained attention for their potential to enhance translation quality by optimizing model weights directly on preferences induced by quality estimators. This study focuses on Contrastive Preference Optimization (CPO) and conducts extensive experiments to evaluate the impact of preference-based alignment on translation quality. Our findings indicate that while CPO consistently outperforms Supervised Fine-Tuning (SFT) on high-quality data with regard to the alignment metric, it may lead to instability across downstream evaluation metrics, particularly between neural and lexical ones. Additionally, we demonstrate that relying solely on the base model for generating candidate translations achieves performance comparable to using multiple external systems, while ensuring better consistency across downstream metrics.