In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation
Armel Zebaze, Benoît Sagot, Rachel Bawden
2024-08-05
Summary
This paper discusses a method called In-Context Example Selection via Similarity Search, which improves machine translation (MT) by choosing the best translation examples based on their similarity to the text being translated. This is especially helpful for languages that have fewer resources available for training models.
What's the problem?
Machine translation often relies on examples to help translate text accurately. However, there hasn't been a systematic way to select these examples, leading to mixed results when using random selections. This is particularly problematic for low-resource languages, where there isn't much training data available, making it hard for models to learn effectively.
What's the solution?
The authors conducted a study comparing different methods of selecting translation examples using similarity search with multilingual sentence embeddings. They tested this approach across various language pairs, including English to French, German, Swahili, and Wolof. Their findings showed that selecting examples based on similarity significantly improved translation quality for low-resource languages compared to random selection. They also suggested a new evaluation method for measuring the performance of translation models.
Why it matters?
This research is important because it enhances the effectiveness of machine translation systems, especially for languages that lack sufficient training data. By improving how examples are selected for translations, this method can lead to better communication and understanding across different languages, making technology more accessible to speakers of low-resource languages.
Abstract
The ability of generative large language models (LLMs) to perform in-context learning has given rise to a large body of research into how best to prompt models for various natural language processing tasks. In this paper, we focus on machine translation (MT), a task that has been shown to benefit from in-context translation examples. However no systematic studies have been published on how best to select examples, and mixed results have been reported on the usefulness of similarity-based selection over random selection. We provide a study covering multiple LLMs and multiple in-context example retrieval strategies, comparing multilingual sentence embeddings. We cover several language directions, representing different levels of language resourcedness (English into French, German, Swahili and Wolof). Contrarily to previously published results, we find that sentence embedding similarity can improve MT, especially for low-resource language directions, and discuss the balance between selection pool diversity and quality. We also highlight potential problems with the evaluation of LLM-based MT and suggest a more appropriate evaluation protocol, adapting the COMET metric to the evaluation of LLMs. Code and outputs are freely available at https://github.com/ArmelRandy/ICL-MT.