Early-Exit and Instant Confidence Translation Quality Estimation
Vilém Zouhar, Maike Züfle, Beni Egressy, Julius Cheng, Jan Niehues
2025-02-25
Summary
This paper talks about new methods to make machine translation quality estimation faster and more efficient while maintaining accuracy, using models called Instant Confidence COMET and Early-Exit COMET.
What's the problem?
Quality estimation for machine translation is important for checking how good translations are, but current methods are slow, expensive, and not very practical for large-scale use. They also lack clear ways to measure uncertainty in their predictions.
What's the solution?
The researchers introduced two new models: Instant Confidence COMET, which provides quick and accurate quality estimates with lower costs, and Early-Exit COMET, which can stop computations early while still giving reliable results. They also applied these models to tasks like reranking translations, reducing the computing power needed by 50% without losing much accuracy.
Why it matters?
This matters because it makes machine translation systems more efficient and accessible, especially for large-scale projects. By cutting down on costs and computation time, these methods could help improve translation tools used in industries like global communication, localization, and content creation.
Abstract
Quality estimation is omnipresent in machine translation, for both evaluation and generation. Unfortunately, quality estimation models are often opaque and computationally expensive, making them impractical to be part of large-scale pipelines. In this work, we tackle two connected challenges: (1) reducing the cost of quality estimation at scale, and (2) developing an inexpensive uncertainty estimation method for quality estimation. To address the latter, we introduce Instant Confidence COMET, an uncertainty-aware <PRE_TAG>quality estimation</POST_TAG> model that matches the performance of previous approaches at a fraction of their costs. We extend this to Early-Exit COMET, a quality estimation model that can compute quality scores and associated confidences already at early model layers, allowing us to early-exit computations and reduce evaluation costs. We also apply our model to <PRE_TAG>machine translation reranking</POST_TAG>. We combine Early-Exit COMET with an upper confidence bound bandit algorithm to find the best candidate from a large pool without having to run the full evaluation model on all candidates. In both cases (evaluation and reranking) our methods reduce the required compute by 50% with very little degradation in performance.