Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent
Shanbo Cheng, Zhichao Huang, Tom Ko, Hang Li, Ningxin Peng, Lu Xu, Qini Zhang
2024-08-01
Summary
This paper presents CLASI, a new system for simultaneous speech translation that aims to perform at the level of human interpreters. It uses advanced techniques to translate spoken language in real-time while maintaining high quality.
What's the problem?
Current speech translation systems often struggle to provide accurate translations quickly, especially when dealing with complex or informal speech. Many existing systems rely on separate steps for recognizing speech and translating it, which can lead to errors and delays. This makes it challenging to achieve the same level of fluency and accuracy as a human interpreter.
What's the solution?
To address these issues, the authors developed CLASI, which integrates multiple advanced technologies into a single system. It uses a data-driven approach to balance translation quality and speed, allowing it to handle specific terminology effectively. CLASI also incorporates a multi-modal retrieval module that gathers relevant information to enhance translations. By considering the audio input, historical context, and additional data, CLASI can generate translations that are more accurate and natural-sounding.
Why it matters?
This research is important because it represents a significant step toward achieving human-like performance in speech translation. By improving the quality and efficiency of translations, CLASI can be used in various real-world situations, such as international conferences or multilingual meetings. This could greatly enhance communication across languages and cultures, making information more accessible to everyone.
Abstract
In this paper, we present Cross Language Agent -- Simultaneous Interpretation, CLASI, a high-quality and human-like Simultaneous Speech Translation (SiST) System. Inspired by professional human interpreters, we utilize a novel data-driven read-write strategy to balance the translation quality and latency. To address the challenge of translating in-domain terminologies, CLASI employs a multi-modal retrieving module to obtain relevant information to augment the translation. Supported by LLMs, our approach can generate error-tolerated translation by considering the input audio, historical context, and retrieved information. Experimental results show that our system outperforms other systems by significant margins. Aligned with professional human interpreters, we evaluate CLASI with a better human evaluation metric, valid information proportion (VIP), which measures the amount of information that can be successfully conveyed to the listeners. In the real-world scenarios, where the speeches are often disfluent, informal, and unclear, CLASI achieves VIP of 81.3% and 78.0% for Chinese-to-English and English-to-Chinese translation directions, respectively. In contrast, state-of-the-art commercial or open-source systems only achieve 35.4% and 41.6%. On the extremely hard dataset, where other systems achieve under 13% VIP, CLASI can still achieve 70% VIP.