On Non-interactive Evaluation of Animal Communication Translators
Orr Paradise, David F. Gruber, Adam Tauman Kalai
2025-10-21
Summary
This paper explores how to tell if an AI that translates animal languages, like whale songs into English, is actually working well, without needing to directly communicate with the animals themselves.
What's the problem?
Normally, to check if a machine translation is good, you compare it to a correct, human-made translation. But what if you're translating something where a 'correct' translation doesn't exist – like whale language? How do you know if the AI isn't just making things up, or 'hallucinating' fluent but false translations? It's hard to validate the translation without being able to ask a whale if it makes sense.
What's the solution?
The researchers suggest a way to test the AI by looking *only* at the English it produces. They translate the animal communication bit by bit, and then scramble the order of those translated bits. If the original order makes more sense than the scrambled order, that suggests the AI is capturing some real structure in the animal language. They tested this idea on human languages with limited translation data and even made-up languages to prove it works, and it matched up well with traditional translation checks when those were available.
Why it matters?
This is important because it could make developing AI translators for animals safer, more ethical, and cheaper. You wouldn't need expensive and potentially disruptive interactions with the animals to check if the AI is learning correctly. It also suggests that, at least initially, we might not *need* to fully understand an animal language to start evaluating if a translation is on the right track.
Abstract
If you had an AI Whale-to-English translator, how could you validate whether or not it is working? Does one need to interact with the animals or rely on grounded observations such as temperature? We provide theoretical and proof-of-concept experimental evidence suggesting that interaction and even observations may not be necessary for sufficiently complex languages. One may be able to evaluate translators solely by their English outputs, offering potential advantages in terms of safety, ethics, and cost. This is an instance of machine translation quality evaluation (MTQE) without any reference translations available. A key challenge is identifying ``hallucinations,'' false translations which may appear fluent and plausible. We propose using segment-by-segment translation together with the classic NLP shuffle test to evaluate translators. The idea is to translate animal communication, turn by turn, and evaluate how often the resulting translations make more sense in order than permuted. Proof-of-concept experiments on data-scarce human languages and constructed languages demonstrate the potential utility of this evaluation methodology. These human-language experiments serve solely to validate our reference-free metric under data scarcity. It is found to correlate highly with a standard evaluation based on reference translations, which are available in our experiments. We also perform a theoretical analysis suggesting that interaction may not be necessary nor efficient in the early stages of learning to translate.