SwiLTra-Bench: The Swiss Legal Translation Benchmark
Joel Niklaus, Jakob Merane, Luka Nenadic, Sina Ahmadi, Yingqiang Gao, Cyrill A. H. Chevalley, Claude Humbel, Christophe Gösken, Lorenzo Tanzi, Thomas Lüthi, Stefan Palombo, Spencer Poff, Boling Yang, Nan Wu, Matthew Guillod, Robin Mamié, Daniel Brunner, Julio Pereyra, Niko Grupen
2025-03-06
Summary
This paper talks about SwiLTra-Bench, a new tool created to test and improve how well AI can translate legal documents in Switzerland, where multiple official languages are used
What's the problem?
Switzerland needs legal documents in four different languages, but relying on human experts who are both lawyers and skilled translators creates a bottleneck. This slows down the process and makes it harder for people to access legal information quickly
What's the solution?
The researchers created SwiLTra-Bench, a large collection of over 180,000 pairs of translated legal documents in Swiss languages and English. They used this to test different AI translation systems, including advanced models and specialized legal translators. They also developed SwiLTra-Judge, an AI system to evaluate translations in a way that matches human expert opinions
Why it matters?
This matters because it could help make legal information more accessible to everyone in Switzerland, regardless of what language they speak. By improving AI translation for legal documents, it could speed up the legal process, reduce costs, and ensure that everyone has equal access to important legal information. This could lead to a fairer and more efficient legal system in multilingual countries like Switzerland
Abstract
In Switzerland legal translation is uniquely important due to the country's four official languages and requirements for multilingual legal documentation. However, this process traditionally relies on professionals who must be both legal experts and skilled translators -- creating bottlenecks and impacting effective access to justice. To address this challenge, we introduce SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. Our systematic evaluation reveals that frontier models achieve superior translation performance across all document types, while specialized translation systems excel specifically in laws but under-perform in headnotes. Through rigorous testing and human expert validation, we demonstrate that while fine-tuning open SLMs significantly improves their translation quality, they still lag behind the best zero-shot prompted frontier models such as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM evaluation system that aligns best with human expert assessments.