Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance
Xueqing Peng, Triantafillos Papadopoulos, Efstathia Soufleri, Polydoros Giannouris, Ruoyu Xiang, Yan Wang, Lingfei Qian, Jimin Huang, Qianqian Xie, Sophia Ananiadou
2025-02-27
Summary
This paper talks about Plutus, a new AI system designed to understand and process Greek financial language, which includes a benchmark called Plutus-ben for testing AI models and a specialized AI model called Plutus-8B trained on Greek financial texts
What's the problem?
Even though Greece plays an important role in the global economy, there weren't any good AI tools that could understand Greek financial language. This is because Greek is a complex language and there wasn't enough financial data in Greek for AI to learn from. This made it hard for AI to work well with Greek financial information
What's the solution?
The researchers created Plutus-ben, a set of tests to see how well AI can handle Greek financial language tasks like finding important information in text, answering questions, and summarizing financial documents. They also made Plutus-8B, an AI model trained specifically on Greek financial texts. To help with this, they put together new collections of Greek financial data and had experts check it for accuracy
Why it matters?
This matters because it helps make AI more useful for Greek finance, which could improve how financial information is processed and understood in Greece. It also sets an example for creating similar tools in other languages, making AI more inclusive and helpful for finance around the world. By sharing their work openly, the researchers are encouraging others to build on what they've done and make even better tools for understanding financial language in different cultures
Abstract
Despite Greece's pivotal role in the global economy, large language models (LLMs) remain underexplored for Greek financial context due to the linguistic complexity of Greek and the scarcity of domain-specific datasets. Previous efforts in multilingual financial natural language processing (NLP) have exposed considerable performance disparities, yet no dedicated Greek financial benchmarks or Greek-specific financial LLMs have been developed until now. To bridge this gap, we introduce Plutus-ben, the first Greek Financial Evaluation Benchmark, and Plutus-8B, the pioneering Greek Financial LLM, fine-tuned with Greek domain-specific data. Plutus-ben addresses five core financial NLP tasks in Greek: numeric and textual named entity recognition, question answering, abstractive summarization, and topic classification, thereby facilitating systematic and reproducible LLM assessments. To underpin these tasks, we present three novel, high-quality Greek financial datasets, thoroughly annotated by expert native Greek speakers, augmented by two existing resources. Our comprehensive evaluation of 22 LLMs on Plutus-ben reveals that Greek financial NLP remains challenging due to linguistic complexity, domain-specific terminology, and financial reasoning gaps. These findings underscore the limitations of cross-lingual transfer, the necessity for financial expertise in Greek-trained models, and the challenges of adapting financial LLMs to Greek text. We release Plutus-ben, Plutus-8B, and all associated datasets publicly to promote reproducible research and advance Greek financial NLP, fostering broader multilingual inclusivity in finance.