Eka-Eval : A Comprehensive Evaluation Framework for Large Language Models in Indian Languages

Summary

This paper talks about Eka-Eval, a new evaluation framework designed to test and measure the performance of large language models, especially focusing on Indian languages alongside global ones. It offers a wide range of benchmarks covering different skills like reasoning, math, and reading comprehension.

What's the problem?

The problem is that most existing evaluation tools focus mainly on English and don’t cover many Indian languages, which makes it hard to test how well language models perform for the diverse languages spoken in India. There is also a need for a unified system that can handle many tasks efficiently.

What's the solution?

The researchers created Eka-Eval as a modular, easy-to-use system that includes over 35 different test datasets, including some dedicated to Indian languages. It supports running evaluations across multiple devices and models, handles large context tasks, and provides detailed reports on model performance across different areas and languages.

Why it matters?

This matters because it allows AI developers and researchers to fairly test and improve language models for a much wider range of languages, making AI tools more accessible and useful to people in India and other multilingual regions.