IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
Sankalp KJ, Ashutosh Kumar, Laxmaan Balaji, Nikunj Kotecha, Vinija Jain, Aman Chadha, Sreyoshi Bhaduri
2025-01-29
Summary
This paper talks about IndicMMLU-Pro, a new way to test how well artificial intelligence (AI) understands and works with languages from India. It's like creating a really tough exam for AI that covers many Indian languages and tests different language skills.
What's the problem?
AI has gotten really good at understanding and using English, but it's not as good with Indian languages. This is a big deal because over 1.5 billion people speak these languages. Indian languages are tricky for AI because they're very different from each other and have unique features that don't exist in English. It's hard to know if AI is actually good at understanding these languages without a proper test.
What's the solution?
The researchers created IndicMMLU-Pro, which is like a super comprehensive test for AI in Indian languages. This test covers major languages like Hindi, Bengali, and Tamil, and checks how well AI can understand, reason with, and create content in these languages. They carefully designed the test to include the special features of Indian languages and to cover a wide range of tasks. They also used this test to see how well current AI models perform with these languages.
Why it matters?
This matters because as AI becomes more important in our daily lives, we need to make sure it works well for everyone, not just English speakers. By creating this test, researchers can now measure how good AI is at understanding Indian languages and find ways to make it better. This could lead to AI that's more helpful for people in India, whether they're using it for education, work, or just everyday tasks. It also helps preserve and promote the rich diversity of Indian languages in the digital world.
Abstract
Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models.