IPBench: Benchmarking the Knowledge of Large Language Models in Intellectual Property
Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Linwei Li, Yilin Yue, Shiqiang Wang, Jiayan Li, Yihang Wu, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hongfei Lin, Hamid Alinejad-Rokny, Shiwen Ni, Yuan Lin
2025-04-23
Summary
This paper talks about IPBench, a new test that checks how well large language models (LLMs) understand and handle questions about intellectual property, like patents and copyrights, in both English and another language.
What's the problem?
The problem is that while LLMs are getting better at answering general questions, it's unclear how good they are at dealing with complicated legal topics such as intellectual property, especially when these questions come from different countries or are written in different languages.
What's the solution?
To find out, the researchers created IPBench, which includes a wide variety of real-world questions about intellectual property in more than one language. They used this benchmark to test different LLMs and see where they do well and where they struggle.
Why it matters?
This matters because intellectual property laws are important for things like inventions, music, and art, and many people rely on AI for help with legal information. By showing where LLMs fall short, IPBench helps researchers improve these models so they can give better, more reliable answers in the future.
Abstract
A large, diverse bilingual benchmark, IPBench, is introduced to evaluate LLMs in real-world intellectual property applications, revealing substantial room for improvement.