SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence
Yiheng Wang, Yixin Chen, Shuo Li, Yifan Zhou, Bo Liu, Hengjian Gao, Jiakang Yuan, Jia Bu, Wanghan Xu, Yuhao Zhou, Xiangyu Zhao, Zhiwang Zhou, Fengxiang Wang, Haodong Duan, Songyang Zhang, Jun Yao, Han Deng, Yizhou Wang, Jiabei Xiao, Jiaqi Liu, Encheng Su, Yujie Liu
2026-01-07
Summary
This paper introduces SciEvalKit, a new tool for thoroughly testing how well AI models can do science, covering many different scientific fields and types of scientific tasks.
What's the problem?
Currently, there isn't a good, unified way to test AI models specifically on scientific skills. Existing tests are often too general and don't really measure if an AI can *think* like a scientist – things like understanding scientific images, reasoning through problems, writing code for experiments, or even forming new scientific ideas. It's hard to compare different AI models because they're tested in different ways, and the tests aren't always based on real-world scientific challenges.
What's the solution?
The researchers created SciEvalKit, which is like a collection of standardized science tests for AI. It focuses on key scientific abilities like understanding scientific data (images, graphs, etc.), reasoning about science, generating code for experiments, and understanding scientific knowledge. It includes tests from fields like physics, chemistry, astronomy, and materials science, using real data scientists actually work with. The toolkit is designed to be flexible, so researchers can easily test different AI models and add their own tests too, making sure the results are clear and repeatable.
Why it matters?
SciEvalKit is important because it provides a common standard for evaluating AI in science. This will help researchers track progress in developing AI that can actually assist with scientific discovery. By having a reliable way to measure these skills, we can build better AI tools for scientists and accelerate breakthroughs in various fields.
Abstract
We introduce SciEvalKit, a unified benchmarking toolkit designed to evaluate AI models for science across a broad range of scientific disciplines and task capabilities. Unlike general-purpose evaluation platforms, SciEvalKit focuses on the core competencies of scientific intelligence, including Scientific Multimodal Perception, Scientific Multimodal Reasoning, Scientific Multimodal Understanding, Scientific Symbolic Reasoning, Scientific Code Generation, Science Hypothesis Generation and Scientific Knowledge Understanding. It supports six major scientific domains, spanning from physics and chemistry to astronomy and materials science. SciEvalKit builds a foundation of expert-grade scientific benchmarks, curated from real-world, domain-specific datasets, ensuring that tasks reflect authentic scientific challenges. The toolkit features a flexible, extensible evaluation pipeline that enables batch evaluation across models and datasets, supports custom model and dataset integration, and provides transparent, reproducible, and comparable results. By bridging capability-based evaluation and disciplinary diversity, SciEvalKit offers a standardized yet customizable infrastructure to benchmark the next generation of scientific foundation models and intelligent agents. The toolkit is open-sourced and actively maintained to foster community-driven development and progress in AI4Science.