TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation

Summary

This paper introduces TOMG-Bench, a new benchmark designed to evaluate how well large language models (LLMs) can generate molecules based on text descriptions. It includes various tasks related to molecule editing, optimization, and custom generation.

What's the problem?

Generating molecules from text is complex and requires precise understanding. Existing methods often rely on specific datasets that limit the models' ability to learn and adapt to new tasks, making it hard to evaluate their true capabilities in open-domain scenarios.

What's the solution?

TOMG-Bench consists of three main tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and customized molecule generation (MolCustom). Each task has several subtasks with thousands of test samples. The benchmark also features an automated evaluation system to measure the quality and accuracy of the generated molecules. Additionally, a specialized dataset called OpenMolIns was created to help improve the models' performance on these tasks.

Why it matters?

This research is significant because it provides a structured way to assess the abilities of LLMs in generating molecules from text. By identifying strengths and weaknesses in current models, TOMG-Bench can guide future improvements in AI-assisted chemistry, potentially leading to advancements in drug discovery and material science.