ResearchTown: Simulator of Human Research Community
Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, Jiaxuan You
2024-12-24
Summary
This paper talks about ResearchTown, a new system designed to simulate how human research communities work using large language models (LLMs). It aims to understand how ideas are generated and how researchers collaborate.
What's the problem?
Understanding how research communities function is complex, especially when it comes to generating new ideas and collaborating effectively. Traditional methods of studying this process can be limited and may not capture the dynamic interactions between researchers and their work.
What's the solution?
To address this, the authors created ResearchTown, which models the research community as a network of agents (researchers) and data (research papers). They introduced a framework called TextGNN that simulates various research activities, like reading and writing papers. To evaluate how well this simulation works, they developed ResearchBench, a benchmark that assesses the quality of the simulation by checking how accurately it can predict outcomes based on the relationships in the network. Their experiments showed that ResearchTown can realistically simulate collaborative research activities and generate new interdisciplinary ideas.
Why it matters?
This research is important because it provides a new way to study and understand research communities using AI. By simulating these interactions, ResearchTown could help improve how researchers collaborate and innovate, ultimately leading to faster discoveries and advancements in various fields.
Abstract
Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified and modeled as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire novel research directions.