ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research· ScholarCatalyst:开启新研究的论文检索基准
What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prior idea, buried in an ever-growing archive of research, a new problem needs. To study this skill, we draw on researchers who know firsthand which earlier work advanced their completed projects, with papers serving as pointers to the ideas within. Using our automated pipeline that makes author annotation scalable, we build ScholarCatalyst by having 184 lead authors of 207 recent computer science papers label which candidates did or could have advanced their project, each with a detailed rationale. We introduce a retrieval task with author-provided judgments: given an initial research question, retrieve these papers from only the literature av
通过标注论文来研究科学家如何发现促进新研究的关键文献
- 核心方法
- 构建自动化标注流程,让184位作者标注207篇计算机科学论文,生成ScholarCatalyst数据集,并引入基于这些标注的检索任务
- 适合谁读
- 研究者、AI工程师
- 要解决的问题
- 如何从庞大的文献库中找到能够启发新研究的关键论文
- 关键实验
- 评估了不同检索模型在ScholarCatalyst数据集上的性能
- 主要贡献
- 创建了ScholarCatalyst数据集,提供了科学家如何选择和利用已有文献的见解
- 意义与局限
- 有助于理解科学家的文献选择策略,提升AI系统的文献检索能力,但目前数据集规模有限