The Review Lottery: Calibrating an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)· 同行评审噪音评估
How much of a conference accept/reject decision would change if the same paper were reviewed by a different set of reviewers? Running a second independent program committee is the gold standard for answering this, but it is prohibitively expensive: done only twice (NeurIPS 2014 and 2021). We build an observational estimator of this quantity from public review data alone, calibrate it twice, and apply it to nine years of ICLR (2017-2025; 36,113 papers, 134,912 reviews). The estimator decomposes scores with a Bayesian ordered-probit model into paper quality and reviewer noise, maps scores to decisions with a logistic model, and simulates two independent committees (posterior draws B=1,000; committee sizes k=2,3,4). Estimated disagreement rates are 23-30% at k=2 and 18-24% at k=4; 30-50% of a
用公共评审数据评估同行评审的不确定性
- 核心方法
- 构建一个基于贝叶斯有序概率模型的观察性估算器，将评分分解为论文质量和评审者噪音两部分，并通过逻辑回归模型映射评分到接收决定，模拟两个独立的评审委员会。
- 适合谁读
- 研究者、会议组织者、学术期刊编辑
- 要解决的问题
- 同行评审过程中不同评审者的评价可能导致论文接收决定的变化，这被称为评审噪音。本论文旨在量化这种评审噪音。
- 关键实验
- 对ICLR 2017-2025年（36,113篇论文，134,912条评论）的数据进行了分析，估算不同评审委员会人数下的评审不一致率。
- 主要贡献
- 提供了一种仅使用公共评审数据评估同行评审噪音的方法，并应用该方法对ICLR 2017-2025年的论文评审进行了量化分析。
- 意义与局限
- 揭示了同行评审中的不确定性，有助于改进评审流程和减少随机性对论文评审结果的影响；但可能受数据和模型假设的限制。