Language-model groups overstate consensus when replaying human deliberation on a reasoning task· 大模型群体在推理任务中高估共识
Full-consensus rates are often treated as indicators of collective cognition, yet depend on how participation and final states are operationalized. We replayed 100 held-out human Wason groups with matched large language model (LLM) agent groups, seeding one belief-anchored agent per participant's pre-discussion answer and scoring agents and people with the same code. Across human scoring definitions, estimates ranged from 24.0% to 57.0%; about one fifth of participants never posted, whereas agents almost always did. Agent groups remained more consensual in two post-unblinding sensitivity analyses: the submit-based comparison (n = 98) yielded gaps of 34.0 and 43.9 percentage points for chat and reasoning modes, and the participation-matched comparison (n = 45) yielded gaps of 34.1 and 44.4
研究大模型群体在推理任务中的共识高估问题。
- 核心方法
- 通过重演100个未公开的人类Wason小组讨论，将每个参与者预讨论的答案作为起始信念，使用相同的评分代码来评估人类和大语言模型（LLM）代理的共识率。
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 探讨大模型群体在模拟人类讨论过程中是否高估了共识程度。
- 关键实验
- 进行了100个小组的重演实验，包括提交答案的比较和参与度匹配的比较，分别在聊天和推理模式下评估共识率的差距。
- 主要贡献
- 揭示了大模型群体在推理任务中高估共识的倾向，并提供了与人类共识率的对比数据。
- 意义与局限
- 该研究有助于理解大模型在集体认知任务中的表现，指出其在模拟人类讨论时的局限性，为模型设计和应用提供参考。