Candidate supply and answer selection shape the value of LLM judging in multi-agent systems· 多代理系统中的 LLM 裁判价值
Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is difficult because generation, communication and final answer-selection rules usually change simultaneously. We conceptualize multi-agent reasoning as an evolutionary pipeline of candidate generation, peer communication and terminal selection, wherein consensus without quality control can exhibit patterns of memetic drift. We study two questions: (1) when an LLM judge provides effective selection pressure by supplying a signal of answer correctness for candidates generated in a multi-agent system, and (2) when using that signal improves the reported answer. To map judge reliability, we analysed 15,336 questions from MMLU-Pro, GPQA, MedXpertQA and Mu
研究 LLM 裁判在多代理系统中如何通过候选人生成和答案选择提升最终答案质量。
- 核心方法
- 将多代理推理过程分为候选人生成、同伴交流和最终选择三个阶段，引入 LLM 裁判来评估候选答案的正确性，并研究其对最终答案的影响。
- 适合谁读
- 研究者
- 要解决的问题
- 在多代理系统中，尽管存在正确答案的潜力，最终报告的可能是错误的答案。
- 关键实验
- 分析了来自 MMLU-Pro、GPQA、MedXpertQA 和 Multi-choice 数据集的15,336个问题。
- 主要贡献
- 分析了15,336个问题，探索了LLM裁判在多代理系统中的有效性和使用时机，提出了 memetic drift 的现象。
- 意义与局限
- 为提高多代理系统中答案选择的质量提供了理论依据，但没有深入探讨 LLM 裁判的具体实现方法。