JEV-as-a-Judge: Accept When Confident, Escalate When Unsure· JEV-as-a-Judge: 有信心时接受，不确定时升级
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident
研究决策型LLM在低成本下进行高效评估的能力。
- 核心方法
- 提出JEV-as-a-Judge模型，专注于决策而非生成，用于初步评估并识别需更深入评估的情况。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 大型语言模型作为评估工具时，推理成本高且置信度可靠性有待提升。
- 关键实验
- 与16种生成型和奖励模型的法官进行对比，通过盲评人类裁判验证，JEV在普通偏好和基于证据的事实性上表现接近最强LLM法官（差异在3个百分点内），成本仅为0.36%。
- 主要贡献
- JEV模型以极低的成本达到了与最先进LLM相近的评估准确性，尤其在高置信度决策中表现突出。
- 意义与局限
- 该研究为大规模应用LLM评估提供了经济高效的解决方案，但仍存在局限性，如在复杂推导和抵抗精心编写的错误答案方面。适合研究者和工程师阅读。