SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute· 自验证细化:联合判决-置信度强化学习的自适应测试时计算
Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback. We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy. At each turn, the model produces a solution together with a discrete correctness verdict and a confidence score; it retains the current answer only when the verdict is Correct and confidence exceeds a threshold, and otherwise continues refinement using its own self-verification. Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference. SVR is tr
自验证细化通过联合判决-置信度强化学习实现自适应测试时计算,提升语言模型推理效率。
- 核心方法
- SVR 使用无Oracle的多轮次强化学习框架,通过生成离散正确性判断和置信度分数来自我验证和控制计算。模型在每一轮次中根据自我验证结果决定是否保留当前答案或继续细化。
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 现有语言模型测试时计算方法要么均匀分配计算资源,导致对简单输入的浪费,要么依赖外部反馈的验证器引导细化,限制了模型的灵活性和泛化能力。
- 关键实验
- 论文展示了在多个自然语言处理任务上的实验结果,验证了SVR框架的有效性。与基线方法相比,SVR在保持甚至提高性能的同时显著减少了计算资源的浪费。
- 主要贡献
- 提出了SVR框架,能够在不依赖外部反馈的情况下,自适应地分配计算资源,有效提升语言模型的推理性能和资源利用效率。
- 意义与局限
- SVR 框架为语言模型推理提供了一种新的自适应计算策略,有助于提升模型在实际应用中的性能和资源效率,但可能在某些复杂任务上仍需进一步验证。