A Privileged Teacher That Only Scores Cannot Search: Critique-Conditioned On-Policy Self-Distillation for Reasoning· 无法搜索的特权教师：基于批评的在线策略自蒸馏
On-policy self-distillation promises to replace the sparse outcome signal of verifiable-reward RL with dense token-level credit: a teacher conditioned on the reference solution scores the student's own rollout, and the resulting distributional shift is distilled back into the policy. We investigate whether this scoring signal actually contains corrective credit. Using gold error spans in student rollouts, we test the two signals consumed by self-distillation objectives, the full-distribution KL and the token log-ratio, for two properties: whether they localize errors and whether they push probability away from erroneous student tokens. They exhibit neither of these properties nor can they reliably distinguish correct from incorrect rollouts within the same problem. We argue that this failu
研究在线策略自蒸馏的有效性，发现其评分信号无法有效纠错。
- 核心方法
- 通过分析学生模型在任务中的错误片段，评估两种自蒸馏目标信号（全分布KL散度和令牌对数比）是否能准确定位错误并减少错误令牌的概率。
- 适合谁读
- 研究者
- 要解决的问题
- 论文探讨了在线策略自蒸馏方法中，评分信号是否能够提供有效的纠错信息。
- 关键实验
- 使用金标准错误片段测试了不同目标信号在错误定位和概率调整方面的性能，结果表明这些信号无法有效区分正确与错误的输出。
- 主要贡献
- 揭示了当前在线策略自蒸馏方法的局限性，提出评估纠错能力的新视角。
- 意义与局限
- 指出在线策略自蒸馏方法的潜在问题，对改进模型训练和提高RL任务性能具有重要启示；局限在于仅测试了特定任务和信号。