Mismatch Matters: On-Policy Distillation Beyond Token Agreement· 正策略蒸馏问题探讨
On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses. We therefore shift our focus from agreement to teacher-student mismatch, and find that mismatch tokens can be mainly categorized into two types: student-excess tokens and student-deficit tokens. Student-excess tokens are generated by the student but assigned near-zero probability by the teacher; their log-ratio corrections grow unbounded and destabilize the update. Student-deficit tokens, in contrast, are preferred by the teacher but rarely sampled by the student; their absence blocks the transfer of the teache
探讨正策略蒸馏中的故障模式，提出减少教师与学生模型不匹配的新方法。
- 核心方法
- 作者将注意力从令牌一致性转向教师学生模型之间的不匹配，识别出两种主要的不匹配类型：学生过分令牌和学生缺失令牌，并提出通过修正这些令牌的对数比率来稳定模型更新。
- 适合谁读
- 适合研究者和工程师阅读
- 要解决的问题
- 论文旨在解决正策略蒸馏过程中学生模型与教师模型之间的不匹配问题，具体表现为尽管学生模型在单个令牌上几乎完美地与教师模型达成一致，但在整体响应中仍存在缺陷。
- 关键实验
- 实验表明，使用提出的修正方法可以有效减少模型训练过程中的不稳定性和提高生成文本的质量，但是具体实验数据和设置详情未提供。
- 主要贡献
- 提出对数比率修正方法来处理正策略蒸馏中的不匹配问题，有助于提高学生模型的整体性能和稳定性。
- 意义与局限
- 该研究有助于改进大型语言模型的后训练流程，尤其是在正策略蒸馏方面，但可能需要进一步实验验证方法的有效性和泛化性。