A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO· 课程难度递进在 GRPO 中的益处和限制
Curriculum learning typically organizes training samples from easy to hard. In GRPO, medium-difficulty samples are attractive for online selection because they are more likely to yield informative group-relative learning signals, and prioritizing them can improve training efficiency. However, it remains unclear whether persistently selecting such samples also leads to the strongest final model performance, or whether training can benefit from progressing toward harder samples. Meanwhile, sample difficulty changes as the policy evolves, so fixed difficulty estimates can become misaligned with the current policy. We therefore re-estimate sample difficulty under the current policy and, inspired by the easy-to-hard principle of curriculum learning, design and evaluate a staged curriculum that
课程难度递进对 GRPO 训练的影响与优化。
- 核心方法
- 重新估计样本难度以匹配当前策略，并设计分阶段的课程学习方法，以实现难度从易到难的渐进。
- 适合谁读
- 研究者
- 要解决的问题
- 探讨课程学习中持续选择中等难度样本与向更高难度样本过渡对 GRPO 训练效率及最终模型性能的影响。
- 关键实验
- 通过实验验证了分阶段课程学习策略在不同任务上的有效性和局限性。
- 主要贡献
- 提出了一种动态调整样本难度的分阶段课程学习策略，改善了 GRPO 的训练效果。
- 意义与局限
- 研究揭示了课程学习在 GRPO 中的优化路径，为未来研究提供了新思路，但动态难度调整可能存在计算成本。