Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocation· 学习何时思考:测试时计算资源的自适应分配
Reasoning language models trained with reinforcement learning typically operate under a fixed token budget rather than an explicitly adaptive one, which can lead to over-computation on easy problems and insufficient computation on difficult ones. We study whether a model can learn to allocate its own reasoning effort by choosing, as the first token of its response, one of three modes: \textsc{NoThink} (answer as quickly as possible), \textsc{Short} (brief reasoning), or \textsc{Long} (extended reasoning). The choice is learned inside Group Relative Policy Optimization (GRPO) with no separate router, through a shaped reward that makes each mode worthwhile at a different response length, together with hard per-mode token caps that keep the modes distinct. On a 1.5B distilled model trained on
自适应分配模型推理努力,提高效率。
- 核心方法
- 通过强化学习训练模型,在响应时选择三种模式之一(不思考、简短推理或长篇推理),并使用组相对策略优化(GRPO)和成形奖励来学习这种选择。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 现有的语言模型在推理过程中通常采用固定的计算资源分配,导致简单问题上过度计算和复杂问题上计算不足。
- 关键实验
- 实验数据未完全提供,论文描述了1.5B参数量的蒸馏模型训练过程。
- 主要贡献
- 提出了一种新的自适应计算分配方法,使模型能够根据问题难度自动调整推理模式。
- 意义与局限
- 意义在于提高模型推理效率和适应性,但在复杂度较高和奖励机制设计上的局限性可能影响模型的普遍适用性。