Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry· 提升状态策略优化：通过转换几何内部化自校正
Preference optimization ranks final answers and trajectory RL maximizes aggregate returns, yet neither credits the local change that turns a flawed reasoning state into a better one. We introduce Lifted State Policy Optimization (LSPO), which establishes this finite, cost-adjusted transition as the primary unit of learning. Each textual answer is augmented with a learned continuous coordinate, forming a lifted state that an energy function evaluates and orders. A shaped reward drives energy descent net of edit and step costs, and an internalization loss absorbs selected low-energy answers into the direct generator. On Qwen3.5-27B, the resulting single-pass policy reaches 71.8% aggregate accuracy across mathematics, science, code, and logic benchmarks, matching test-time reflection at 71.9%
通过转换几何方法提升策略优化效果。
- 核心方法
- 引入提升状态策略优化（LSPO），将有限的成本调整转换作为主要学习单元，通过能量函数评估和排序提升状态。
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 解决偏好优化和轨迹强化学习中忽略局部转换改善的问题。
- 关键实验
- 在Qwen3.5-27B模型上，LSPO在数学、科学、代码和逻辑基准测试中达到71.8%的总体准确率。
- 主要贡献
- 提出了一种新的策略优化方法，能够在单次传递中达到高准确率，与测试时间反思效果相当。
- 意义与局限
- 意义在于通过内部化自校正机制提高策略优化效果，但局限性可能在于计算成本和适用范围。