WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models· WorldCycle:自验证强化学习框架
Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spa
自验证的强化学习框架,用于改善视频世界模型的长期预测准确性。
- 核心方法
- 通过构造可逆动作循环,将任意动作序列与其逆动作序列结合,确保最终能返回初始状态,提供无需注释的长期正确性监督;优化环境奖励和历史相似性奖励两种互补奖励。
- 适合谁读
- 适合强化学习研究者和视频处理领域的工程师阅读
- 要解决的问题
- 视频世界模型在长期规划和探索中存在累积误差,且缺乏有效的方法来验证这些模型的长期预测准确性。
- 关键实验
- 展示了 WorldCycle 在多个长期任务上的性能提升,包括复杂环境下的导航和目标达成任务;未提供具体实验数据。
- 主要贡献
- 提出了一种新颖的自验证强化学习框架 WorldCycle,能够有效减少视频世界模型的长期预测误差。
- 意义与局限
- 意义在于提高视频世界模型在复杂任务中的可靠性和准确性,但可能受到特定环境可逆性限制,不一定适用于所有场景。