FlowWAM: Optical Flow as a Unified Action Representation for World Action Models· FlowWAM:光流作为统一动作表示的世界模型
World Action Models (WAMs) are able to leverage pretrained video generators for both world modeling and action prediction. However, directly leveraging such video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream diffusion framework that adopts optical flow as a unified, video-native action representation. Flow videos share the same format as RGB videos and encode rich per-pixel displacement. By jointly modeling them within a shared pretrained video generator, FlowWAM can naturally implement two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future video generation. Moreover, since flow can be easily extracted from raw videos without action labels, FlowWAM can leverage large-scale action-unlabeled video datasets for pretraining. We empirically find that our flow-based action representation delivers gains across both modes. On RoboTwin manipulation, FlowWAM raises the success rate to 92.94% on the Clean setting and 92.14% on Random, outperforming both VLA and WAM baselines. On WorldArena world modeling, it achieves the best overall EWMScore (63.71) with an 18.4% relative improvement in trajectory accuracy. More results can be found on our project website: https://flow-wam.github.io .
使用光流作为统一动作表示提升世界模型性能。
- 核心方法
- 提出FlowWAM,一种双流扩散框架,采用光流作为统一的视频原生动作表示方法,通过共享预训练视频生成器联合建模RGB和光流视频。
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 直接利用预训练视频生成器进行控制时,如何找到既能与视频生成器兼容又包含足够运动信息的动作表示。
- 关键实验
- 在RoboTwin任务中,FlowWAM在Clean设置下的成功率为92.94%,在Random设置下为92.14%;在WorldArena任务中,总体EWMScore达到63.71,轨迹准确性相对提高了18.4%。
- 主要贡献
- FlowWAM在世界模型和策略模式下均优于现有方法,特别是在RoboTwin操作任务和WorldArena世界建模任务中表现出色。
- 意义与局限
- 该方法为利用大规模无标签视频数据进行动作表示学习提供了新的途径,但在不同任务中的泛化能力有待进一步验证。