What 30,000 Hours of Ego-centric Video Does Not Teach· 30000小时的视角视频教不会什么
World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.
30,000小时视角视频对建模世界模型中的代理和物体互动影响的评估。
- 核心方法
- 使用包含1,000多种场景类型和14,000名贡献者的30,000小时视角视频数据集,直接评估了世界模型在代理行为和物体互动保真度上的表现,并引入了一种监督方案来改善物体动态建模。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 世界模型虽然有潜力替代基于物理的模拟器,但其在建模代理和物体互动时的效果仍不理想,尤其是在数据量大幅增加的情况下。
- 关键实验
- 通过一个具有挑战性的分布外基准测试,评估了代理和物体互动的保真度,结果显示数据量增加100倍后,代理建模效果显著提升,但物体保真度提升缓慢。
- 主要贡献
- 发现数据量增加对代理建模有效但对物体保真度提升有限,并提出了一种新的训练方法来专门提高物体动态的建模能力。
- 意义与局限
- 该研究揭示了世界模型在代理与物体互动建模上的不对称性,强调了训练方法的重要性,为未来研究提供了方向。