ai.hackcv
论文精选 60arXiv

Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision· 时间自蒸馏:无监督视频视觉状态跟踪

We introduce S$^3$T (Self-Supervised Self-Distillation over Time), which, to the best of our knowledge, is the first fully self-contained framework for continuous video state tracking. Our method treats temporal sampling density as privileged information, based on the hypothesis that a denser view of the same clip recovers the running state more accurately. This view serves as the teacher, while a sparse-view student with the same weights learns to match its next-token distribution. The model generates its own target, so training requires no labels, separate teacher, or reward signal, and adds no inference cost. On LLaVA-OneVision-2-8B, S$^3$T improves VSTAT accuracy by $+1.74$ as a single model, $+2.38$ with souping, and $+2.70$ with additional vision-encoder adaptation, while prior self-evolving methods leave state tracking largely unchanged. The capability learned from unlabeled synthetic clips transfers to real videos, improving performance by $+7.95$ on VSTAT-YouTube state-tracking questions and $+4.50$ on MVBench Action Count.

AI 解读论文

提出S$^3$T,利用时间自蒸馏无监督学习视频视觉状态跟踪。

核心方法
S$^3$T方法通过将时间采样密度视为特权信息,让密集采样视角作为教师模型,稀疏采样视角作为学生模型,学生模型学习匹配教师模型的下一个token分布,整个过程自监督且无需额外的教师模型或奖励信号。
适合谁读
研究者、工程师
要解决的问题
这篇论文旨在解决视频中的视觉状态跟踪问题,特别是在没有监督标签的情况下如何有效学习。
关键实验
实验结果表明,S$^3$T在LLaVA-OneVision-2-8B模型上单模型提升了1.74%的VSTAT准确性,通过增强和视觉编码器适应分别提升了2.38%和2.70%;在VSTAT-YouTube数据集上提升了7.95%,在MVBench Action Count上提升了4.50%。
主要贡献
S$^3$T是首个完全自包含的视频状态跟踪框架,通过无监督学习显著提高了视觉状态跟踪的准确性,并且该方法具有较低的推理成本。
意义与局限
该方法为视频处理任务提供了一种新的无监督学习思路,能够有效提升视觉状态跟踪的效果,特别适用于大规模视频数据的处理,但可能对不同类型视频的泛化能力有限。
领域:cs.CV作者:Shravan Venkatraman、Wenshuai Zhao、Mohammad Hassan Vali
相关推荐
论文精选 60已解读

Principia: Relational Physics Tests for Video Models· Principia:视频模型的物理测试

Evaluating physical reasoning in video models is difficult because absolute motion measure…

领域:cs.CV作者:Varun Varma Thozhiyoor、Shivam Tripathi、Venkatesh Babu Radhakrishnan
📎 arXiv🕒 09-04 01:59🔗 arxiv.org

本站内容由 LLM 精选聚合,原文版权归 arXiv 所有 · 摘录仅供参考