JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition· JarvisGUI:面向跨设备 GUI 代理的动态任务组合
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which
提出 JarvisGUI,评估跨设备 GUI 代理的动态任务组合能力
- 核心方法
- 通过轻量级类型系统将 GUI 任务定义为输入输出转换,构建跨设备动态任务组合的评估基准
- 适合谁读
- 研究者
- 要解决的问题
- 现有 GUI 代理评估主要集中在单设备静态任务,缺乏对跨设备工作流程的测试
- 关键实验
- 未提供
- 主要贡献
- 引入 JarvisGUI 动态评估基准,支持评估跨 Android、Windows 和 Ubuntu 平台的 GUI 代理
- 意义与局限
- 有助于更准确评估 GUI 代理的真实应用能力,但当前仅提供评估框架,未展示具体实验结果