CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents· CUA-Universe: GUI+CLI 混合代理的可扩展动态环境
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We intr
提出 CUA-Universe, 用于 GUI+CLI 混合代理的可扩展动态环境
- 核心方法
- CUA-Universe 提供了一个可在真实应用程序上同时支持 GUI 和 CLI 的环境,减少了手动工程需求,使代理能高效利用两种模式
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 现有计算机使用代理在 GUI 和 CLI 模式之间缺乏有效协调,导致任务执行效率低下
- 关键实验
- 未提供
- 主要贡献
- 解决了 GUI 和 CLI 代理的有效结合问题,提供了一个可扩展的混合环境,提高了任务执行效率
- 意义与局限
- 该环境有助于研发更加高效的计算机使用代理,但可能需要针对不同应用进一步优化和调整