PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents· PhoneCLI: 从应用界面到移动代理的可调命令
Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it determini
将应用导航编译为可调用命令，提高移动代理效率。
- 核心方法
- PhoneCLI 通过离线探索应用的GUI，将屏幕、交互元素和导航路径编译成可调用的命令，每个屏幕对应一个确定的命令序列。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 现有的移动GUI代理在操作应用时速度慢、成本高且易出错，主要因为重复的导航操作。
- 关键实验
- 未提供
- 主要贡献
- 无需应用内部API、运行时插桩或模型训练，即可将移动应用导航编译为可调用命令，提高导航效率和可靠性。
- 意义与局限
- 这项技术可以显著提高移动代理的操作效率，减少运行成本，但可能不适用于非常动态或个性化的应用导航。