StebBench: Diagnosing Real-World Procedural Understanding in AI Agents· StebBench: 诊断AI代理的实际过程理解能力
Evaluating an agent's planning ability remains challenging for open-domain tasks, where verifying procedures through execution is often impractical. Existing evaluations typically assess only the final plan and do not distinguish procedural errors such as missing, incorrect, or misordered actions. We introduce STEPBENCH, a benchmark of 6,885 verified procedures crawled from wikiHow across 81 open-domain categories. From each procedure, we derive 5 diagnostic tasks: four with deterministic answers involving withheld, incorrect, or shuffled steps, and one requiring next-step generation. We additionally evaluate full-plan generation to examine how diagnostic performance correlates with end-to-end plan validity. Using the same procedures across tasks lets us directly compare different planning
诊断AI代理在实际场景中的过程理解能力的新基准
- 核心方法
- 从wikiHow获取6,885个验证过程，跨81个开放域类别，设计5个诊断任务测试AI能力
- 适合谁读
- 研究者, 工程师
- 要解决的问题
- 开放域任务中，评估代理规划能力时难以验证过程的具体步骤及其顺序正确性
- 关键实验
- 实验包括缺失步骤、错误步骤、顺序错乱步骤的识别任务及下一步预测和完整计划生成任务
- 主要贡献
- 提出STEPBENCH基准，能够更全面地评估AI代理的过程理解能力
- 意义与局限
- 为AI代理的规划能力评估提供了一个更细致和实用的工具，但目前仅基于文本，缺乏实际执行的验证