FDE-Gym: The Last Mile of AI Delivery· FDE-Gym：AI交付的最后一公里
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged
FDE-Gym 仿真客户环境，评测AI交付实际效果。
- 核心方法
- 设计了包含来自20个客户场景、涵盖18个领域的95个端到端交付任务的FDE-Gym基准测试平台，通过模拟客户环境和利益相关者交互，以及操作生产模拟系统来评估AI代理的交付能力。
- 适合谁读
- 研究者、工程师、产品
- 要解决的问题
- 现有AI代理基准测试无法反映真实世界中前部署工程师(FDEs)面临的模糊需求、现场系统操作和未言明的约束问题。
- 关键实验
- 采用模拟客户环境下的95个任务进行了评估，结果显示FDE-Gym能够有效评测AI代理的实际交付效果。
- 主要贡献
- 提供了首个全面评估AI交付任务的基准测试，能够更好地反映FDEs在现实工作中遇到的挑战。
- 意义与局限
- 为AI系统的实际应用提供了重要工具，有助于提高真实世界交付效率和质量；但也存在对特定行业细节处理的局限。