An Empirical Study of Harness Design for Coding Agents· 自动编码代理的Harness设计实证研究
Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window bud
探究自动编码代理Harness设计中各组件的有效性
- 核心方法
- 通过简化的设计固定执行循环，改变计划、操作空间和上下文管理三个组件，进行176组对比实验
- 适合谁读
- 适合软件工程领域的研究者、开发自动编码工具的工程师以及对自动化软件开发感兴趣的学者
- 要解决的问题
- 当前对自动编码代理Harness的设计评估大多将其视为一个整体，无法明确个体组件的具体效果
- 关键实验
- 在SWE-Bench Verified和Terminal-Bench 2.1数据集上评估了四个模型，实验设置包括五种上下文管理策略、四种上下文窗口预算及计划和操作空间的定向删除实验
- 主要贡献
- 首次在组件级别详细评估了自动编码代理Harness设计的效果，特别是上下文管理的重要性
- 意义与局限
- 研究为优化自动编码代理提供了具体方向，尤其是在上下文管理的改进上，但研究范围有限，未考虑所有可能的设计变体