SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents· SWE-Gate: 代码代理仅通过功能测试不够
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.
功能测试不足以评估代码代理的全面能力。
- 核心方法
- 提出了 SWE-Gate 基准，从真实的 pull request 评论中提取审查约束，并合成仓库级别的修复实例，每个实例提供功能测试和约束测试。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 现有的代码代理评估基准主要关注功能测试，忽略了代码审查中的接受约束。
- 关键实验
- 使用四个不同能力级别的 LLM 后端进行实验，揭示了在 644 个通过功能测试的修复中，有 221 个未能满足提供的审查约束。
- 主要贡献
- 构建了包含 303 个仓库级别修复实例的 SWE-Gate 基准，展示了功能成功与完全修复规范下的成功之间存在显著差距。
- 意义与局限
- 强调了功能测试的局限性，指出全面评估代码代理能力需要考虑审查约束，对改进代码代理的评估方法有重要影响。