ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding· ParanoiaEval: 评估编程代理中的过度防御
As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing work evaluates related agent behaviors from separate perspectives, but lacks a systematic framework for unifying these behaviors. To bridge this gap, we introduce ParanoiaEval, the first benchmark for unified evaluation of risk-treatment capabilities in coding agents. Grounded in the well-established Avoidance-Transfer-Mitigation-Acceptance framework in software engineering risk management, ParanoiaEval operationalizes its 4 fundamental treatments for coding-agent settings and contains 200 evidence-controlled repository-level task pairs, each differing only in treatment-defining evidence. We further introduce dedicated metrics for risk-tre
提出首个评估编程代理风险处理能力的基准框架ParanoiaEval。
- 核心方法
- 基于软件工程风险管理中的避免、转移、减轻和接受框架，ParanoiaEval为编程代理设置了4种基本风险处理方式，并包含200个差异仅在处理证据的任务对。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 随着编程代理自主完成现实世界工作的能力增强，判断其采取的风险措施是否必要变得重要，但缺乏统一评估框架。
- 关键实验
- 包含200个差异仅在处理证据的仓库级任务对，使用专用指标评估风险处理行为。
- 主要贡献
- 引入了ParanoiaEval，填补了评估编程代理风险处理能力的统一框架空白。
- 意义与局限
- 为编程代理的风险管理提供了系统性评估工具，有助于优化代理行为，提高代码质量与安全性。局限在于应用场景的广泛性和指标的全面性有待进一步验证。