QuoteBench: How Matched Scores Can Hide Command-Path Failures· QuoteBench: 匹配分数如何掩盖命令路径失败
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recove
探讨命令路径失败对大语言模型生成的Bash命令的影响。
- 核心方法
- 通过QuoteBench工具进行最终状态验证，引入一个有意未转义的解析器，评估生成的命令在不同配置下的成功率。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 大语言模型生成的Bash命令在通过接口时可能会被序列化、包装和重新解析，导致即使执行分数匹配也难以区分命令生成错误与后续处理错误。
- 关键实验
- 在56个单次任务上进行实验，这些任务来自14个真实事故案例家族。
- 主要贡献
- 揭示了命令路径失败对大语言模型执行效果的显著影响，并提供了量化分析的方法。
- 意义与局限
- 为评估和改进大语言模型与实际系统交互的可靠性提供了新视角，但实验环境与真实世界仍有差异。