Real-Time Detection and Repair of LLM Agent Failures· 大型语言模型代理故障的实时检测与修复
LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at =9), predicting its own failure region
通过低成本监控实时检测和预测LLM代理故障
- 核心方法
- 使用基于单类回声状态网络的集成监控系统结合CUSUM告警,仅通过正常运行数据训练,实现实时故障检测
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 解决大型语言模型代理在任务执行过程中出现的循环、错误累积、偏离目标等故障问题
- 关键实验
- 在2823个代理任务中测试了三种本地模型和一个商用API的代理,展示了方法的有效性
- 主要贡献
- 提出了一种低成本且高效的LLM代理故障检测方法,能在不超过5%误报率的情况下检测71%的故障
- 意义与局限
- 该方法可以显著降低LLM代理监控成本,提高故障检测效率,但目前仅在有限框架和模型上进行了测试