Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation· 确保闭合环 AI 评估安全的协议
An AI evaluation can be perfectly reproducible and still support the wrong claim. This risk is acute in closed-loop systems: policy determines visited states, observable components, and which failures leave a measurable trace. We propose a claim-safe protocol with three actions. Refuse: abstain when a clean reference stream or matched runtime comparison lacks support. Decompose: report protocol execution, operational false admission, and structural hypotheses separately rather than as one PASS/FAIL label. Refresh: treat distribution-shift alarms as requests to invalidate and recompute a reference map, not as fault evidence. We instantiate the protocol in an aggregate-only simulator with 24 policy components, three demand regimes, two fault-mask families, and independent development and hel
提出确保闭合环 AI 评估安全的协议，提高评估准确性和可靠性。
- 核心方法
- 设计了一种包括拒绝、分解、刷新三个动作的协议，以确保评估过程中声明的安全性和可靠性。
- 适合谁读
- 研究者、系统评估工程师
- 要解决的问题
- 闭合环系统中的 AI 评估可能因不可见的偏差和故障而支持错误的声明。
- 关键实验
- 在仅聚合仿真器中进行了实例化，包含24个策略组件、三种需求制度、两个故障掩码家族及独立开发和帮助环境。
- 主要贡献
- 提供了一种新颖的方法来识别和纠正评估过程中的偏差，提高了闭合环系统的评估准确性。
- 意义与局限
- 该协议有助于改进 AI 系统的评估过程，减少因评估偏差导致的决策错误。局限在于其复杂性和实施成本，以及对特定环境的适应性。