ai.hackcv
论文精选 82arXiv

Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems· 无规则的判决:评估LLM合规系统的规则敏感性

Large language model compliance systems are deployed on the assumption that a verdict depends on the regulatory rule it is given. We test this directly across five models and 20 regulatory and platform-policy domains: delete, swap, or negate the governing rule while holding the case fixed, and check whether the verdict changes (OCS) or the model's internal representation of compliance shifts at all (ICS-delta). Neither moves much: models' verdicts are often invariant to substantial perturbations of the supplied rule, and the guard model, evaluated here under a custom-rule adaptation of its native taxonomy, is the least rule-sensitive and least accurate of the five, barely above chance (51%, versus 90-92% for general-purpose models). This reflects easy cases more than blanket neglect: on ca

领域:cs.AI作者:Saisab Sadhu、Aadit Sengupta、Vinay kumar Sankarapu
相关推荐
论文精选 65已解读

Simulated Control Evals· 模拟控制评估

Control evaluations measure whether safeguards can prevent an AI model from causing undesi…

领域:cs.AI作者:Jeffrey Olmo、David Zhang、Tyler Tracy
📎 arXiv🕒 10-23 01:26🔗 arxiv.org

本站内容由 LLM 精选聚合,原文版权归 arXiv 所有 · 摘录仅供参考