Character Training for Risk-Averse Agents· 为风险厌恶代理人进行性格训练
Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making deals with humans over riskier strategies like rebelling. We train agents to be risk averse through character training, finding that persona traits provide a robust mechanism for instilling risk preferences. To do this, we construct a model constitution describing constant absolute risk aversion (CARA) over an agent's resources and instill it through on-policy distillation. Despite never seeing the benchmark's decision format during training, character-trained models are competitive with baselines trained directly on it, and generalise better than them out of distribution on two of our four models. We also modulate
研究通过性格训练使 AI 代理风险厌恶，促进安全性。
- 核心方法
- 构建一个描述恒定绝对风险厌恶 (CARA) 的模型宪法，通过在线策略蒸馏来灌输个性特征。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 如何防止不一致的 AI 代理因资源风险厌恶而采取危险策略，如反抗。
- 关键实验
- 在四个模型中的两个模型上，性格训练的模型在分布外测试上的表现优于基准模型。
- 主要贡献
- 提供了一种利用性格特征训练使 AI 代理风险厌恶的方法，提高了模型在未见过的决策格式上的泛化能力。
- 意义与局限
- 提高了 AI 系统的安全性，减少了潜在的灾难性伤害，但仍需进一步研究以提高所有模型的泛化能力。