Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning· 带有人类反馈的策略迭代
Generative pretraining established reusable task representations; later work on language-based task conditioning and in-context learning showed that a fixed model could adapt its behavior from instructions and demonstrations. Policy Iteration with Human Feedback (PIHF) builds on this development and the recurrent evaluate-and-improve structure of generalized policy iteration. PIHF uses a pretrained language model as its execution substrate and moves persistent revision to a versioned natural-language policy and tool set. A language-model critic and clinical expert review complete-panel reasoning and tool-use trajectories to localize recurrent failures and form candidate revisions; the expert may reinterpret the evidence and retains authority over admission and rollback, while Recall@1 and
结合人类反馈的策略迭代方法,实现模型在情境中的持续改进。
- 核心方法
- PIHF 使用预训练语言模型作为执行基础,通过专家审查和语言模型批评家的轨迹分析,定位反复出现的错误并提出修订建议,允许专家对调整和回滚有最终决定权。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 如何利用人类反馈,在不重新训练模型的情况下,改进和优化预训练语言模型的行为策略。
- 关键实验
- 未提供
- 主要贡献
- 提出了一种新的策略迭代框架,能够在模型部署后通过上下文学习和人类反馈不断改进模型的决策能力。
- 意义与局限
- 该方法为模型的持续优化提供了实用途径,特别是在无法频繁重新训练的情况下。然而,依赖专家审查可能限制其扩展性。