Verifiable Social Reasoning for LLM Assistants· LLM助手的可验证社会推理
LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others' intentions, typically lack verifiable ground truth. To address these challenges, we introduce Fuse, a multi-agent simulation framework for studying user-mediated social reasoning. In Fuse, a target agent with a hidden motive interacts with other agents including one representing the user, who then consults the evaluated assistant to infer the target's motive, providing verifiable ground truth by construction. Simulation faithfulness is validated through a human study with 24k annotations. We ap
LLM助手在社会咨询中的推理能力验证框架。
- 核心方法
- 提出名为Fuse的多代理模拟框架，通过隐藏动机的目标代理与用户代理的互动，构建可验证的社会推理场景，允许评估者根据用户的咨询来推断目标代理的动机。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 如何有效评估LLM助手在社会咨询场景中的推理能力，尤其是当他人的意图等社会属性缺乏可验证的真实标准时。
- 关键实验
- 通过一个人类研究和24,000次标注验证了模拟框架的忠实度。
- 主要贡献
- 提供了LMM助手在社会咨询任务中推理能力的评估方法和工具，确保社会属性推理的可验证性。
- 意义与局限
- 为研究和开发更可靠的社会智能LLM助手提供了重要基础，但仍需进一步探索如何将该框架应用到更复杂和真实的社会场景中。