Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions· LLM 决策记忆的挑战
When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-fir
LLM 决策记忆中的问题与对策分析
- 核心方法
- 通过引入 CrossProfile 基准测试，研究罕见性和相关性两种记忆策略的效果，并分析其失败原因
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 研究 LLM 在组决策过程中，压缩讨论内容为简短记忆时，如何有效保留有用信息而避免误入歧途
- 关键实验
- 使用 Qwen2.5-7B、Llama-3.1-8B 和两个 DeepSeek 模型进行了 480 个任务的实验
- 主要贡献
- 提出了 CrossProfile 基准测试，揭示了 LLM 决策记忆中罕见性和相关性策略的局限性
- 意义与局限
- 为 LLM 决策记忆机制提供了重要参考，指出罕见性和相关性策略在某些任务中的失效问题，有助于未来研究优化 LLM 的记忆能力