EnigmaForge: The Question Is Hidden in the Story· EnigmaForge：故事中的谜题
Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and logbook margins is a small logic puzzle whose solution is unique - proved by a SAT solver at generation time, with an ablation certificate showing every clue is load-bearing. Because instances are generated rather than collected, the corpus renews forever. The headline measure is intuition: task success when handed only the story, with world reconstruction as the secondary axis. Twenty-five frontier models ran over 600 instances (17,400 scored records) under three matched conditions. Intuition reshuffles the leaderboard: a 22x spread where fact recovery spans 1.6x, the second-best fact-recoverer ranks fourteenth, one model is indifferent to b
通过生成的故事来测试模型的直觉和世界重构能力。
- 核心方法
- 使用生成的旧文档集合，其中埋藏着一个小逻辑谜题，且每个线索都是必要的，通过无问题的方式测试模型的直觉。
- 适合谁读
- 研究者
- 要解决的问题
- 现有的基准测试过多依赖于直接给出问题，而非测试模型从复杂信息中提取问题的能力。
- 关键实验
- 25个前沿模型在600个实例上进行了测试，共17,400条评分记录。
- 主要贡献
- 提出了一种新颖的基准测试方法，能够更全面地评估模型的推理和理解能力。
- 意义与局限
- 该方法能够更好地评估模型的真实推理能力，但由于是生成的数据，可能无法完全反映模型在真实世界中的表现。