How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks· 前沿模型在物理学中的表现如何？
Low reported scores on leading physics benchmarks, including those featured in the Artificial Analysis Intelligence Index (2026), suggest that frontier language models still struggle with advanced physics, a demanding test of their scientific reasoning and quantitative problem-solving abilities. Yet this impression does not always align with domain experts' experiences using these models in their work. We revisit these reported findings by evaluating frontier models on six widely used physics benchmarks and auditing them with experts, focusing on text-only problems with verifiable final answers. For each subfield of physics, faculty and graduate researchers with relevant expertise carefully review problem statements, reference solutions, and model responses to distinguish genuine model err
前沿语言模型在物理学问题上的实际表现与现有评估存在差距。
- 核心方法
- 通过专家重新评估和审核六个广泛使用的物理基准测试，专注于可验证最终答案的文本问题，详细审查问题陈述、参考解决方案和模型响应。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 现有物理基准测试低估了前沿语言模型在高级物理学问题上的科学推理和定量解决问题的能力。
- 关键实验
- 重新评估了六个物理基准测试，包括问题陈述、参考解决方案和模型响应的详细审核。
- 主要贡献
- 揭示了现有评估方法的缺陷，展示了前沿模型在某些物理领域接近饱和的表现。
- 意义与局限
- 该研究有助于改进对语言模型的评估方法，更好地反映其在科学领域的实际能力，但仍需进一步研究其他领域的表现。