Phantom Gains: Auditing Self-Improvement Against a Measured Null
Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repa
审慎评估语言模型自我改进中的测量误差。
- 核心方法
- 通过与未训练但经过相同管道处理的对照组进行对比，识别出七种测量失败情况，分析了这些失败如何在没有对照组的情况下导致虚假结论。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 论文解决了在评估语言模型自我改进时因测量误差导致的虚假增益问题。
- 关键实验
- 实验包括对 Qwen3-8B 模型进行三轮 LoRA 自训练，并与冻结的对照组进行对比分析。
- 主要贡献
- 揭示了当前评估语言模型自我改进的方法中存在的测量误差问题，并提出了改进的审计方法。
- 意义与局限
- 研究强调了评估语言模型自我改进时的严谨性，对防止虚假进步有重要影响。