Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models· 超越因果掩码
Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs e
探讨自适应序列模型中隐藏的未来令牌泄漏问题及其影响。
- 核心方法
- 通过层级诊断和实现检查，识别并修复模型中的全序列摘要广播问题，防止未来令牌泄漏。
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 自适应序列模型在使用正确的因果掩码时仍可能出现未来令牌泄漏，导致模型评估失真。
- 关键实验
- 关键实验包括对24层模型的前缀不变性审计及生成过程中的参考模型困惑度分析。
- 主要贡献
- 揭示了未来令牌泄漏的机制，并提出了有效的修复方案，提高了模型生成的一致性和性能。
- 意义与局限
- 该研究有助于提升自适应序列模型的可靠性和公正性，但方法的广泛适用性仍有待验证。