RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons· RACE：可扩展的大模型神经元功能一致性统计估计
Discovering stable neuron behavior across entire domains remains a challenge in mechanistic interpretability. Existing methods often rely on instance-level point estimates or computationally expensive procedures, which either obscure population-level variability or limit scalable domain-wide analysis. We present RACE (Residual Alignment for Consistency Estimation), a forward-pass statistical framework that evaluates the domain-wide functional consistency of Transformer neurons. Perturbation experiments demonstrate that RACE achieves superior domain specificity compared to gradient-based point estimates. Meanwhile, token-distribution-level results verify the association between the selected neurons and the target domain. Furthermore, its computational overhead is two orders of magnitude low
提出 RACE 框架，评估大模型神经元的功能一致性。
- 核心方法
- RACE 通过正向传播的统计方法，评估 Transformer 神经元在整个目标域内的功能一致性，使用扰动实验和令牌分布级结果验证方法的有效性。
- 适合谁读
- 研究者
- 要解决的问题
- 现有方法在评估大模型神经元功能一致性时，存在实例级别的点估计或计算成本高的问题，难以进行大规模跨域分析。
- 关键实验
- 扰动实验与令牌分布级结果验证了 RACE 的有效性和域特异性。
- 主要贡献
- RACE 能够实现更优越的域特异性评估，同时计算开销低两个数量级。
- 意义与局限
- 意义在于提供了一种可扩展的方法来评估大模型神经元的功能一致性，有助于深入理解模型的机制。但方法的适用性需进一步验证。