Beyond SAEs: Consistent Concept Discovery with Sparse Coding· 超越 SAE：用稀疏编码一致发现概念
Sparse autoencoders (SAEs) explain foundation model representations by decomposing activations into sparse combinations of learned dictionary atoms. However, the learned dictionaries can vary across training runs, limiting the reliability and reproducibility of SAE-based analyses. We propose Consistent Sparse Dictionary Learning (ConSDL), which replaces the standard linear encoder with per-sample sparse coding. This is motivated by our synthetic analysis showing that coefficients from a single linear encoder pass are suboptimal compared with those obtained by per-sample optimization, suggesting that sparse coding is a reliable mechanism for dictionary learning. Across 16 synthetic settings, ConSDL achieves near-perfect dictionary and coefficient recovery. These gains translate to state-of-
提出 ConSDL，通过样本级稀疏编码提升 SAE 分析的一致性和可靠性。
- 核心方法
- 引入 Consistent Sparse Dictionary Learning (ConSDL)，用样本级稀疏编码代替线性编码器，优化字典学习过程。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- SAE 学习的字典在不同训练中变化大，影响分析结果的可靠性和可重复性。
- 关键实验
- 在16种合成设置下进行实验，验证 ConSDL 的性能优于传统 SAE 方法。
- 主要贡献
- 在16种合成设置下，ConSDL 实现了近乎完美的字典和系数恢复，显著提升了 SAE 的一致性和可靠性。
- 意义与局限
- 提高了概念发现的可靠性，对理解基础模型内部机制有重要意义；但可能计算成本较高。