DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon· DGA-Muon: 解耦几何对齐自适应缩放
While NorMuon has achieved strong empirical performance in large-scale pretraining by enhancing Muon with row-wise adaptive scaling, its underlying adaptive mechanism remains poorly understood. In this work, we provide the first systematic analysis of NorMuon's adaptivity, revealing that it originates primarily from orthogonalization-induced geometry rather than genuine optimization-relevant information. Under exact orthogonalization, the adaptive scaling factors degenerate into a single global scalar for square and wide matrices, while for tall matrices their variation arises from the non-uniform distribution of row energy after orthogonalization. Under approximate orthogonalization, the orthogonality residual introduces additional variation into the scaling factors, leading to the \texti
通过解耦几何对齐自适应缩放改善大模型预训练方法。
- 核心方法
- 分析 NorMuon 的自适应机制,发现其主要来自于正交化诱导的几何结构,而非优化相关信息。对于不同形状的矩阵,提出了解耦几何对齐自适应缩放(DGA-Muon)方法以改进这一机制。
- 适合谁读
- 研究者
- 要解决的问题
- 现有的 NorMuon 方法在大模型预训练中虽然表现良好,但其自适应机制尚未被充分理解。
- 关键实验
- 未提供
- 主要贡献
- 首次系统地揭示了 NorMuon 自适应机制的来源,并提出了改进方法 DGA-Muon,优化了不同矩阵形状下的自适应缩放。
- 意义与局限
- 该研究有助于理解大型预训练模型中的自适应机制,为未来优化方法的研究提供了理论基础。局限性在于未提供实验验证改进效果。