Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity· 渐进式稀疏化，适用于极端稀疏条件下的大模型剪枝
Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse
提出 ETAPS 方法，实现大模型在极端稀疏条件下的高效剪枝和质量保持。
- 核心方法
- ETAPS 方法通过渐进式紧缩全局稀疏性约束，并在每个阶段适应性地调整稀疏连接，以减少优化状态和辅助变量的开销，同时保持模型质量。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 在高稀疏度条件下，大语言模型的性能下降和成本增加问题。
- 关键实验
- 展示了在多个大语言模型上，ETAPS 方法在不同稀疏度下的性能表现，并与现有方法进行了对比。
- 主要贡献
- 在保持模型性能的同时，实现了更高效的剪枝过程，尤其适用于极端稀疏条件。
- 意义与局限
- 该方法有望降低大语言模型的存储和推理成本，但可能需要更长的训练时间。