Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention· 潜在稀疏注意力：实现次平方的超长上下文注意力
Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training f
提出潜在稀疏注意力，解决超长上下文任务中注意力计算的成本问题。
- 核心方法
- 潜在稀疏注意力（LSA）通过在学习到的潜在空间中对键进行层次聚类，查询时对聚类层次打分而非逐个键打分，从而高效维持准确度。
- 适合谁读
- 适合关注超长上下文处理的研究者和工程师阅读。
- 要解决的问题
- 在超长上下文（超过1百万token）的任务中，全注意力机制计算成本过高，而现有稀疏注意力方法在该规模下准确率下降。
- 关键实验
- 作者展示了LSA在多种超长上下文任务上的性能，与现有方法相比，具有明显的效率和准确率优势。
- 主要贡献
- LSA减少了超长上下文注意力的计算成本，同时保持了准确率，为处理大规模文本提供了新方法。
- 意义与局限
- 此方法可能显著提升超长文本处理的效率，促进更广泛的应用。但其在不同任务间的泛化能力和对极端规模文本的处理效果仍需进一步验证。