ai.hackcv
论文精选 82arXiv

Function Lives Where Variance Doesn't: Task-Weighted Charts of a Language Model's Computation

How many dimensions does a language model's computation actually use? The question is ill-posed until one names a functional. Task-weighted charts make it well-posed: low-dimensional coordinate systems fit against a chosen functional of the representation, under the functional's own metric, turning distillation into plain least squares. Across six models from three families, spanning 70m to 7B parameters, next-token prediction needs 70--90% of the residual stream's width to stay within 5% of intact perplexity, a width consumed by the rare tail of language, and the variance profile predicts none of it: two directions carry 90% of GPT-2's activation variance and almost none of its function. Dimension is per-functional: the model's own uncertainty reads from six coordinates where the full pre

领域:cs.AI作者:Alexandre Quemy
相关推荐
论文精选 65已解读

Simulated Control Evals· 模拟控制评估

Control evaluations measure whether safeguards can prevent an AI model from causing undesi…

领域:cs.AI作者:Jeffrey Olmo、David Zhang、Tyler Tracy
📎 arXiv🕒 10-23 01:26🔗 arxiv.org

本站内容由 LLM 精选聚合,原文版权归 arXiv 所有 · 摘录仅供参考