Capability from Access Structure, Not Scale: Lower Bounds and Pre-Registered Tests for Hybrid Sequence Models· 模型规模与能力
The Platonic Representation Hypothesis (PRH) holds that as models scale, representations of heterogeneous networks converge toward a shared model of reality. We propose its sequel and boundary, the Capability Convergence Hypothesis (CCH): under a fixed per-token inference budget, representational convergence does not entail capability convergence. Capability instead converges toward a class, the access-complete hybrid: any architecture holding both a compressive O(1)-state channel and a scalable verbatim-index channel. We anchor it on a witness task, the Newton's-apple problem in an infinite s
模型能力取决于结构而非规模
- 核心方法
- 提出能力收敛假设(CCH),并在固定推理预算下通过牛顿苹果问题测试混合序列模型的能力
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 探讨大规模模型是否必然带来表征和能力的收敛
- 关键实验
- 通过牛顿苹果问题进行预注册测试
- 主要贡献
- 明确指出压缩状态通道与可扩展索引通道共同存在的架构能实现能力收敛
- 意义与局限
- 挑战了关于模型规模与能力之间关系的普适假设,强调模型结构的重要性