Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models· 形式而非内容?
Frozen small code LLMs are deployed locally, yet the information guiding a retry after a failed attempt is still measured without placebo controls in the self-repair literature. We treat a failed program as a conjecture and an execution counterexample as an oracle-relative refutation, and introduce PoPE (Popperian Placebo-controlled Evaluation): a methodology for measuring whether evidence that falsifies LLM-generated code can be used operationally by that same model. In PoPE, error content is paired with channel-specific placebos that keep the predeclared scaffold while ablating task-relevant content or deranging the task-error assignment. Frozen small code models (0.5-1.5B) are evaluated under preregistered rules through a prompt channel and a weight channel (small-data adapter training), with four generations per arm-unit pair. In the prompt channel, public-tier screening unlocked 12 units under the content-ablated form placebo versus 10 under the live error-pattern arm on a 40-unit resistant band; the result was recorded as mechanism-null. In the weight channel, an 8-8 tie was observed between the error-content adapter and the intervention-free baseline (p=1.0), while the SHA-deranged placebo adapter stayed ahead with 10 unlocks; content-attributable superiority was not confirmed. These results do not constitute evidence of equivalence or non-inferiority. Equivalence was not tested separately. Findings are restricted to the public-tier screening endpoint; hidden-tier confirmation was deferred by design. We read this not as compiled criticism disappearing as information, but as the loss of its external role in testing a new conjecture: when a representation learned from the oracle is written back into the generation state, testing is replaced by conditioning. No working JEPA-RL controller is claimed. PoPE is presented as a placebo-controlled, retestable measurement standard.
研究冻结小规模代码模型在自我修正时的形式作用.
- 核心方法
- 引入PoPE方法,将错误内容与特定通道的安慰剂结合,通过提示和权重调整评估模型自我修正能力.
- 适合谁读
- 研究者,尤其是关注LLM自我修正机制和模型评估方法的学者
- 要解决的问题
- 探索如何有效地评估冻结的小规模代码生成模型在遇到错误时是否能通过自我修正机制改进其输出.
- 关键实验
- 在提示通道和权重通道下,分别对有错误内容和安慰剂的干预进行了实验,未发现显著差异.
- 主要贡献
- 提供了一个安慰剂控制的、可重复的测量标准PoPE,用于评估LLM的自我修正机制.
- 意义与局限
- 研究结果表明,错误信息的形式可能比内容更重要,但未提供强有力的证据支持这一点。设计上限制了研究范围,建议进一步探索。