TraceForge: Distilling with Your Own Thinking· TraceForge: 用你的思维进行蒸馏
Using an LLM to generate training data for another is among the most general ways to train one, and it remains common practice. Some frontier teams have closed that route for their own models: what comes back is an answer, or a short summary, without the detailed reasoning trace behind it. That is a little ironic: they were themselves built on distilling human knowledge, and distilling an open-source model is still entirely routine; the restriction applies to their own outputs alone. Recent work recovers the hidden reasoning by replaying encrypted blocks across models. It somehow works, but it breaks the providers' terms of service, so it is not something a product can be built on long-term. We built the alternative. We take the context and the teacher's answer as input, and generate the r
提出TraceForge，从大语言模型的回答中恢复隐藏的推理过程。
- 核心方法
- TraceForge通过输入上下文和教师模型的简短回答，生成详细的推理轨迹，采用合规的方法模拟语言模型的思考过程。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 如何从前沿大语言模型的输出中恢复详细的推理过程，而不违反服务条款。
- 关键实验
- 展示了TraceForge在多个任务上的表现，证明其生成的推理轨迹与人类思路高度一致，但未提供具体数据。
- 主要贡献
- 首次提出一种合规的、有效的方法来从简短的回答中恢复详细的推理过程，增强模型的可解释性。
- 意义与局限
- 为大语言模型的可解释性提供了新思路，但可能需要进一步的性能优化和广泛验证。