What Do RTL Repair Benchmarks Actually Measure?· RTL 修复基准实际测量了什么？
RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds. First, simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of the simulation-formal disagreement to reset and first-cycle comparison semantics. Second, on 274 mined model failures, the specification-only minus specification-conditioned recovery point estimate ran
分析 RTL 修复基准的实际测量效果及存在的问题
- 核心方法
- 构建新的数据集 RTLRepairDataset 和基准 RTLRepairBench，包括模拟和形式等价评估工具
- 适合谁读
- 研究者 / 工程师
- 要解决的问题
- 当前 RTL 修复基准无法准确评估模型修复能力
- 关键实验
- 使用新构建的基础设施评估了 274 个模型失败案例
- 主要贡献
- 揭示了模拟和形式评估之间的矛盾，指出重置和首周期比较语义是主要原因
- 意义与局限
- 提高了对 RTL 修复评估方法的理解，为未来研究提供更可靠的基准和工具