Which Differences Matter? Controlled Tests of Rule Transfer, Joint Reliability, and Information Use in Decision Models
Decision models map stated rules and observed states to discrete answers. Aggregate accuracy leaves unresolved which rule and information changes guide those answers. We connect controlled input changes to transfer, joint reliability, and directed information use through executable arithmetic rules. Semantic rule separation and crossed rule and state tests make transfer and joint success separately measurable. We compare equivalent wordings, calibrate joint correctness through marginal-preserving reassociation, and alter tables to define alternative answer targets. Adaptation gains increase with nearest-source answer agreement; fully conflicting quartets yield small pooled gains and different gain signs across source splits. Equivalent rewrites have opposing task-level effects, linking tra
探讨决策模型中哪些规则和信息变化真正影响模型答案。
- 核心方法
- 通过可控输入变化的实验，结合可执行的算术规则，评估模型在规则迁移、联合可靠性以及信息使用方面的能力。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 现有的模型评估方法无法明确区分哪些规则或信息变化真正影响了模型的决策输出。
- 关键实验
- 比较了等效表述的效果，通过边缘保留重关联校准联合正确性，并通过修改表格定义不同的答案目标。
- 主要贡献
- 提出了一种新的评估框架，可以独立测量模型在规则迁移和联合可靠性方面的表现。
- 意义与局限
- 提供了对决策模型内部机制更深入的理解，但可能仅适用于特定类型的模型和任务。