Scaling Clinical Judgment to Evaluate Medical AI· 大规模评估医学AI的临床判断
Blinded physician evaluation has been considered by many to be the gold standard for assessing clinical reasoning in large language models (LLMs). This is difficult to scale; thus, prior studies typically rely on small physician panels, often from a single institution or specialty, which both limits the scientific questions investigated and makes it unclear whether findings would be reproduced with a different set of evaluators. To more rigorously and scalably study clinical reasoning in AI models, here we introduce PrecepTron, an LLM fine-tuned for physician-level evaluation of open-ended responses. PrecepTron was trained using low-rank adaptation (LoRA) of a 32-billion-parameter model on a small number of physician examples. We also release GRAND-ROUNDS, a new large-scale physician-annot
引入PrecepTron，用于大规模评估医学AI的临床判断能力。
- 核心方法
- 使用低秩适应(LoRA)对32亿参数的大模型进行微调，构建专门用于医师水平评估AI开放性回答的PrecepTron模型，并开发大规模医师标注数据集GRAND-ROUNDS。
- 适合谁读
- 研究者、工程师、医疗机构
- 要解决的问题
- 现有医学AI评估方法受限于小范围、单一机构的医师小组，缺乏普适性和可扩展性。
- 关键实验
- 在GRAND-ROUNDS数据集上对PrecepTron进行了评估，显示出其在不同临床场景下的一致性和可靠性。
- 主要贡献
- 提供了可扩展的医学AI临床推理评估工具和大规模数据集，推动医学AI研究的标准化。
- 意义与局限
- 意义在于提高了医学AI评估的效率和准确性，为跨机构和跨专业的AI性能评估提供了可能。局限性包括可能存在的偏见和对新临床情境的泛化能力。