MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations· MedRoundsQA:多轮医疗咨询的角色与难度感知评估
Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultatio
多轮医疗咨询评估基准,考虑角色与难度
- 核心方法
- 构建MedRoundsQA多轮医疗咨询评估基准,采用1,387个跨17专科的病例,转化为24个结构化临床记录槽位,并根据不同患者角色生成对话;使用模型不确定性对难度进行分类
- 适合谁读
- 研究者、工程师、医疗从业者
- 要解决的问题
- 现有医疗评估基准集中在单一回合的多选题上,无法真实反映临床实际情况
- 关键实验
- 对15个LLM医生代理进行了评估,显示多轮诊断比单回合更能测试真实临床能力
- 主要贡献
- 提供了更贴近实际医疗咨询过程的多轮评估基准,及基于模型不确定性的难度分类方法
- 意义与局限
- 有助于更好地评估医疗AI系统,提高其在实际应用场景中的可靠性和准确性;但对话的生成和评估标准可能需要进一步完善