ai.hackcv
论文精选 82arXiv

PersonaChatBench: Evaluating Chat Assistants with Diverse Simulated Users· PersonaChatBench:用多样化模拟用户评估聊天助手

Conversational assistants are expected to help people with diverse backgrounds, preferences, and communication styles. Yet evaluations centered on a single cooperative user provide limited insight into how reliably an assistant serves users with different personas. We introduce PersonaChatBench, a framework for evaluating task competence and responsiveness to diverse user needs. PersonaChatBench pairs the same underlying task with multiple simulated users, each guided by a private persona card that specifies distinct needs, preferences, and interaction behaviors. By holding the task fixed while varying the user, the framework examines how assistant performance changes across user conditions. We evaluate 20 language models across six domains, separately assessing native task outcomes, fulfi

领域:cs.CL作者:Zheyuan Deng、Taoran Li、Jingwen Zhang
相关推荐
论文精选 82

Elucidating the Design Space of LM Data Attribution· 阐明大模型数据归属的设计空间

Data attribution methods for Language Models (LMs) are typically evaluated using downstrea…

领域:cs.CL作者:Steven Yu、Sheng-Yu Wang、Kimia Hamidieh
📎 arXiv🕒 10-30 01:59🔗 arxiv.org

本站内容由 LLM 精选聚合,原文版权归 arXiv 所有 · 摘录仅供参考