Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale· 生产级客服体验AI代理的模拟
Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust. We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Ma
生产级客服AI代理的模拟方法，有效预测试其性能。
- 核心方法
- 通过构建假设驱动的模拟流程，使用合成客户与模拟工具输出，来测试和筛选客服AI代理，无需依赖生产环境。
- 适合谁读
- 研究者、工程师、产品经理
- 要解决的问题
- 如何在保障客户体验的同时，有效测试和优化客服AI代理，尤其是在受监管行业中。
- 关键实验
- 在Nubank的Card Delivery代理及其扩展版本Card Ma上使用Snowglobe模拟器进行了测试。
- 主要贡献
- 提出了一种在大规模客服场景中有效预测试AI代理性能的方法，减少了对客户的潜在负面影响。
- 意义与局限
- 该方法可以提高客服AI代理的可靠性和效率，减少手动测试的局限性，对受监管行业尤有价值。局限性在于模拟环境可能无法完全复现真实场景的所有复杂性。