End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services· 面向边缘的LLM推理请求调度
Large language model (LLM)-powered agentic AI services increasingly demand low-latency inference, motivating the deployment of LLMs across distributed edge servers. However, heterogeneous communication and computing capabilities, together with dynamically evolving inference states, make the edge server selection for each incoming request time-varying and tightly coupled across slots. In this paper, we investigate an online request scheduling framework for edge LLM inference that jointly minimizes long-term average end-to-end latency and regulates workload distribution across heterogeneous edge servers. Two main challenges arise in this context. First, conventional latency models cannot accurately capture the fine-grained dynamics of multi-stage LLM execution. Second, the latency consequenc
研究面向边缘的LLM推理请求调度，减少端到端延迟和平衡负载。
- 核心方法
- 提出一种在线请求调度框架，针对多阶段LLM执行的细粒度动态特性设计新的延迟模型，并通过优化算法解决时间变化和紧密耦合的调度问题。
- 适合谁读
- 研究者、工程师
- 要解决的问题
- 如何在异构边缘服务器上有效调度LLM推理请求，同时最小化长期平均端到端延迟和均衡工作负载。
- 关键实验
- 通过仿真实验验证了所提出算法在不同场景下的性能，与现有方法相比，能够显著降低延迟并提高负载均衡。
- 主要贡献
- 1. 设计了能够准确反映多阶段LLM执行动态特性的延迟模型；2. 提出了一种能够同时最小化端到端延迟和平衡工作负载的在线调度算法。
- 意义与局限
- 本研究有助于提升边缘计算环境中LLM推理的效率和用户体验，但算法复杂度较高，实际部署时可能需要更多资源。