publications

Showing 2 of 2 publications

2026

1. Preprint

   HarmonyRoute: Maximize Compute-Budget Utility with Coordinated Multi-Model Routing

   Boning Huangfu, Boxiao Du, Chen Chen

   2026 · Preprint coming soon

   AbsDetails

   HarmonyRoute routes LLM requests that share a fixed compute budget, such as a spending cap on rented model APIs or the capacity of a private GPU cluster, to maximize their total satisfaction. It predicts how candidate models compare from past requests with the same routing choices, plans each choice against the compute later requests will need, and drops or upgrades in-flight requests during generation. On Azure Conversation and BurstGPT traces, it improves total satisfaction by 18.72% over the strongest existing router.

2. arXiv

   GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources

   Boxiao Du^*, Boning Huangfu^*, Yizhou Luo, and 5 more authors(^*Equal contribution)

   arXiv preprint arXiv:2605.16867, 2026

   AbsDOIarXivBibHTMLPDFDetails

   GoodServe is a goodput-optimized serving system for agentic LLM inference over heterogeneous GPU resources. It combines predict-and-rectify routing, just-enough instance selection, SLO-risk monitoring, and request migration to improve goodput for latency-sensitive agentic workloads.

   @article{goodserve2026,
     title = {GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources},
     author = {Du, Boxiao and Huangfu, Boning and Luo, Yizhou and Chen, Chen and Li, Zijun and Yu, Minchen and Fan, Xiaoyi and Guo, Minyi},
     journal = {arXiv preprint arXiv:2605.16867},
     year = {2026},
     arxiv = {2605.16867},
     doi = {10.48550/arXiv.2605.16867}
   }