HarmonyRoute: Maximize Compute-Budget Utility with Coordinated Multi-Model Routing

Abstract

HarmonyRoute routes LLM requests that share a fixed compute budget, such as a spending cap on rented model APIs or the capacity of a private GPU cluster, to maximize their total satisfaction. It predicts how candidate models compare from past requests with the same routing choices, plans each choice against the compute later requests will need, and drops or upgrades in-flight requests during generation. On Azure Conversation and BurstGPT traces, it improves total satisfaction by 18.72% over the strongest existing router.

Citation

@misc{harmonyroute2026,
  title = {HarmonyRoute: Maximize Compute-Budget Utility with Coordinated Multi-Model Routing},
  author = {Huangfu, Boning and Du, Boxiao and Chen, Chen},
  year = {2026}
}