Enterprises deploying large language models (LLMs) face a fundamental tension: inference costs are skyrocketing as demand surges, yet users expect high-quality responses codified in Service Level Agreements (SLAs). Existing cost-aware routing approaches often rely on complete feedback signals, offline training, and extensive per-workload tuning, and most lack SLA guarantees or inference-time adaptivity. A new algorithm, SLARouter, directly addresses these shortcomings.
The Problem with Existing Routing Solutions
Current cost-optimal LLM routing methods require complete feedback (e.g., quality scores for every request) and offline training on large datasets. They also need careful tuning for each specific workload and typically do not provide theoretical guarantees on user satisfaction. According to the paper published on arXiv by researchers Woisetschläger, Herbert, Mammadli, Arastun, Zhang, Ryan, and Wang, Shiqiang, these limitations make existing approaches impractical for production systems where feedback is sparse and one-sided (e.g., only complaints are logged).
SLARouter: Online Learning from Sparse Feedback
SLARouter is an online routing algorithm that learns a cost-optimal policy from the sparse, one-sided user feedback available in real-world deployments. It provides theoretical guarantees for both cost optimality and strict SLA compliance. Unlike prior work, SLARouter operates without offline training or per-benchmark tuning, adapting to changing request patterns in real time. The algorithm is designed to work under limited feedback signals, making it suitable for commercial environments where explicit ratings are rare.
Performance Benchmarks and Cost Savings
The researchers evaluated SLARouter across a wide range of LLM benchmarks. Results show that SLARouter satisfies SLA constraints without requiring per-benchmark tuning, reducing operating cost by up to 2.2x over existing baselines. This cost reduction is achieved while maintaining strict adherence to user satisfaction guarantees, according to the paper. The study does not name specific baseline algorithms or benchmarks but emphasizes that SLARouter consistently outperforms them.
Implications for Enterprise AI Deployment
For CTOs and technology procurement leaders, SLARouter offers a pragmatic path to controlling LLM inference expenses without compromising service quality. By eliminating the need for extensive offline training and per-workload tuning, it reduces operational overhead. The theoretical SLA guarantees provide confidence that user expectations will be met, addressing a key concern in commercial settings. While the paper is a research preprint and not yet a commercial product, the algorithm could be integrated into existing LLM infrastructure or form the basis for future enterprise tools. As LLM usage continues to grow, solutions like SLARouter that balance cost and quality with minimal data requirements will become increasingly valuable.