Enterprise AI agents that maintain long conversational contexts — such as supply chain chatbots handling multi-step logistics queries or customer support bots with extensive session histories — face a persistent bottleneck: the key-value (KV) cache. As context length grows, this cache consumes GPU memory and increases latency. A new pre-print from researchers at AMD and other institutions introduces UltraQuant, a 4-bit KV-cache compression method that promises to cut time-to-first-token by 3.47x in cache-pressured late rounds and improve output throughput by 1.63x over the standard FP8 baseline.
The Problem with Context-Heavy Agents
Context-heavy agents reuse long prefixes across many short turns, a pattern that strains serving infrastructure. According to the arXiv paper by Chakrabarti et al., concurrent users amplify pressure on the KV cache, determining whether GPUs stay utilized. Existing FP8 KV caching (as in vLLM) provides a baseline, but the authors argue that joint measurement of task quality, cache residency, and serving throughput is needed for agent workloads. Traditional 4-bit compression often degrades quality; UltraQuant aims to make it robust.
UltraQuant's Approach
The researchers frame 4-bit KV caching around multi-turn agent workloads. They describe practical design choices to make the 4-bit path robust:
- Asymmetric K/V treatment: separate quantization strategies for keys and values to preserve quality.
- Walsh-Hadamard rotation: a fast transform that distributes outlier values across dimensions, improving quantization fidelity.
- QJL removal: skipping Query-Johnson-Lindenstrauss projections to simplify the pipeline.
- Block-scale variants: grouping tokens into blocks with shared scaling factors for finer granularity.
The UltraQuant path specifically uses an FP4 approximation: FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on AMD's CDNA4 architecture. This combination enables efficient execution on AMD GPUs.
Results on AMD GPUs
On a long-context, multi-turn agentic workload, UltraQuant demonstrated significant performance gains over the FP8 KV baseline:
| Metric | Improvement vs FP8 Baseline |
|---|---|
| P50 time-to-first-token (late rounds, cache-pressured) | 3.47x faster |
| P50 time-to-first-token (all rounds) | 2.3x faster |
| Output throughput | 1.63x higher |
These results highlight the method's effectiveness under the exact conditions where context-heavy agents suffer most: long prefixes, many short turns, and high concurrency.
Implications for Enterprise AI
For enterprise customers deploying AI agents — whether for trade document analysis, customs classification, or logistics visibility — every millisecond of latency directly impacts user experience and throughput. UltraQuant's 4-bit compression allows more agent instances per GPU, lowering total cost of ownership. The technique is particularly relevant for companies building on AMD infrastructure, where native FP4 and scaled-MFMA support on CDNA4 can be leveraged without custom kernel development.
The paper's emphasis on robustness — through asymmetric K/V processing and block-scale variants — addresses a common barrier to adopting low-bit quantization in production. TurboQuant-style rotation serves as a quality anchor, while vLLM FP8 caching provides the deployment benchmark. As LLM-powered agents become standard in trade finance, supply chain coordination, and customs automation, efficient caching techniques like UltraQuant will be critical to scaling these systems cost-effectively.
The research is led by Chakrabarti, Inesh, Limpus, David, Rana, Aditi Ghai, Bao, Bowen, Tiwari, Spandan, Crepaldi, Thiago, and Sirasao, Ashish. The full paper is available on arXiv.