4-bit

UltraQuant: 4-bit KV Caching Achieves 3.47x Latency Cut for Context-Heavy AI Agents

A new technique called UltraQuant compresses the key-value cache in large language models to 4 bits, achieving a 3.47x reduction in latency and 1.63x increase in throughput for context-heavy, multi-turn agent workloads on AMD GPUs. The method, detailed in an arXiv paper, adapts rotation and codebook quantization for robust long-context serving.