Topic
quantization
UltraQuant: 4-bit KV Caching Achieves 3.47x Latency Cut for Context-Heavy AI Agents
A new technique called UltraQuant compresses the key-value cache in large language models to 4 bits, achieving a 3.47x reduction in latency and 1.63x increase in throughput for context-heavy, multi-turn agent workloads on AMD GPUs. The method, detailed in an arXiv paper, adapts rotation and codebook quantization for robust long-context serving.
Researchers Identify Shrinkage Bias in LLM FP4 Pretraining, Propose UFP4 Recipe for Stability
A new study from researchers on arXiv identifies 'Shrinkage Bias' in E2M1-based FP4 pretraining for large language models, a systematic error that accumulates across layers. The proposed UFP4 recipe, using uniform grids like E1M2/INT4, demonstrates lower BF16-relative loss degradation on models up to 124B parameters, urging hardware support for uniform 4-bit formats.
RaBiT: Residual-Aware Binarization Training for Accurate and Efficient Large Language Models
Researchers propose RaBiT, a quantization framework that resolves pathological feature co-adaptation in residual binarized LLMs. RaBiT delivers state-of-the-art 2-bit accuracy and 4.49x inference speed-up on an RTX 4090, rivaling hardware-intensive Vector Quantization methods.
New Automated Quantization Framework AQ4SViT Compresses Spiking Vision Transformers for Embedded AI
Researchers propose AQ4SViT, an automated quantization framework for Spiking Vision Transformers that uses a search gating policy to find optimal compression settings. It offers two variants: Greedy search for speed and Beam search for deeper compression. Experimental results on ImageNet show up to 6.6x faster search time and up to 90% memory savings while maintaining accuracy within 1.5% of the original model.