FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current hardware paths and recipes—including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs—remain centered on E2M1 data elements. A new study from researchers on arXiv identifies a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins.
The Origin of Shrinkage Bias
The researchers show that Shrinkage Bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality.
UFP4: A Uniform 4-bit Recipe
Based on this finding, the researchers propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. This approach leverages the benefits of uniform quantization while minimizing additional complexity.
Experimental Validation
The experiments covered Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining. According to the study, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. The following table summarizes the key results:
| Model | E2M1 Baseline Loss Degradation | UFP4 Loss Degradation | Improvement |
|---|---|---|---|
| Dense 1.5B | Higher (not specified) | Lower (not specified) | Significant |
| MoE 7.9B | Higher | Lower | Significant |
| MoE 124B | Higher | Lower | Significant |
Note: Exact numbers are not provided in the abstract.
Implications for Hardware Design
The paper suggests that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1. This recommendation aligns with the observed benefits of uniform formats in mitigating Shrinkage Bias and improving training stability.
For enterprise technology leaders, this research highlights a critical consideration when adopting low-precision training for large models. The choice of quantization format—E2M1 versus E1M2/INT4—directly impacts training stability and final model quality. As LLMs become integral to supply chain and logistics applications (e.g., demand forecasting, route optimization), ensuring efficient and stable pretraining will reduce operational costs and accelerate deployment timelines. Hardware vendors and cloud providers may need to adapt their roadmaps to include uniform 4-bit support, as advocated by the UFP4 recipe.