Artificial Intelligence #distributed optimization#low-rank
LoRDO Algorithm Cuts Communication by 10x for Distributed AI Model Training
LoRDO (Low-Rank Distributed Optimization) unifies low-rank optimization with infrequent synchronization to reduce communication overhead in distributed training of foundation models. According to an arXiv paper, it achieves near-parity with low-rank DDP at scales 125M–720M parameters while cutting communication by approximately 10x, and shows further gains in very low-memory settings.
Jul 8, 2026 1 source