Topic
low-rank adaptation
SDS-LoRA: New Low-Rank Adaptation Method Fixes Gradient Distortion in Large Model Fine-Tuning
A new paper on arXiv introduces SDS-LoRA, a low-rank parameterization that overcomes anisotropic gradient scaling in LoRA. By structurally decoupling singular values from the backward pass, SDS-LoRA ensures gradients are only applied through orthonormal bases, improving convergence and reducing the performance gap to full fine-tuning. Experimental results across natural language and vision benchmarks show enhanced adaptation performance.
PreLort: Prefix-Nested LoRA Enables Federated Fine-Tuning Across Heterogeneous Hardware Ranks
A new method called PreLort addresses the challenge of aggregating federated LoRA adapters with different ranks due to heterogeneous hardware. By organizing adapter dimensions into a prefix hierarchy and introducing segment-wise aggregation and prefix-nested training, PreLort consistently outperforms existing heterogeneous federated LoRA methods in accuracy and ROUGE-L while achieving lower perplexity.