Topic
lora
FreeStyle: Scalable Style-Content Dual-Reference Generation via Community LoRA Mining
FreeStyle is a scalable dual-reference generation framework that leverages community LoRAs as compositional anchors for style and content. It introduces a two-stage curriculum with attention-level enrichment and frequency-aware RoPE modulation to suppress leakage from style references. The framework is evaluated on a new benchmark covering style similarity, content preservation, and leakage rejection, achieving a strong balance among these objectives.
Techniques for Peak Memory Reduction for LoRA Fine-tuning of LLMs on Edge Devices
A new paper introduces four complementary techniques to reduce peak memory during LoRA fine-tuning of large language models on edge devices. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to 26x and 28x memory reduction, respectively, without sacrificing model quality.
SDFLoRA: Selective Decoupled Federated LoRA for Privacy-Preserving Fine-Tuning with Heterogeneous Clients
Federated learning for LLMs faces challenges from heterogeneous client ranks and data distributions. SDFLoRA proposes a structure-aware LoRA framework that decouples updates into shared and private components, enabling stable aggregation, personalization, and improved differential privacy. Experiments show it outperforms existing federated LoRA baselines.
SDS-LoRA: New Low-Rank Adaptation Method Fixes Gradient Distortion in Large Model Fine-Tuning
A new paper on arXiv introduces SDS-LoRA, a low-rank parameterization that overcomes anisotropic gradient scaling in LoRA. By structurally decoupling singular values from the backward pass, SDS-LoRA ensures gradients are only applied through orthonormal bases, improving convergence and reducing the performance gap to full fine-tuning. Experimental results across natural language and vision benchmarks show enhanced adaptation performance.
PreLort: Prefix-Nested LoRA Enables Federated Fine-Tuning Across Heterogeneous Hardware Ranks
A new method called PreLort addresses the challenge of aggregating federated LoRA adapters with different ranks due to heterogeneous hardware. By organizing adapter dimensions into a prefix hierarchy and introducing segment-wise aggregation and prefix-nested training, PreLort consistently outperforms existing heterogeneous federated LoRA methods in accuracy and ROUGE-L while achieving lower perplexity.