Topic
knowledge distillation
DiverseDistill: New Knowledge Distillation Method Recovers Over 70% of Performance Gap Using Teacher Committees
Researchers propose DiverseDistill, a knowledge distillation framework that combines a large foundation model with domain-specific experts as a diverse committee. The method recovers 73–114% of the teacher-student performance gap on recommendation and vision tasks while requiring no parameter updates or architectural changes.
Reinforcement-Aware Knowledge Distillation Boosts LLM Reasoning Efficiency
Researchers propose RL-aware distillation (RLAD) to address distribution mismatch and objective interference in knowledge distillation for LLM reasoning. The method uses Trust Region Ratio Distillation (TRRD) to selectively imitate teacher policies during reinforcement learning. RLAD outperforms offline distillation, standard GRPO, and KL-based on-policy distillation across logic and math benchmarks.
New Research Reveals Distinct Training Dynamics of On-Policy Distillation for Large Language Models
A research paper on arXiv characterizes the training dynamics of on-policy distillation (OPD) for large language models, finding that OPD occupies a distinct update geometry compared to supervised fine-tuning and reinforcement learning with verifiable rewards. The study shows OPD updates affect fewer weights, avoid principal directions, and exhibit subspace locking.
Mosaic: Data-Free Knowledge Distillation Framework Uses Mixture-of-Experts to Tackle Heterogeneous Federated Learning
Researchers propose Mosaic, a novel data-free knowledge distillation framework that leverages Mixture-of-Experts (MoE) to overcome model and data heterogeneity in federated learning. Mosaic trains local generative models to synthesize data, forms an MoE from client models, and distills it into a global model. Experiments show consistent outperformance over state-of-the-art approaches on image and multimodal benchmarks.
New Algorithm for Multi-Turn AI Agents Reduces Compounding Errors in Knowledge Distillation
A new algorithm called Guided On-Policy Distillation (Guided-OPD) addresses the failure mode where small student models compound errors in multi-turn tasks. By mixing teacher and student turns and using a curriculum that decays teacher intervention, the method improves average score by 21.1% and success rate by 25.5% over vanilla OPD.
Improved Knowledge Distillation Framework Achieves 99.04% Accuracy for Land-Use Classification
A research paper on arXiv presents an improved knowledge distillation framework for compressing deep neural networks used in land-use image classification. By integrating hard label supervision with soft losses (KL divergence and cosine similarity), the method achieves 99.04% accuracy on three land-use datasets, outperforming baseline and single-loss distillation approaches while substantially reducing model size.