distributed environments

Mosaic: Data-Free Knowledge Distillation Framework Uses Mixture-of-Experts to Tackle Heterogeneous Federated Learning

Researchers propose Mosaic, a novel data-free knowledge distillation framework that leverages Mixture-of-Experts (MoE) to overcome model and data heterogeneity in federated learning. Mosaic trains local generative models to synthesize data, forms an MoE from client models, and distills it into a global model. Experiments show consistent outperformance over state-of-the-art approaches on image and multimodal benchmarks.