Topic
mixture-of-experts
Process-Verified Reinforcement Learning for Theorem Proving via Lean: A New Path to AI Reliability
A new arXiv preprint presents process-verified reinforcement learning for theorem proving, using the Lean proof assistant as a symbolic process oracle. By parsing proof attempts into tactic sequences and leveraging Lean's type-theoretic feedback, the method delivers dense, verifier-grounded credit signals. Experiments with STP-Lean and DeepSeek-Prover-V1.5 show tactic-level supervision outperforms outcome-only baselines on MiniF2F and ProofNet benchmarks.
New Research Proposes Adversarial Reweighting to Calibrate Mixture-of-Experts Models Under Distribution Shift
A new study examines how mixture-of-experts (MoE) models behave under distribution shift and proposes an adversarial reweighting approach to maintain calibration. The method improves the accuracy-calibration tradeoff across model classes, prediction tasks, and distribution shifts.
Mosaic: Data-Free Knowledge Distillation Framework Uses Mixture-of-Experts to Tackle Heterogeneous Federated Learning
Researchers propose Mosaic, a novel data-free knowledge distillation framework that leverages Mixture-of-Experts (MoE) to overcome model and data heterogeneity in federated learning. Mosaic trains local generative models to synthesize data, forms an MoE from client models, and distills it into a global model. Experiments show consistent outperformance over state-of-the-art approaches on image and multimodal benchmarks.
SPRI: SVD-Partitioned Residual Initialization Boosts Data-Constrained MoE Upcycling for Multilingual Translation
Researchers propose SPRI, a method that initializes Mixture-of-Experts (MoE) models from pretrained dense models using SVD-partitioned residuals. Evaluated on multilingual speech-to-text translation, SPRI achieves gains of 2.58 BLEU and 3.32 COMET over fine-tuned dense models, and outperforms prior MoE upcycling baselines by 3.39 BLEU and 4.34 COMET points.
Expert Tying Reduces Memory Footprint of Mixture-of-Experts LLMs by Nearly Half
A new arXiv paper from Jaggi proposes Expert Tying, an architectural modification for Mixture-of-Experts LLMs that shares expert parameters across consecutive transformer layers. Pretraining experiments show memory footprint reduction by almost 2x with virtually no degradation in perplexity or downstream quality, evaluated on OLMoE, Qwen3, and DeepSeek-style architectures.