Large-scale training of language models using agentic reinforcement learning (RL) presents a unique challenge: the workload alternates between compute-bound prefill, bandwidth-bound decoding, CPU-heavy environment execution, and bursty reward evaluation. Existing RL systems typically colocate all stages on a single GPU cluster or decouple them only coarsely, leading to hardware underutilization and synchronization overhead. According to a recent paper posted on arXiv, a team of researchers has developed RollArt, a disaggregated system for multi-task agentic RL that maps each pipeline stage to best-fit hardware, dramatically improving throughput and stability.
Disaggregated Architecture
RollArt decouples the RL pipeline at the trajectory level. Instead of synchronizing generation, environment interaction, and reward scoring, the system allows these stages to proceed independently. Specifically:
- Prefill-heavy tasks are routed to compute-optimized GPUs.
- Decode-heavy tasks are routed to bandwidth-optimized GPUs.
- Environment execution is offloaded to CPU clusters.
- Stateless reward computation is moved to serverless infrastructure.
This disaggregation ensures that slow or failed environment steps never block the other stages. The system also overlaps rollout with training through staleness-bounded asynchronous weight synchronization, eliminating idle time.
Measured Performance Gains
The researchers report that RollArt achieves a 1.31–2.05× reduction in training time compared to various RL systems. The system was evaluated by training a hundreds-of-billions-parameter mixture-of-experts (MoE) model for the Qoder product on an Alibaba cluster with above 3,000 GPUs. The deployment demonstrated both stability and scalability at production scale.
| Metric | Improvement |
|---|---|
| Training time reduction | 1.31–2.05× |
| GPU cluster size | >3,000 GPUs |
| Model type | MoE (hundreds-of-billions parameters) |
| Decoupling granularity | Trajectory level |
Enterprise Implications
For organizations training large-scale AI models, RollArt's disaggregated approach offers a blueprint for reducing infrastructure costs and time-to-deployment. By matching hardware to workload characteristics—compute GPUs for prefill, bandwidth GPUs for decoding, CPU clusters for environments—enterprises can maximize utilization of heterogeneous clusters. The asynchronous training scheme also reduces synchronization penalties, which is critical when training models with hundreds of billions of parameters.
The paper's validation on Alibaba's production environment with thousands of GPUs indicates that the approach is ready for large-scale deployment. While the system is currently demonstrated for agentic RL, the principles of workload disaggregation and asynchronous training could extend to other multi-phase ML workflows.