Open-Source RL Libraries Explore Asynchronous Training

AI Impact Summary

Open-source RL libraries are increasingly adopting asynchronous training architectures to overcome bottlenecks in synchronous RL training, particularly with long rollouts from reasoning models. This approach, involving disaggregating inference and training across separate GPU pools and utilizing a rollout buffer, allows for concurrent execution and significantly improves GPU utilization. This survey of 16 open-source libraries highlights key design principles and trade-offs for building asynchronous RL training systems, offering valuable insights for optimizing model training workflows.