TRL: Co-located vLLM for Efficient GRPO Training

AI Impact Summary

TRL has introduced a significant improvement in efficiency by co-locating vLLM with the training process. Previously, vLLM ran as a separate server, leading to significant GPU idle time and reduced throughput, particularly in online learning setups like GRPO. This new co-located approach allows the training and generation tasks to share the same GPUs, dramatically reducing idle time, optimizing GPU utilization, and ultimately boosting overall performance and reducing costs. This change unlocks the full potential of vLLM within the TRL framework.