OpenAI TRL: New Vision Language Model Alignment Techniques

AI Impact Summary

OpenAI is introducing new alignment techniques for Vision Language Models (VLMs) within the TRL framework. These include Mixed Preference Optimization (MPO), Group Relative Policy Optimization (GRPO), and Group Sequence Policy Optimization (GSPO), designed to improve multimodal alignment and scaling with modern VLMs. The update introduces support for existing alignment methods like RLOO and Online DPO, enabling more efficient and scalable multimodal alignment. This represents a significant advancement in VLM training and performance, offering improved model responses and robustness.