Universal Assisted Generation: 1.5x-2.0x Faster Decoding with Any Assistant Model

AI Impact Summary

Intel Labs and Hugging Face have introduced Universal Assisted Generation (UAG), a technique that dramatically accelerates inference speed for any language model by leveraging a small assistant model. This allows for 1.5x-2.0x speedups across a wide range of models, including gemma-2-9b and Mixtral-8x22B-Instruct-v0.1, by translating token formats between the target and assistant models. This expands assisted generation beyond models with native small variants, unlocking performance gains for a broader set of LLMs.