Hugging Face Transformers 4.45.0 enables dynamic speculative decoding as default for assisted generation

AI Impact Summary

Dynamic speculative decoding in Hugging Face Transformers accelerates autoregressive generation by using a fast draft model to propose tokens and a larger target model to verify, enabling multiple tokens per forward pass. The feature is integrated as the default mode in Transformers 4.45.0 and supports an array of model pairs (OPT, Llama, Pythia, CodeGen, Flan-T5) with varying speedups up to 2.7x on some tasks. Because performance depends on model pair and workload, teams should expect latency improvements but may want to tune assistant_confidence_threshold and num_assistant_tokens to balance speed and accuracy. No code changes are required to enable it; you can adjust thresholds via generation_config for fine-grained control.