Transformers v5 tokenization redesign separates tokenizer architecture from vocab and adds model-aware wrappers

AI Impact Summary

Transformers v5 introduces a major tokenizer redesign that decouples the tokenizer architecture from the trained vocabulary and exposes a Rust-based _tokenizer backend, enabling introspection and custom training with minimal friction. The change spans models and examples shown, including SmolLM3-3B, bert-base-uncased, google/gemma-3-270m-it, openai/gpt-oss-20b, and google-t5/t5-base, via the new AutoTokenizer wrapper and the apply_chat_template flow for chat-style prompts. This enables domain-specific tokenization and model-aware preprocessing, improving context efficiency and integration agility, but will require updates to downstream code to align with the new tokenizer API and model-specific preprocessing behavior.