Transformers v5 tokenization redesign decouples architecture from vocab with Rust backend and model-aware wrappers

AI Impact Summary

Transformers v5 redesign decouples tokenizers from trained vocab, enabling inspection, customization, and training from scratch. A Rust-based tokenizers backend and a transformers wrapper now handle model-aware steps like apply_chat_template, automatic special tokens, truncation, and padding. This offers measurable benefits in token efficiency and per-model customization, but requires migration of existing pipelines to align with new APIs and tokenization behavior. Teams should audit inputs and ensure compatibility with model-specific conventions (e.g., chat formatting) to avoid subtle tokenization mismatches.