Hugging Face adds PyTorch / XLA TPU support for Cloud TPUs

AI Impact Summary

Hugging Face now enables training Transformers on Cloud TPUs through PyTorch / XLA, exposing the familiar Trainer API to TPU-backed runs. The integration leverages PyTorch / XLA’s XLA device (xm.xla_device), TPU core parallelism, and xm.optimizer_step for synchronized gradient updates across eight TPU cores per Cloud TPU device. It also covers TPU-specific data loading, checkpointing on CPU, and the lazy execution model of XLA graphs, which are critical for performance tuning and avoiding host-device stalls. This unlocks scalable, cost-efficient TPU training for teams already using the Hugging Face ecosystem.