Hub migration from Git LFS to Xet storage with CAS-backed S3 and bridge-based compatibility

AI Impact Summary

Hugging Face is migrating the Hub from Git LFS to the Xet storage backend, a CAS-backed solution on S3 designed to scale with AI workloads (20 PB, well over a dozen PBs of models/datasets). The architecture uses a Git LFS Bridge and a background migration orchestrator to move LFS data to Xet without disrupting downloads or uploads, preserving workflows for non-Xet-aware clients. With Xet default for new users and ongoing migrations across 500k repos and 1M+ users, the system delivers higher sustained throughput (CAS peaks in the hundreds of Gb/s) while maintaining backward compatibility via bridges, CDN, and DynamoDB metadata.