Hugging Face migrating Hub to Xet — 500K repos migrated

AI Impact Summary

Hugging Face is migrating the Hub's storage backend from Git LFS to Xet, a move that has already seen over 1 million users leveraging the new system. This migration, driven by the Hub's growth and the need for scalable AI builder storage, utilizes a complex architecture including a Git LFS Bridge, background content migrations, and a content addressed store (CAS) to ensure seamless transitions and minimal disruption to existing workflows. The team’s proactive design choices, like avoiding a hard cutover and supporting existing client integrations, have enabled this massive, near-silent migration of over 500,000 repositories and 20PB of data.