Hugging Face migrates 6% of Hub downloads to Xet storage

AI Impact Summary

Hugging Face has migrated 6% of the Hub’s download traffic to Xet storage, validating the technology and integration points. This migration involved shifting repositories off LFS and to Xet, demonstrating a shift in storage strategy with significant performance improvements – specifically, a 10x reduction in database update times thanks to content-defined chunking. The team proactively addressed challenges like CAS download overhead due to inefficient data retrieval and pod load imbalance caused by un-flushed page caches, showcasing a robust operational response.