From Chunks to Blocks: Accelerating Uploads and Downloads on the Hub

AI Impact Summary

OpenAI is introducing a new capability to accelerate uploads and downloads on the Hub by leveraging chunking and aggregation. This shift from a file-centric approach to a content-addressed store (CAS) with blocks and shards dramatically reduces redundancy and improves transfer speeds, potentially by a factor of 2-3x. This change is driven by the need to manage the massive scale of data on the Hub, particularly with model and dataset repositories, and addresses network and infrastructure overheads associated with traditional file transfers.