Hugging Face Hub expands dataset hosting to 500 GB per-file, streaming, and in-browser tools

AI Impact Summary

Hugging Face Hub is expanding capabilities for hosting and sharing open ML datasets, including terabyte-scale hosting and per-file limits increasing from 50 GB to 500 GB via a backend update. The Datasets library enables easy upload/download and streaming so large datasets can be used without full downloads, while the Dataset Viewer and SQL Console allow in-browser exploration and querying. Built-in access controls and security scanners (malware, secrets, pickle, ProtectAI) help govern shared data, improving governance for enterprises and government partners and supporting broader reproducibility and collaboration.