Parquet CDC enabled on Hugging Face Xet storage via PyArrow and Pandas

AI Impact Summary

Parquet Content-Defined Chunking (CDC) is now supported in PyArrow and Pandas, enabling the Hugging Face Hub's Xet storage layer to deduplicate data at the chunk level. This allows uploading or downloading only the parts of Parquet files that have changed, dramatically reducing data transfer and storage costs for large datasets and improving update speeds. Activation is done via use_content_defined_chunking (e.g., df.to_parquet(..., use_content_defined_chunking=True)), with cross-repo deduplication enhancing efficiency in collaborative environments.