Parquet Content-Defined Chunking enables dedup on Hugging Face Hub Xet storage

AI Impact Summary

Parquet CDC is now supported in PyArrow and Pandas, enabling content-defined deduplication when writing Parquet data to Hugging Face Hub's Xet storage. By setting use_content_defined_chunking=True (and using hf:// URIs), only changed chunks are uploaded or downloaded, cutting data transfer and storage costs for large Parquet datasets. This benefits workflows that frequently re-upload or modify datasets (even across repositories) and requires PyArrow 21+ to leverage the hf:// path API. Teams should validate integration in their data pipelines and ensure their tooling enables the CDC flag where applicable.