Improving Parquet Dedupe on Hugging Face Hub — offset dependency limits deduplication

AI Impact Summary

The Hugging Face Hub team is investigating Parquet deduplication challenges due to the format's reliance on absolute file offsets within column headers, which hinders efficient deduplication after modifications or deletions. This impacts storage efficiency for datasets stored as Parquet files, particularly those undergoing frequent updates, leading to increased storage needs and potential performance bottlenecks. Addressing this requires either a fundamental change to the Parquet file format or leveraging techniques like content-defined row groups to decouple row group boundaries.