Weaviate v1.18: Introducing HNSW+PQ Vector Compression

AI Impact Summary

Weaviate is introducing HNSW+PQ vector compression, leveraging Product Quantization (PQ) to reduce memory footprint and cost. This involves moving vector and graph representations to disk, fetching compressed vectors from disk during searches and only loading full vectors when needed. This approach addresses the scalability challenges of storing large vector datasets, particularly when dealing with datasets like Gist or DeepImage96, by significantly reducing memory requirements and minimizing disk read latency.