Weaviate Sphere Dataset Import: 1B Objects - 6 GKE Nodes

AI Impact Summary

Weaviate ingested approximately 1 billion Sphere dataset objects using a cluster of 6 Google Kubernetes Engine (GKE) nodes, each equipped with 961 GiB of RAM and 40 vCPUs. The import process, spanning 48 hours, demonstrated remarkably consistent performance, with import speed remaining largely linear regardless of the data volume already processed. This highlights the system's scalability and efficient use of resources, particularly the HNSW index which mitigates the logarithmic complexity of nearest neighbor searches.