Modal and Weaviate: Scalable Text Search at Scale

AI Impact Summary

Modal and Weaviate are being used to ingest and search ~50 million objects from Wikipedia, leveraging serverless infrastructure for scalability and GPU acceleration. The key optimizations – async indexing, product quantization, and tuned HNSW parameters – dramatically reduced the ingestion time from under two hours to under fifteen minutes for the entire Wikipedia dataset. This demonstrates a practical approach to building performant data ingestion pipelines at scale.