RTEB beta: New Retrieval Embedding Benchmark for real-world generalization

AI Impact Summary

RTEB beta introduces a hybrid open/private dataset benchmark to measure true generalization of embedding models, addressing the generalization gap seen with public-only evaluations. It aims to reveal overfitting when models memorize test data, by comparing performance on private datasets evaluated by MTEB maintainers. With enterprise-focused domains (law, healthcare, finance, code) and multilingual support, RTEB aligns evaluation with production retrieval workloads used in RAG, agents, and recommender systems, helping teams select models that generalize to unseen data. Expect adoption to become a new validation gate, influencing model selection and benchmarking practices for retrieval-based applications.