RTEB Beta: Retrieval Embedding Benchmark for generalization across open and private datasets

AI Impact Summary

RTEB beta introduces a hybrid benchmark to evaluate embedding-model retrieval accuracy using both open and private datasets, addressing the generalization gap seen with public-only evaluations. For enterprise use cases (RAG, agents, and recommendation systems), this provides a more realistic signal of how models perform on unseen data, since private datasets are evaluated by MTEB maintainers. Teams should adjust their evaluation pipelines to align with NDCG@10 as the leaderboard metric and plan governance around private-data evaluation, which may affect benchmarking timelines and reporting. This framework helps detect overfitting by comparing performance across open vs private data, guiding model selection and risk management for production retrieval systems.