Judge Arena launches LLM-as-Judge evaluation leaderboard

AI Impact Summary

Judge Arena introduces a crowdsourced evaluation framework where two LLMs act as judges, scoring and critiquing a sample, followed by human-facing votes to select the judge alignment. It surfaces an Elo-based leaderboard across 18 (listed) state-of-the-art models, enabling real-time benchmarking of evaluators. This shifts evaluation strategy from ad-hoc human judgment to a data-driven comparison of AI judges, with potential impact on which models are trusted for automated evaluation and which are chosen for how outputs are scored in production pipelines.