QIMMA: Quality-first Arabic LLM Leaderboard with rigorous validation

AI Impact Summary

QIMMA introduces a quality-validation pipeline that pre-screens Arabic benchmarks before model evaluation, revealing systemic quality issues that can bias rankings. By consolidating 109 subsets from 14 benchmarks into a 52,000-sample, seven-domain suite and applying multi-model automated assessment plus human review, it provides a more trustworthy basis for measuring Arabic LLM capability. The shift to rigorous quality validation—and modifications to Arabic prompts and evaluation workflows—could materially change how organizations compare Arabic models and select solutions, with LightEval, EvalPlus, and FannOrFlop as the evaluation backbone. Expect rankings to shift as data quality improvements propagate through the evaluation signals, and consider adopting or aligning with this validated suite for procurement and benchmarking.