BigCodeArena enables end-to-end evaluation of code generation with sandboxed execution across 10 languages and 8 environments

AI Impact Summary

BigCodeArena now provides end-to-end evaluation of code-generation models by executing generated code in isolated sandboxes across 10 languages and 8 environments. This live-run capability surfaces real-world correctness and runtime behavior, addressing edge cases that static analysis or unit tests miss, and enables side-by-side comparisons with execution feedback. The platform uses Elo-based Bradley-Terry rankings with bootstrap confidence intervals to quantify model performance under language- and environment-matched settings, offering robust differentiation across frontier models (e.g., o3-mini, o1-mini, Claude-3.5-Sonnet) and GPT-like contenders. For engineering teams, this increases the reliability of benchmarking results and can accelerate model iteration and buy-in, but it also implies greater compute/resource needs and sandbox security considerations.