MMLU-Pro: An Enhanced Benchmark Designed to Evaluate Language Understanding Models Across Broader and More Challenging Tasks

The new MMLU-Pro benchmark addresses several limitations of existing benchmarks like GLUE and SuperGLUE. Firstly, it incorporates more challenging, reasoning-intensive tasks, which helps to differentiate closely performing systems and better evaluates the advanced reasoning capabilities of state-of-the-art LLMs. Secondly, it increases the number of distractor options from three to nine, making the benchmark more robust and less sensitive to prompt variations. Lastly, MMLU-Pro involves a two-round expert review process to reduce dataset noise and enhance question quality, ensuring a broader and more discriminative evaluation across 14 diverse domains, encompassing over 12,000 questions.

MMLU-Pro enhances the challenge and complexity of tasks compared to its predecessor, MMLU, in several ways:

1. Increased Answer Options: MMLU-Pro significantly increases the number of answer options from four to ten per question1. This enhancement makes the evaluation more realistic and challenging, as it reduces the score from random guessing and necessitates more discerning reasoning for correct selection.

2. Higher Difficulty: The new dataset includes more reasoning-focused problems, increasing the overall difficulty. As a result, models utilizing Chain-of-Thought (CoT) reasoning can outperform Perplexity (PPL) by up to 20%.

3. Performance Stability: Due to the increased options, model performance on MMLU-Pro is more stable. For instance, Llama-2-7B shows less than 1% performance variance with different prompts, compared to 4-5% on the original MMLU.

4. More Robust and Challenging Questions: MMLU-Pro incorporates more complex, reasoning-intensive tasks, increasing the number of distractor options from three to nine1. This feature provides a broader and more discriminative evaluation, making the benchmark better suited for assessing the advanced reasoning capabilities of state-of-the-art LLMs.

5. Rigorous Construction Process: MMLU-Pro's dataset construction involves integrating questions from various high-quality sources and undergoes a rigorous process of filtering and refining, removing overly simple or erroneous questions. This process ensures a diverse and challenging question set for evaluation.

In conclusion, MMLU-Pro's enhancements in terms of answer options, difficulty, performance stability, and construction process contribute to a more challenging and robust benchmark for evaluating the capabilities of large language models.