Benchmarking the Benchmarks: Audit Reveals 12% Error Rate in GPQA and MMLU-Pro, Prompting Release of ‘Clean’ Datasets
Core Event Summary
A rigorous expert audit of industry-standard benchmarks—GPQA-Diamond, MMLU-Pro, and MMMU-Pro—has uncovered that up to 12% of questions are fundamentally broken due to formatting issues, incorrect ground truths, or multiple valid answers. The researchers have subsequently released “Clean” versions of these datasets to provide a more accurate ceiling for frontier LLM performance.
- ▶ The Artificial Ceiling: The perceived stagnation of LLM performance on complex reasoning tasks is partially an artifact of benchmark noise rather than a plateau in machine intelligence.
- ▶ Reliability Crisis: The high error rate in MMLU-Pro suggests that current leaderboards may be misrepresenting the true delta between top-tier models.
- ▶ Shift to Precision Eval: The industry is moving from “Scale-first” to “Quality-first” evaluation, where the integrity of the test set is as critical as the model parameters.
Bagua Insight
For too long, the AI community has treated benchmarks as absolute ground truth. This audit exposes the “dirty secret” of GenAI evaluation: as models become more sophisticated, they begin to outsmart the very tests designed to measure them, often getting penalized for identifying ambiguity or errors in the questions. At Bagua Intelligence, we view this as a pivotal moment. If a benchmark has a 12% inherent error rate, any model scoring above 88% is essentially hallucinating or over-fitting to noise. We are entering the “Precision Era” of evaluation. The bottleneck for proving AGI-level reasoning is no longer just compute or data—it’s the scarcity of flawless, expert-verified evaluation rubrics. If your model’s GPQA score has plateaued, it might not be a lack of reasoning power; it might just be that the model is too smart for a broken test.
Actionable Advice
- Update Pipelines: Engineering teams should immediately integrate the “Clean” versions of GPQA and MMLU-Pro into their CI/CD pipelines for a more realistic performance baseline.
- Re-evaluate SOTA Claims: Take marginal gains on standard benchmarks with a grain of salt unless they are validated against these audited sets.
- Internal Audit: Apply similar auditing rigor to proprietary RAG evaluation sets to ensure that “data rot” isn’t skewing your internal product roadmap.