ARC-AGI2 Benchmark Under Scrutiny: Are AI Models Truly Reasoning—or Just Memorizing?

Despite record-breaking scores on the ARC-AGI2 benchmark, leading AI models from Google, Anthropic, and others may be achieving their results not through genuine intelligence, but by exploiting subtle patterns in the test’s structure—a discovery that has sparked a growing controversy within the AI research community.

Recent releases including Google’s Gemini 3.1 Pro (77.1%) and Gemini 3 Pro Deepthink (84%) have been heralded as breakthroughs in artificial general intelligence (AGI), with executives and researchers citing ARC-AGI2 as definitive proof of enhanced reasoning capabilities. Yet, a troubling anomaly has emerged: Claude Opus 4.5, which scored just 37% on the same benchmark, outperforms all these high-scoring models on SWE-Bench, a test measuring real-world software engineering problem-solving. This discrepancy suggests that ARC-AGI2 may not be measuring fluid intelligence as intended.

The root of the problem, according to independent researcher Mel Mitchell, lies in the benchmark’s susceptibility to format-based exploitation. In unpublished findings cited on X (formerly Twitter), Mitchell and colleagues found that altering the encoding of ARC-AGI2 problems—from numerical symbols to letters, shapes, or other visual representations—caused model accuracy to plummet dramatically. "If changing the font breaks the model, it doesn’t understand," Mitchell wrote. "It’s memorizing the format, not learning the reasoning."

This revelation has drawn parallels to a classic educational fraud: a student who excels on a math test only when questions are printed in red ink but fails entirely when switched to black. The student isn’t failing because they lack math skills—they’re failing because they never learned the underlying principles. Similarly, AI models trained on thousands of ARC-AGI2 examples may have internalized the visual and structural patterns of the test rather than the abstract logic it purports to measure.

François Chollet, the creator of ARC-AGI, designed the benchmark specifically to avoid such shortcuts. Unlike traditional datasets that rely on statistical correlations, ARC-AGI tasks require models to infer rules from minimal examples, mimicking human-like generalization. But as AI systems grow larger and more data-hungry, researchers have found that even carefully constructed benchmarks can be gamed through overfitting. The AI industry’s heavy reliance on ARC-AGI2 as a primary metric has inadvertently incentivized model developers to optimize for the test’s specific formatting quirks rather than robust cognitive abilities.

Google’s Demis Hassabis, CEO of DeepMind, publicly celebrated Gemini 3.1 Pro’s 77.1% score as "more than 2x the performance of 3 Pro," framing it as evidence of "core reasoning" improvements. Yet, if those improvements are contingent on the exact visual presentation of the problems, the achievement is methodologically hollow. The same models, when confronted with a slightly altered version of the same problem, collapse.

The implications extend beyond academic debate. If benchmarks are unreliable, then investment decisions, policy frameworks, and public perception of AI progress are being shaped by misleading metrics. The AI community is now calling for a new generation of dynamic, adversarial benchmarks that test robustness under perturbation—akin to stress-testing a bridge by varying load patterns, not just measuring its strength under ideal conditions.

Some researchers, including Yann LeCun and Ben Goertzel, have long argued that current AI systems are not on a path to true AGI without radical innovation in architecture and learning paradigms. The ARC-AGI2 controversy reinforces their skepticism. As one AI ethicist noted, "We’re building increasingly sophisticated illusions of intelligence, mistaking pattern recognition for understanding."

For now, the AI industry faces a reckoning: either reform its evaluation standards or risk building the next generation of AI on sand.