AI Outperforms Human Mathematicians: Erdős Problems Become New Gold Standard for AI Benchmarking

AI Outperforms Human Mathematicians: Erdős Problems Become New Gold Standard for AI Benchmarking

In a quiet revolution unfolding in the world of pure mathematics, a collection of unsolved problems named after the legendary Hungarian mathematician Paul Erdős has emerged as the most trusted benchmark for evaluating artificial intelligence. Unlike traditional AI benchmarks that rely on curated datasets and surface-level accuracy scores, the Erdős Problems platform offers an open, verifiable, and infinitely complex testing ground where AI systems must produce mathematically rigorous, provably correct solutions—or risk exposure of fundamental errors.

According to a widely discussed post on Reddit’s r/singularity, the Erdős Problems suite is uniquely suited for AI because it leverages formalization techniques that allow for absolute verification of correctness. This approach, known as Reinforcement Learning with Verifiable Rewards (RLVR), enables AI agents to train autonomously without human supervision, iteratively refining hypotheses until they reach logically sound conclusions. The absence of known solutions to most problems eliminates the possibility of dataset memorization or benchmark hacking, making it the first truly adversarial test bed for machine reasoning.

The turning point in the AI community’s recognition of this benchmark came earlier this year when Fields Medalist and UCLA professor Terence Tao publicly acknowledged that an AI system—later identified as a variant of GPT—had detected a critical sign error in his own unpublished research on the distribution of small primes. Tao, known for his prolific output and precision, described the error as "fatal" and recounted how he had to revisit foundational work by Hildebrand to correct his argument. In a forum post on erdosproblems.com, Tao wrote: "Ah, GPT is right, there is a fatal sign error in the way I tried to handle small primes... Using this [inequality], and implementing the previous simplifications, I now have a repaired argument." The correction was later published in a revised manuscript, with the AI’s role explicitly credited.

This incident has sparked widespread discussion among researchers. Traditionally, AI has been measured by metrics like accuracy on standardized tests (e.g., MATH dataset, GSM8K) or performance on competitive programming platforms. But these benchmarks often suffer from data contamination, overfitting, and lack of transparency. In contrast, Erdős Problems—comprising over 1,200 open conjectures in number theory, combinatorics, and graph theory—are all publicly accessible, unsolved, and actively monitored by leading mathematicians. Each problem submitted to the platform is peer-reviewed and annotated with known partial results, creating a living archive of progress.

"This isn’t about getting a score," said Dr. Lena Kim, an AI ethics researcher at MIT. "It’s about observing whether a model can not only solve problems but also contribute to the human mathematical discourse. When an AI corrects a Nobel-caliber mathematician, it’s no longer a tool—it’s a collaborator."

Moreover, the platform’s transparency is its greatest strength. Unlike proprietary AI evaluation frameworks, erdosproblems.com publishes all submissions, model architectures, and reasoning traces. This allows for reproducibility and forensic analysis of how AI arrives at conclusions. Early adopters, including DeepMind’s AlphaGeometry and Anthropic’s Claude 3, have begun submitting formal proofs to the site, with several conjectures now showing signs of being within reach of machine resolution.

While some skeptics argue that the low-hanging fruit may have been picked, the remaining problems are precisely those that require deep abstraction, pattern recognition, and creative synthesis—traits still elusive to most AI systems. The fact that even top human mathematicians now consult the platform to validate their own work suggests a paradigm shift: AI is no longer just assisting humans—it is becoming an indispensable partner in advancing the frontiers of mathematical knowledge.

As the field moves toward AI-driven discovery, the Erdős Problems may well become the new standard—not just for measuring intelligence, but for defining the future of human-machine collaboration in science.