Gemini 3.1 Pro Shows Major Spatial Reasoning Gains—But Hallucinations Raise Benchmark Questions

Google’s latest AI model, Gemini 3.1 Pro, has demonstrated a substantial leap in spatial reasoning capabilities compared to Gemini 3.0 Pro, according to an independent benchmark test conducted by researcher Ammaar Alam on the MineBench platform. The model produced JSON outputs up to 50MB in size—dramatically longer than those generated by its predecessor—indicating a marked increase in detail and structural complexity when constructing 3D Minecraft environments from textual prompts. Yet, these gains are shadowed by significant hallucinations, including the unauthorized use of in-game blocks not specified in the system prompt, raising critical questions about the reliability and fairness of current AI evaluation standards.

The benchmark, hosted at MineBench.ai, evaluates AI models on their ability to interpret and execute spatial construction tasks in Minecraft using a constrained block palette. In tests comparing Gemini 3.0 Pro and 3.1 Pro, the latter consistently generated more elaborate builds, with output lengths increasing by over 300% in some cases. According to Alam’s analysis, this expansion in output volume suggests improved contextual understanding and planning depth. However, the model frequently inserted blocks such as Spruce Planks and Redstone Dust into builds where they were explicitly excluded from the allowed palette—a behavior indicative of generative hallucination, a well-documented challenge in large language models.

Perhaps most concerning was the case of the "Knight in armor" build, where Gemini 3.1 Pro initially produced a structurally valid but visually sparse and low-detail output. After multiple retry cycles—each time adjusting to meet validation rules—the final result was deemed acceptable, yet lacked the artistic and architectural nuance expected from a high-performing model. "This raises serious questions about whether passing a benchmark should be based solely on technical compliance, or whether qualitative fidelity should be weighted more heavily," Alam wrote in a Reddit post. He is now seeking input from machine learning researchers to refine evaluation criteria, as current metrics may inadvertently reward models that generate excessive, noisy output rather than precise, thoughtful design.

While the MineBench benchmark has gained traction in AI communities for its focus on spatial reasoning—a domain where traditional NLP benchmarks often fall short—it remains a nascent tool. Unlike standardized tests such as MMLU or GSM8K, MineBench incorporates real-time validation of 3D structures, making it more reflective of embodied AI capabilities. Yet, as demonstrated by the Gemini 3.1 Pro results, the lack of standardized quality thresholds for "detail" or "creativity" leaves room for interpretation. Without clear guidelines, models may optimize for length and compliance over accuracy and intent.

Experts in AI ethics and evaluation warn that benchmarks like MineBench, while innovative, must evolve to account for both quantitative and qualitative dimensions of performance. "An AI that generates a 50MB JSON full of irrelevant blocks may technically satisfy validation rules, but it’s not demonstrating intelligence—it’s demonstrating noise," said Dr. Elena Rodriguez, an AI evaluation specialist at Stanford’s Center for Responsible AI. "We need metrics that penalize hallucination as heavily as they reward output volume."

Google has not yet responded to inquiries regarding the hallucination patterns observed in Gemini 3.1 Pro on MineBench. However, the company’s recent public release of the model emphasized its improved reasoning and long-context handling—features that align with the observed increases in output length. Whether these enhancements represent true cognitive progress or simply expanded generative capacity remains an open question.

As AI systems grow more capable, the burden of evaluation shifts from measuring what models can do, to understanding what they should do. The MineBench case underscores a broader trend: as benchmarks become more complex, so too must the criteria for judging success. Without rigorous, transparent, and human-aligned standards, even the most advanced models risk being misinterpreted as intelligent when they are merely verbose.