A New Benchmark: LLM Performance on Age of Empires 2 Build Orders

In 2026, AI researchers developed the first specialized benchmark system to measure the real-time decision-making capabilities of large language models (LLMs) in strategic games. This system is based on build orders played in Age of Empires II: Definitive Edition. The goal is to evaluate whether AI models can understand and apply not only text-based knowledge but also timing, resource management strategies, and dynamic combat scenarios within a realistic game environment.

Build Orders: The Testing Ground for AI’s Strategic Intelligence

Age of Empires II is a complex strategy game featuring over 10,000 distinct build orders and thousands of strategic variables. These orders determine when a player constructs specific buildings and units, which technologies to prioritize, and when to launch attacks against opponents. Researchers challenged state-of-the-art LLMs—including GPT-4o, Claude 3.5, Grok-2, and LocalLLaMA—to accurately generate complete build orders and optimize them according to in-game scenarios.

In the tests, models were expected not only to list correct structures but also to dynamically adjust their build orders based on the game’s developmental stage (e.g., the Feudal Age). For instance, it was insufficient for a model to simply issue the instruction: “Transition to 12 villagers and produce 2 horse archers in the Feudal Age”; it also needed to propose an alternative strategy in case of resource depletion.

Performance Results: GPT-4o Emerges as the Leader

As of February 2026, in 1,200 test scenarios, GPT-4o achieved the highest accuracy rate at 87.3%. Claude 3.5 scored 81.1%, while Grok-2 reached 76.4%. However, all models made errors in high-level strategic decisions such as “resource optimization” and “responding to opponent movements.” In particular, models that miscalculated timing during transitions from the “Dark Age” to the “Castle Age” significantly increased their likelihood of losing the game within 15 minutes.

An interesting finding was that open-source models like LocalLLaMA, despite their smaller parameter sizes, achieved performance close to GPT-4o on specific build orders. This demonstrates that success can be achieved through high-quality data and training restricted to specialized strategic game datasets.

Future: The Boundary Between AI and Strategic Games

This benchmark offers a critical model not only for Age of Empires II but also for developing future autonomous systems—such as logistics, military strategy simulations, or automated economic decision-making systems. Researchers plan to extend this methodology to other strategy games like StarCraft II and Civilization VI.

AI’s ability to think like a human in games is not merely about entertainment—it also advances our understanding of real-world decision-making processes. This study demonstrates that AI’s capabilities extend beyond language processing to include measurable spatial, temporal, and resource-based reasoning.

• Build orders are among the most realistic methods for measuring AI’s strategic thinking capacity.
• GPT-4o emerged as the top-performing model in 2026.
• Open-source models can compete through data quality.
• This approach can be applied to future automated logistics and military simulations.