Open-Weight LLMs Face System Design Challenge: Qwen 3, GLM-5, and Kimi k2.5 Tested

Open-Weight LLMs Face System Design Challenge: Qwen 3, GLM-5, and Kimi k2.5 Tested

A groundbreaking evaluation of open-weight large language models (LLMs) has revealed new insights into their capability to handle sophisticated system design tasks, challenging assumptions about the superiority of proprietary AI systems. Developer Ruhal Doshi recently launched hldbench.com, a community-driven platform that benchmarks models like Qwen 3, GLM-5, and Kimi k2.5 against real-world architectural challenges — including designing an enterprise-scale Retrieval-Augmented Generation (RAG) system comparable to Glean.

The initiative emerged after Doshi’s earlier benchmark, which focused exclusively on closed-source models like GPT-4 and Claude, faced criticism for excluding the rapidly evolving open-source ecosystem. In response, he rebuilt the test suite to include models with publicly available weights, enabling local deployment and transparent evaluation. The results, now publicly accessible and sortable by scoring criteria such as scalability, completeness, and error handling, show that open-weight models are not only competitive but in some cases outperform their proprietary counterparts on nuanced design problems.

Two core problems were presented to each model: the baseline task of designing a ChatGPT-like web application, and the significantly more complex challenge of architecting an enterprise RAG system — one that must handle multi-tenant data ingestion, semantic search at scale, query routing, caching, authentication, and real-time feedback loops. Models were evaluated on their ability to produce coherent, technically accurate system diagrams and accompanying documentation, with human reviewers scoring outputs against 12 predefined metrics.

Qwen 3, developed by Alibaba’s Tongyi Lab, demonstrated exceptional clarity in component decomposition, particularly in defining data pipelines and vector store integration. GLM-5, from Zhipu AI, showed superior handling of security and compliance considerations, explicitly addressing data sovereignty and audit trails — features often overlooked in other responses. Kimi k2.5, by Moonshot AI, impressed with its nuanced understanding of latency trade-offs and load-balancing strategies, even proposing hybrid cloud-edge architectures that were absent in many proprietary model outputs.

The platform’s open-source library, available on GitHub, is model-agnostic and compatible with OpenAI-compatible APIs, allowing developers to run evaluations locally using frameworks like Ollama or vLLM. Although Doshi has not yet fully validated local inference support, early community contributions have already begun submitting results from quantized versions of Llama 3 and Mistral, suggesting the benchmark could become a de facto standard for evaluating open LLMs in engineering contexts.

Unlike traditional benchmarks that focus on coding or reasoning accuracy, hldbench.com targets the high-stakes domain of system design — a skill critical for software architects and senior engineers. The community scoring feature allows users to rate submissions on granularity, innovation, and practicality, creating a dynamic leaderboard that reflects real-world engineering priorities rather than academic metrics.

This development comes amid growing industry pressure to reduce reliance on proprietary AI and increase transparency in model capabilities. While government agencies like Mexico’s Registro Agrario Nacional (RAN) continue to rely on legacy systems for legal land records, the AI community is rapidly building open infrastructure for the next generation of enterprise applications. As open-weight models become more capable, the line between commercial and community-driven innovation continues to blur — with hldbench.com serving as both a tool and a statement: open models are ready for the big leagues.

Doshi invites developers to contribute new test cases, such as distributed database design or microservice orchestration under failure conditions. "We’re not just measuring intelligence — we’re measuring engineering judgment," he said. "And that’s something no closed API can fully own."