KWBench 2026: The First Benchmark for AI’s Unprompted Problem Recognition in Knowledge Work

KWBench 2026: The First Benchmark for AI’s Unprompted Problem Recognition in Knowledge Work

KWBench, the first benchmark designed to measure unprompted problem recognition in large language models, has exposed a fundamental limitation in today’s frontier AI systems. Unlike traditional evaluations that test how well models execute predefined tasks, KWBench assesses whether an AI can autonomously identify the underlying structure of a complex professional scenario—before solving it. Developed by researchers and validated across six high-stakes domains including fraud analysis, contract negotiations, and clinical pharmacy, KWBench contains 223 tasks rooted in formal game-theoretic patterns such as principal-agent conflicts and strategic omission. Models receive raw data with no hints about the problem type, forcing them to recognize the context independently.

How KWBench Works: No Prompts, Just Raw Data

KWBench eliminates all task cues, presenting AI models with unstructured, real-world inputs like clinical records, legal contracts, or transaction logs. There are no labels, no instructions—only data. This forces models to perform autonomous context detection, mirroring how human experts operate in high-stakes knowledge work. The benchmark’s design is grounded in game theory, embedding 12 core problem patterns that reflect real cognitive challenges in decision-making.

Scoring uses a strict three-tier conjunctive rubric: models must correctly identify the problem type, map it to the right theoretical framework, and propose a valid solution. Failure at any stage results in a zero, ensuring that coincidental accuracy is never rewarded.

Why Traditional Benchmarks Fail AI in Real-World Contexts

Most AI evaluation tools—like MMLU or GSM8K—measure task completion, not problem framing. This creates a dangerous illusion of competence. A model might correctly answer a question about fraud detection, but if it never recognized the hidden principal-agent conflict driving the anomaly, its output is dangerously incomplete.

Studies show over 70% of AI failures in enterprise settings stem from misdiagnosed problems, not poor execution. KWBench reveals that LLMs excel at pattern matching but struggle with spontaneous problem discovery, a gap labeled "contextual blindness" in recent AI cognition research.

Performance Gaps Reveal AI’s Cognitive Blind Spots

The results are sobering. The best-performing model passed only 27.9% of tasks, while the top two models agreed on just 31.7% of their successful responses. Among the top eight models, 44 tasks were solved by exactly one model, demonstrating extreme inconsistency in problem identification.

Crucially, when models were later asked to articulate the relevant game-theoretic concept, they often succeeded—indicating they possess the knowledge but lack the ability to spontaneously apply it. This suggests a disconnect between stored knowledge and situational awareness, a critical flaw in real-world knowledge work where problems are rarely labeled.

Real-World Implications for AI Developers and Auditors

In fraud detection, clinical diagnostics, or compliance auditing, misidentifying the root problem can lead to catastrophic errors. KWBench reveals that current AI systems, despite their fluency, often operate as sophisticated pattern matchers rather than contextual analysts.

Organizations deploying AI for high-stakes decisions must now ask: Does the system understand the problem—or just the prompt? KWBench provides the first scalable metric to answer this. Its open release empowers developers to build autonomous AI that doesn’t just respond—but truly understands.

The Future of Autonomous AI Evaluation

KWBench is not just a benchmark—it’s a call to redefine success in AI. As the industry shifts from task accuracy to problem framing, KWBench sets the new standard for LLM evaluation frameworks. Future AI systems will be judged not by how well they solve what you tell them, but by whether they can find the problem you didn’t know you had.

Researchers have released KWBench as an open benchmark to shift the industry’s focus from task completion to problem framing. By measuring whether AI can recognize the right problem from raw inputs alone, KWBench sets a new standard for evaluating knowledge work competence.