NVIDIA B200 Emerges as Clear Leader in AI Inference Performance and Efficiency

An independent benchmark reveals significant generational leaps in datacenter GPU performance, with cost-per-token becoming a key metric for enterprise AI deployment.

In a detailed performance analysis that has captured the attention of the AI development community, NVIDIA's flagship Blackwell B200 GPU has demonstrated a commanding lead in large language model (LLM) inference, outperforming its predecessor H100 by substantial margins and redefining the economics of AI service deployment. The benchmark, conducted by Cloudrift AI and shared on the r/LocalLLaMA subreddit, provides a rare apples-to-apples comparison across four GPU architectures: the consumer-focused RTX Pro 6000 SE, the H100, the H200, and the new B200.

The Benchmarking Methodology: Beyond Raw Speed

The study employed a sophisticated methodology designed to reflect real-world deployment scenarios. Using the vLLM inference serving framework, researchers tested three distinct models with varying memory and computational requirements to isolate the impact of inter-GPU communication—a critical bottleneck in distributed AI inference.

According to established principles of technical benchmarking, which involve comparing processes and performance metrics to industry bests, the test was structured to evaluate not just peak throughput but also operational economics. The models selected were:

• GLM-4.5-Air-AWQ (4-bit): A model fitting within a single 80GB GPU, testing raw single-GPU performance and replica scaling.
• Qwen3-Coder-480B-AWQ (4-bit): A 480-billion-parameter model requiring 4 GPUs, introducing moderate inter-GPU communication overhead.
• GLM-4.6-FP8: A massive model requiring all 8 GPUs in the test systems, maximizing communication bottlenecks.

The benchmark was optimized for throughput, saturating the GPUs with 64-256 concurrent requests and using an NGINX load balancer to distribute traffic across multiple vLLM instances. Crucially, the analysis incorporated the real estimated cost of ownership per GPU hour—$0.93 for Pro 6000, $1.91 for H100, $2.06 for H200, and $2.68 for B200—to calculate a definitive metric for AI service providers: cost per million tokens generated.

Results: B200's Decisive Victory and the NVLink Advantage

The results paint a clear picture of architectural evolution. On the most communication-intensive workload—the 8-GPU GLM-4.6-FP8 model—the B200 achieved a staggering 8,037 tokens per second, which is 4.87 times faster than the RTX Pro 6000 SE cluster (1,652 tok/s). This massive gap underscores the importance of NVLink, NVIDIA's high-speed GPU interconnect, which is present on the H100, H200, and B200 but absent on the Pro 6000. The Pro 6000 systems relied solely on the slower PCIe bus for GPU-to-GPU communication, a significant handicap for large, sharded models.

The B200 also led on cost efficiency. Despite its higher hourly run cost, its vastly superior throughput made it the cheapest option per million tokens generated across all tested models. This turns the conventional wisdom of "cheaper hardware equals lower cost" on its head for high-utilization AI inference services.

The Surprise Contender and Generational Shifts

The RTX Pro 6000 SE, based on the same Blackwell architecture as the B200 but with slower GDDR memory and no NVLink, emerged as a compelling option for capital-expenditure-sensitive deployments. It beat the H100 on cost-per-token across all models and was competitive with the H200 on the single-GPU workload. This suggests that for smaller models or workloads that don't require heavy multi-GPU communication, the Pro 6000 offers remarkable value.

The H200, featuring high-bandwidth memory (HBM), showed a major step up from the H100, delivering 1.83x to 2.14x the throughput. This highlights the critical role of memory bandwidth in feeding the massive computational engines of modern AI accelerators.

Implications for the AI Industry

This benchmark, following the standard process of identifying, understanding, and adapting outstanding practices from other organizations, provides critical data for AI infrastructure decisions. For large-scale service providers where throughput directly translates to revenue, the B200's performance justifies its premium. For research labs, startups, or applications running smaller models, the RTX Pro 6000 SE presents a lower barrier to entry with modern architecture benefits.

The poor showing of the H100 in this specific setup—being matched by the Pro 6000 on a single-GPU task—indicates how quickly the landscape is evolving. It reinforces that benchmarking is not a one-time activity but a continuous process of measurement and comparison to drive improvement.

As AI models continue to grow in size and complexity, the infrastructure that powers them becomes increasingly stratified. This study provides a vital roadmap, showing that the choice of GPU is no longer just about flops or memory size, but about the holistic system architecture—interconnects, memory bandwidth, and software optimization—that determines real-world efficiency and cost.

The full methodology, code, and results are available on GitHub, allowing for independent verification and further analysis by the community.