NVIDIA to Pre-train Nemotron-3 Super/Ultra on FP4, Targets H1 2026 Release

In a groundbreaking revelation from a recent interview with AI2’s Nathan Lambert, NVIDIA’s VP of Applied Deep Learning Research, Bryan Catanzaro, disclosed that the company is pioneering the use of 4-bit floating-point (FP4) precision to pre-train its upcoming Nemotron-3 Super and Ultra large language models—a technique never before publicly attempted at scale. According to the interview published on Interconnects.ai, this move leverages NVIDIA’s proprietary hardware architecture to maximize throughput, despite the significant numerical stability challenges inherent in training state-of-the-art models with such extreme quantization. The models are expected to be released in the first half of 2026, marking a potential inflection point in efficient AI training methodologies.

While the timeline remains broad, the decision to adopt FP4 underscores NVIDIA’s aggressive pursuit of performance-per-watt leadership. Historically, training large language models has relied on FP16 or BF16 precision, with post-training quantization used to optimize inference. By contrast, FP4 pre-training would compress model weights and activations from the earliest stages, potentially reducing memory footprint and computational demands while maintaining competitive accuracy. If successful, this innovation could redefine how future foundation models are developed, especially for edge and on-prem deployment scenarios where resources are constrained.

Equally revealing was Catanzaro’s candid explanation of NVIDIA’s open-source strategy. When asked why the company invests in releasing open models like Nemotron, he bluntly stated the business rationale: "so people will keep buying NVIDIA GPUs." This transparent admission highlights a strategic pivot: rather than monetizing models directly, NVIDIA is betting on ecosystem lock-in. By providing high-performing, open-weight models optimized for its hardware, NVIDIA incentivizes enterprises and developers to deploy AI workloads on its GPUs—both in the cloud and on-premises. Analysts suggest this approach is a direct counter to competing cloud providers’ proprietary models, positioning NVIDIA as the indispensable infrastructure layer.

Perhaps the most unexpected insight came from Catanzaro’s repeated characterization of NVIDIA as a "company of volunteers." In response to questions about organizational coordination, he emphasized decentralization: "Everybody that works at NVIDIA is a volunteer. ... We let smart people figure out what they should be doing and then kind of self-organize." This philosophy, he explained, thrives in a fast-moving industry where talent can easily move between firms. By fostering intrinsic motivation and autonomy, NVIDIA cultivates innovation without top-down micromanagement. This cultural model, while unusual for a Fortune 500 tech giant, appears to be a key driver behind the rapid progress of projects like Nemotron.

The implications extend beyond hardware and software. If Nemotron-3 Super/Ultra delivers on its promise, it could accelerate the adoption of local AI models across industries—from healthcare diagnostics to financial compliance—where data privacy and latency are paramount. Furthermore, NVIDIA’s open model strategy may pressure competitors like Google, Meta, and Anthropic to follow suit, potentially shifting the AI landscape from closed, API-driven ecosystems toward more distributed, hardware-agnostic innovation.

As the industry watches for H1 2026, all eyes will be on whether FP4 pre-training can deliver on its theoretical promise—and whether NVIDIA’s volunteer-driven culture can sustain its momentum amid increasing global competition. For now, the message is clear: NVIDIA isn’t just selling chips. It’s selling a vision of AI built on open collaboration, hardware supremacy, and the quiet power of motivated individuals.