Nvidia ships Nemotron 3 Ultra: a 550B open-weights Mamba-Transformer built for long-running agents
Nvidia released Nemotron 3 Ultra on June 4 as an OpenMDW-licensed 550-billion-parameter mixture-of-experts hybrid Mamba-Attention model with 55 billion active parameters per token, a 1-million-token context window, and weights live on Hugging Face, OpenRouter, Together AI, Perplexity and the build.nvidia.com NIM endpoint. The model hits 71.9 on SWE-Bench Verified, 94.7 on RULER at 1M tokens, and runs 5.9× faster than GLM-5.1, 4.8× faster than Kimi-K2.6 and 1.6× faster than Qwen-3.5 on GB200 at 8K input / 64K output under NVFP4. It is the first US open-weights release that closes the throughput gap with the Chinese frontier on long-context agentic decode.
Nvidia Research / MarkTechPost · 2026-06-04 · Source ↗ ·
Nvidia ships Nemotron 3 Ultra: a 550B open-weights Mamba-Transformer built for long-running agents
Nvidia released Nemotron 3 Ultra on June 4 as an OpenMDW-licensed 550-billion-parameter mixture-of-experts hybrid Mamba-Attention model with 55 billion active parameters per token, a 1-million-token context window, and weights live on Hugging Face, OpenRouter, Together AI, Perplexity and the build.nvidia.com NIM endpoint. The model hits 71.9 on SWE-Bench Verified, 94.7 on RULER at 1M tokens, and runs 5.9× faster than GLM-5.1, 4.8× faster than Kimi-K2.6 and 1.6× faster than Qwen-3.5 on GB200 at 8K input / 64K output under NVFP4. It is the first US open-weights release that closes the throughput gap with the Chinese frontier on long-context agentic decode.
Nvidia Research / MarkTechPost · 2026-06-04 · Source ↗ ·
Nvidia released Nemotron 3 Ultra on June 4 as the production-tier member of the Nemotron 3 family, the open-weights line the company first previewed at GTC in December 2025. The configuration is a 550-billion-parameter sparse mixture-of-experts with 55 billion active parameters per token, distributed across 108 layers with an 8,192 model dimension. Each layer carries 512 experts and routes the top 22 per token. Attention is asymmetric: 64 query heads and 2 key-value heads, a choice that shrinks the KV cache and matters when decode dominates an agent's runtime. The architecture is a hybrid Mamba-Attention design rather than a pure Transformer, which is the design choice the agent-workloads pitch rests on: Mamba's constant per-step decode cost scales linearly with sequence length, while attention's quadratic cost dominates long-running agents that emit tens of thousands of output tokens per task.
The model ships on Hugging Face in both BF16 and NVFP4 checkpoints, on OpenRouter, Together AI, Perplexity, Nebius Token Factory and Nvidia's own NIM endpoint at build.nvidia.com, and through the NeMo framework for self-hosting. The license is OpenMDW-1.1, an open-weights license that permits commercial use, modification and downstream training. The model operates at 5.03 bits per element in the NVFP4 deployment: NVFP4 experts, FP8 shared and Mamba layers, BF16 attention. On Blackwell the deployment is native FP4; on Hopper the model runs as W4A16 because Hopper lacks native FP4 tensor cores.
What the throughput numbers say
The benchmark Nvidia leads with is throughput on long-output workloads. At 8K input and 64K output on GB200 under NVFP4, Nemotron 3 Ultra runs 5.9 times faster than GLM-5.1-754B, 4.8 times faster than Kimi-K2.6-1T and 1.6 times faster than Qwen-3.5-397B. The comparison runs Nemotron on TRT-LLM and competitors on vLLM, which is the standard each model is tuned for, and which Nvidia discloses in the tech report. The 1.6× over Qwen 3.5 is the load-bearing number, because Qwen 3.5 was the previous open-weights speed leader and uses a comparable MoE structure. The argument is that the Mamba layers' constant decode cost is what carries the gap on long outputs, not the routing or the quantization.
On prefill-heavy work, defined as 50K input and 2K output, Nemotron trails Qwen. The Mamba advantage is concentrated where decode dominates, which is the agent and reasoning case. Cost-to-task-completion runs roughly 30 percent lower than the closest comparable on SWE-Bench Verified and Terminal Bench, because Nemotron uses fewer tokens per turn when reasoning effort is held constant.
"Open innovation is the foundation of AI progress. With Nemotron, we're transforming advanced AI into an open platform that gives developers the transparency and efficiency they need to build agentic systems at scale."— Jensen Huang, Nvidia CEO, at Nemotron 3 family preview
Benchmarks and where it lands
Nemotron 3 Ultra posts 71.9 on SWE-Bench Verified, 56.4 on Terminal Bench 2.1, 570 on IOI 2025 competitive programming (top-3 human level), and 94.7 on RULER at 1M tokens. On AA-Omniscience the model scores 78.7, the highest non-hallucination score in the comparison set Nvidia ran. On the held-out generalization gates Nvidia uses to detect benchmark leakage, PinchBench lands at 90.0 and ProfBench-Search at 56.0. Terminal Bench 2.1 is the one place Nemotron trails the open-weights field: Kimi-K2.6 leads at 67.2. Nvidia's framing is that the multi-harness training regime, in which agent runs were trained across five different frameworks, gives stable scores across deployment configurations (65 to 70.4 on SWE-Bench across frameworks) rather than peak single-framework numbers.
The model exposes three reasoning-effort modes: reasoning-off, regular, and medium-effort. Medium effort uses roughly 2.5 times fewer tokens than regular for a 7 percent accuracy drop. This is the budget control that matters in deployment: agent operators can tune the token spend per task to the value of the task, rather than picking a model per workload.
Training, data, and the divergences Nvidia disclosed
Pretraining ran across 20 trillion tokens in two phases, 15 trillion diversity-biased followed by 5 trillion quality-biased. The corpus includes a 173-billion-token refresh of GitHub code with a September 30, 2025 cutoff. Post-training combined supervised fine-tuning on 50 million cumulative examples, reinforcement learning across 2 million tasks and 55 environments, and a multi-teacher on-policy distillation step that condenses 10-plus specialized teacher models into the single Nemotron student. Datasets, training recipes and code accompany the weight release, an unusual level of disclosure for a frontier-tier open-weights model.
Nvidia disclosed two training loss divergences in the tech report. Near 8 trillion tokens, an output-layer gradient reduction shifted from FP32 to BF16, and BF16's 7 mantissa bits dropped the multi-token-prediction signal; the team reverted to FP32 and recovered. Near 16 trillion tokens, an unconfirmed root cause produced a second divergence, mitigated by early learning-rate annealing and a token-horizon reduction down to 20 trillion. The disclosure is unusual: most labs do not publish training failures.
Why it matters for the agent stack
The release is the first US open-weights model that closes the throughput gap with the Chinese frontier on the long-context decode-heavy regime where agent stacks live. Combined with the 1-million-token RULER score of 94.7, the long-context retrieval claim is at parity with closed-weights frontier models. The deployment story is the rest of the pitch: Nemotron 3 Ultra ships with a tuned TRT-LLM pipeline for Blackwell, a Hopper fallback in W4A16, the open weights for self-hosting, and a hosted NIM endpoint for teams that want a managed API. Together AI, Perplexity, OpenRouter and Nebius are the third-party inference providers carrying the model from day one, which means an agent operator can switch providers without re-engineering the prompt or the harness.
Cursor, Perplexity, Accenture, Cadence, CrowdStrike, Deloitte, EY, Oracle Cloud Infrastructure, Palantir, ServiceNow, Siemens, Synopsys and Zoom were named as Nemotron 3 family customers at the December preview. The Ultra release is the first version in the line targeted at the highest-accuracy tier; Nano (30B / 3B active) and Super (~100B / ~10B active) shipped earlier in the family rollout. The Ultra weights are the version that takes Nvidia from a credible open-weights contender to a credible open-weights leader, and the throughput numbers on decode-heavy workloads suggest the Mamba-Attention architectural bet was the right call for the agent workloads Nvidia is targeting with this generation.