NVIDIA Nemotron 3.5 Lighting is available on Ollama

Most of what your coding agent does all day is boring.

It reads a file. It runs git status. It greps a directory. It parses a JSON blob, decides the schema is wrong, and retries. It calls a tool, gets a 500, and calls it again. Somewhere in the middle of those forty steps, maybe two of them require actual reasoning — the plan at the start, and the tricky decision when something breaks.

And yet every single one of those steps goes to a frontier model. You pay frontier prices for ls. You pay frontier latency for a retry.

NVIDIA released Nemotron 3.5 Lightning on August 11, 2026, and it is aimed squarely at that problem. It landed on Ollama the same day, which means you can run the thing on your own hardware in about the time it takes to make coffee.

I’ve spent enough time debugging local agent setups to be skeptical of launch-day benchmark charts. So this post goes past the announcement: what the architecture actually is, why it’s shaped the way it is, how to run it properly on Ollama, and where the numbers get uncomfortable.

The pitch in one paragraph

Nemotron 3.5 Lightning is a 30-billion-parameter Mixture-of-Experts model with 3 billion active parameters per token. It’s distilled from NVIDIA’s frontier Nemotron 3 Ultra, supports up to a 1M-token context window, and is released under OpenMDW-1.1 with weights, training data, and recipes published. NVIDIA claims up to 4x higher output speed than comparable open models, and 30% faster completion of 10,000 PinchBench tasks than Qwen3.6 35B at similar accuracy.

The framing NVIDIA uses is “the execution layer of always-on agents.” That framing matters more than the benchmark numbers, and I’ll come back to it.

Part 1: The architecture is more interesting than the parameter count

Everyone’s headline is “30B total, 3B active.” Fine. That’s standard MoE arithmetic — a router picks a handful of experts per token, so you get the knowledge capacity of a large model at the compute cost of a small one.

The genuinely interesting part is buried in the model card:

Architecture: MoE — Mamba-2 + MoE + Attention hybrid

This is not a plain transformer with expert layers bolted on. It’s the Nemotron-H lineage: interleaved Mamba-2 state-space layers and MoE layers, with a small number of attention layers mixed in.

Why hybrid, and why it matters for agents specifically

A standard transformer’s KV cache grows linearly with sequence length. Every token you’ve seen has to stay in memory, forever, for every attention layer. At 200k tokens of agent scrollback, the KV cache stops being a footnote in your VRAM budget and starts being the main event.

Mamba-2 layers are state-space models. They carry a fixed-size recurrent state regardless of how long the sequence gets. Ten tokens or a million, the state is the same size.

Now think about what a long-running agent actually looks like in memory. It’s a giant, mostly-append-only transcript: tool call, tool result, tool call, tool result, for hours. That’s exactly the workload where a growing KV cache kills you, and exactly the workload where a constant-size recurrent state wins.

The handful of attention layers are there because pure SSMs are weak at precise recall — “what was the exact function signature in the file I read 30 steps ago.” Attention handles the sharp lookups; Mamba handles the bulk of the sequence cheaply. That’s the trade, and it’s why “1M context” is even a plausible claim for a model this small.

You can see the architecture leaking into the serving flags. From NVIDIA’s vLLM recipe:

shell

--mamba-backend flashinfer \
--mamba-cache-mode align \
--mamba-ssm-cache-dtype float16 \
--enable-mamba-cache-stochastic-rounding \
--mamba-cache-philox-rounds 5

There’s a whole second cache subsystem next to the KV cache, with its own dtype and its own numerical stability tricks. Stochastic rounding on the SSM cache is there because recurrent state accumulates error over long sequences — round it naively in FP16 and drift compounds over thousands of steps.

That’s an unusual amount of engineering to spend on “runs fast on a laptop.” It’s engineering spent on “stays coherent after four hours of tool calls.”

Training shape

Five stages, per the model card:

1. Pre-training — 20T+ tokens, using an NVFP4 recipe, via Megatron-LM. Pre-training data cutoff September 2025.
2. Continued pre-training for MTP — a dedicated phase to train Multi-Token Prediction heads.
3. SFT — synthetic code, math, science, tool calling, instruction following, structured outputs, plus long-range retrieval and multi-document aggregation data.
4. RL — multi-environment GRPO across math, code, science, instruction following, multi-step tool use, multi-turn conversation, and structured output environments. Notably, the RL architecture is asynchronous and uses MTP to accelerate rollout generation — they used the speed feature to train the model faster.
5. PTQ — NVFP4 post-training quantization, W4A16 on routed and shared experts, FP8 per-tensor dynamic scales on the Mamba in_proj/out_proj and the KV cache.

Stage 4 is where the “harness-optimized” claim comes from. This model was trained in agent environments, not just on transcripts of them.

Part 2: Three flavours of speculative decoding

The “4x throughput” number does not come from the MoE architecture alone. It comes from speculative decoding, and Lightning ships with three different approaches:

A block-diffusion draft model is a genuinely novel choice. Diffusion generates all positions in parallel and refines, which is a natural fit for drafting: you don’t need the draft to be autoregressively correct, you just need it to be close enough that the target model accepts most of it.

Thoughtworks, as an early access partner, ran 2,091 measurements across two inference engines, two GPU generations, three workloads, and concurrency from 1 to 128. Their finding: the built-in MTP head delivered 1.46–1.96x the throughput of unaccelerated decoding with equivalent task accuracy — and a purpose-trained EAGLE-3 draft head could only match it, not beat it.

That’s a useful reality check on the 4x figure. 4x is a ceiling under favourable conditions, not a number you should expect on your desk.

And here’s the caveat that matters most for this post: MTP, DSpark, and DFlash are features of the NVIDIA serving stack — vLLM, TensorRT-LLM, SGLang. Ollama’s GGUF path is a different engine. Do not assume you inherit the full speculative decoding stack by running ollama pull. Measure it yourself before you quote the number in a design doc.

Part 3: Running it on Ollama, properly

The basic command is the one from the announcement:

bash

ollama run nemotron-3.5-lightning

Here’s what’s actually in the library:

Tagged tools and thinking — so tool calling and a reasoning mode are both wired up.

The context length trap

This is the single most common reason local agent setups fail silently, and it will bite you here.

Ollama defaults models to a small context window regardless of what the model supports. A model card that says “1M tokens” and an Ollama runtime that gives you 4,096 are two different things. Your coding agent will load a repo, blow past the window, and start behaving like it has amnesia — tool calls half-forming, the agent re-reading files it already read.

Ollama’s own docs are explicit that coding tools want at least 64,000 tokens. Set it:

bash

# Per-session, in the REPL
ollama run nemotron-3.5-lightning
>>> /set parameter num_ctx 65536

# Or globally on the server
OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# Or per-request via the API
curl http://localhost:11434/api/chat -d '{
  "model": "nemotron-3.5-lightning",
  "messages": [{"role": "user", "content": "hello"}],
  "options": { "num_ctx": 65536 }
}'

Sampling parameters — don’t use your usual defaults

NVIDIA recommends temperature 1.0, top_p 0.95.

If you’re used to dialling temperature down to 0.2 for “deterministic” agent behaviour, resist the reflex here. Models post-trained with RL are calibrated at a specific sampling regime, and cranking temperature down on a model tuned for 1.0 tends to produce degenerate loops — the same tool call, over and over. If your agent gets stuck in a repeat cycle, check your temperature before you blame the model.

Hardware reality

The 25GB quantized weights are the floor, not the total:

• RTX 5090 (32GB) — fits, with headroom for a decent context. This is the sweet spot on consumer hardware.
• DGX Spark (GB10, 128GB unified) — the target platform. NVIDIA’s DSpark drafter is named after it. Long context is actually usable here.
• 24GB cards (4090, 3090) — you’re spilling to system RAM. The MoE design softens the blow versus a dense 30B (only 3B activate per token, so less weight traffic per step), but you’re still bandwidth-bound and it will feel it.
• 16GB and below — technically runnable, practically painful for agent loops.
• Apple silicon — the 30b-mlx build at 23GB. Realistically wants 32GB+ unified memory, 64GB to be comfortable.

And to be blunt about the headline spec: you are not running 1M context locally. The Mamba layers keep constant state, but the attention layers still cache, and the activations still need somewhere to live. On a 32GB card, plan for 64k–128k and be happy about it. 1M is a datacenter number.

Part 4: Wiring it into a harness

This is where Ollama has quietly gotten very good. ollama launch configures the agent CLI for you — no ANTHROPIC_BASE_URL exports, no config file archaeology:

bash

ollama launch claude    --model nemotron-3.5-lightning
ollama launch opencode  --model nemotron-3.5-lightning
ollama launch openclaw  --model nemotron-3.5-lightning
ollama launch hermes    --model nemotron-3.5-lightning

Under the hood this leans on Ollama’s Anthropic-compatible API. The manual path still works if you need it in CI:

bash

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434
claude --model nemotron-3.5-lightning

One detail worth knowing when tool calls misbehave

NVIDIA’s serving recipes launch with --tool-call-parser qwen3_coder and --reasoning-parser nemotron_v3.

That first flag tells you something useful: the model emits tool calls in Qwen3-Coder’s format, not the OpenAI JSON convention. If you’re building a custom harness and your tool calls are arriving as unparsed text in the content field, that’s your answer — you need the matching parser, not a different prompt.

There’s also a documented quirk for coding agents on the NVIDIA stack: pass force_nonempty_content: True in the chat template kwargs. Some harnesses choke on an assistant message with tool calls and empty content.

Part 5: Read the benchmarks honestly

Here are NVIDIA’s own numbers, BF16 versus the NVFP4 checkpoint:

Four things I’d flag:

1. SWE-bench Verified at ~52% for a 3B-active model is genuinely strong. That’s patch generation on real GitHub issues. For a model you can run on a gaming GPU, that’s a real number.

2. Terminal-Bench 2.1 at 24.58 is a very different story. Good at “here’s a bug, produce a patch.” Much weaker at open-ended terminal work where the agent has to decide what to even do. That gap is the honest boundary of this model: give it bounded tasks, not open ones.

3. τ³-bench Banking at 9.28 deserves a hard look, given that financial services workflows are on the marketing list of use cases. τ-bench-family evals measure multi-turn, policy-constrained customer interaction. Single digits means: do not put this in front of customers on a regulated workflow without a supervising model and hard guardrails. The use case list describes what you could build; this number describes what you’d have to build around it.

4. The knowledge scores are low by design. AA-Omniscience 17.50 and HLE 11.72 tell you this is not a model to ask factual questions. 3B active parameters cannot store the world. Pair it with retrieval and tools, always — which, to be fair, is exactly the deployment pattern NVIDIA is pitching.

On quantization: NVFP4 versus BF16 is mostly within noise, but not uniformly. Long context (AA-LCR: 52.00 → 49.19) and multilingual SWE-bench (39.33 → 36.47) degrade most. That’s the pattern you’d expect — quantization error compounds over long sequences and on lower-resource distributions. If your workload is long-context, budget for the delta.

Also note NVIDIA’s own caveat: these were measured in-house under their harness (NeMo Gym / NeMo Evaluator), and may differ from vendors’ self-reported numbers. Evaluation recipes are published for reproduction, which is more than most releases offer.

Part 6: The architectural idea worth stealing

Set the model aside for a second. The pattern NVIDIA is really shipping is the two-tier agent.

A frontier model plans and orchestrates. A small, fast local model executes. Alongside Lightning, NVIDIA released NeMo Switchyard, an open source routing library that sits inside the harness and directs each request to the most suitable model — across your mix of open, proprietary, and NVIDIA models, without rewriting the application.

Plans route up. Execution routes down.

This is the same instinct that shows up everywhere in agent infrastructure right now, and I think it’s correct. The economics are hard to argue with: if 90% of your agent’s steps are read file, run test, parse output, retry, and those steps run on hardware you already own, your marginal cost for the bulk of the workload goes to roughly zero. The frontier model handles the 10% that needs it.

The privacy story stacks on top. Local execution means the file contents, the repo, the log lines, the alert payloads never leave the machine. For anyone working under data residency constraints, that’s not a nice-to-have — it’s the difference between “we can use agents” and “we can’t.”

But — and I write about agent sandboxing enough to insist on this — “runs locally” is not “runs safely.” A local agent with shell access is still an agent with shell access. Local inference removes the data exfiltration risk from the model provider. It does nothing about the agent rm -rf-ing your working directory because a tool result contained an instruction it shouldn’t have followed. Run it in a sandbox. The privacy win and the blast radius problem are orthogonal.

Part 7: Customization is the actual differentiator

The part I find most underrated: weights, training data, and recipes are all published under OpenMDW-1.1, including Nemotron-RL Agentic Terminal Pivot, the agentic RL dataset used for some of the coding agent capabilities.

You can LoRA or full-SFT it with NeMo Automodel and Megatron Bridge, and run RL with NeMo RL and NeMo Gym.

Thoughtworks trained two LoRA adapters concurrently on a single node of 8× H100s in a few hours, with nothing leaving their environment. That’s the real unlock. A 30B model is small enough to specialize cheaply, and a specialist that does one narrow job well will beat a generalist on that job — while costing a fraction to run.

One nuance on “open datasets,” though. The model card is unusually transparent, and that transparency reveals that alongside all the public corpora there are purchased and private datasets: Mercor SWE-AgentsV1, HackerRank Coding, Turing Math Data Pack, TAUS Translation Memory, and others listed as undisclosed. Some of the released data requires gating and approval. This is meaningfully more open than most releases and it deserves credit — but “trained on open datasets” is a simplification. Read the card if reproducibility is your goal.

Where I’d actually use this

Yes:

• Coding sub-agents inside a larger harness — run tests, grep the codebase, apply a mechanical refactor, summarize a diff
• Security ops enrichment — classify alerts, query logs, correlate indicators, produce a structured object for a human or a bigger model
• The bulk tier under a frontier planner, with Switchyard or your own routing logic
• A narrow specialist you post-train yourself
• Anything where the data genuinely can’t leave the building

Not yet:

• Open-ended autonomous work with no supervision (see Terminal-Bench)
• Customer-facing regulated workflows without a lot of scaffolding (see τ³-bench)
• Anything that leans on the model’s world knowledge instead of tools
• 1M context on consumer hardware, no matter what the tag says

Getting started

# Pull it
ollama pull nemotron-3.5-lightning

# Give it a real context window
OLLAMA_CONTEXT_LENGTH=65536 ollama serve

# Hand it to a harness
ollama launch opencode --model nemotron-3.5-lightning

Then go do something more useful than reading benchmark charts: point it at a repo, give it a bounded task, and watch where it breaks. That’s the only evaluation that tells you anything about your workload.

References

• Nemotron 3.5 Lightning on Ollama
• NVIDIA Technical Blog — Nemotron 3.5 Lightning
• Model card on Hugging Face
• NeMo Switchyard
• Nemotron 3.5 Lightning cookbook
• Thoughtworks — putting Nemotron 3.5 Lightning to the test
• Ollama Claude Code integration docs