Running Qwen3.6-35B-A3B (FP8) on NVIDIA DGX Spark GB10: A Complete Technical Tutorial

Introduction

The NVIDIA DGX Spark is a personal AI supercomputer powered by the Grace Blackwell GB10 Superchip, featuring 128 GB of LPDDR5X unified memory shared between the 20-core Grace CPU and the Blackwell GPU. This tutorial walks you through setting up and running Qwen/Qwen3.6-35B-A3B-FP8 — a 35B-parameter Mixture-of-Experts language model from Alibaba’s Qwen team — on a single or dual DGX Spark system using vLLM.

Most Searched Forum Thread: This tutorial is based on the #1 most-viewed thread on the NVIDIA DGX Spark / GB10 forum: “Qwen/Qwen3.6-35B-A3B (and FP8) has landed” with over 28,000 views, 309 replies, and 285 likes.

Why Qwen3.6-35B-A3B on DGX Spark?

Qwen3.6-35B-A3B is a MoE model with 35B total parameters but only ~3.5B active parameters per token inference. This architecture makes it extraordinarily memory-efficient while maintaining strong reasoning capability. The FP8 quantized variant fits comfortably within the DGX Spark’s 128 GB unified memory on a single node and delivers impressive token-per-second throughput.

Key capabilities of Qwen3.6:

• Agentic Coding: handles frontend workflows and repository-level reasoning with greater fluency and precision
• Thinking Preservation: retains reasoning context across conversation history for iterative development
• Tool Calling: robust function/tool-call support via the qwen3_coder parser
• Long Context: supports up to 262,144 token context windows

System Requirements

• Hardware: NVIDIA DGX Spark (GB10 Grace Blackwell Superchip), 128 GB unified memory
• OS: DGX OS (Ubuntu 24.04-based, ARM64/aarch64)
• CUDA: 12.8 or later (DGX Spark ships with CUDA 13.0)
• Driver: 570+ (580.x recommended)
• Container Runtime: Docker with NVIDIA Container Toolkit
• Network (for dual-node): Mellanox ConnectX-7 via NVIDIA Sync / NVLink Interconnect

Step 1: Verify Your DGX Spark System

Before pulling models, verify your hardware and driver setup.

# Check GPU and driver info
nvidia-smi

# On DGX Spark, GPU memory appears as unified — check system memory instead
free -h

# Verify CUDA version
nvcc --version

# Confirm aarch64 architecture
uname -m
# Expected output: aarch64

Note: On the DGX Spark, nvidia-smi does not show GPU memory usage (it uses unified memory), which is a known limitation. Use system tools like btop or free to track memory consumption.

Step 2: Pull the vLLM Docker Image for DGX Spark

NVIDIA provides a pre-built vLLM image optimized for the GB10 architecture. The standard upstream vLLM image will not work on ARM64/aarch64 — you must use the DGX-targeted build.

# Pull the NVIDIA-optimized vLLM image for DGX Spark
docker pull nvcr.io/nvidia/vllm:25.05-py3

# Alternatively, pull the latest nightly if you need cutting-edge features
docker pull nvcr.io/nvidia/vllm:latest

If you prefer to build from source with DGX Spark patches:

git clone https://github.com/vllm-project/vllm.git
cd vllm
# Use the DGX Spark community-maintained build flags for SM121/GB10
pip install -e . --extra-index-url https://pypi.nvidia.com

Step 3: Download the Qwen3.6-35B-A3B-FP8 Model

The FP8 checkpoint is hosted on Hugging Face. You can download it directly using huggingface-cli or via a snapshot_download call inside Docker.

# Install huggingface-hub if not present
pip install -U huggingface-hub

# Download the FP8 quantized model (~18 GB)
huggingface-cli download Qwen/Qwen3.6-35B-A3B-FP8 \
  --local-dir /data/models/Qwen3.6-35B-A3B-FP8 \
  --local-dir-use-symlinks False

You can also use the standard BF16 checkpoint if you want full-precision inference (requires more memory):

huggingface-cli download Qwen/Qwen3.6-35B-A3B \
  --local-dir /data/models/Qwen3.6-35B-A3B \
  --local-dir-use-symlinks False

Step 4: Launch vLLM on a Single DGX Spark

This command launches the model as an OpenAI-compatible API server on a single DGX Spark node.

docker run --rm -it \
  --gpus all \
  --runtime=nvidia \
  --shm-size=32g \
  -p 8080:8080 \
  -v /data/models:/models \
  nvcr.io/nvidia/vllm:25.05-py3 \
  vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --host 0.0.0.0 \
    --port 8080 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 131072 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 4 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --attention-backend flashinfer \
    --load-format instanttensor \
    --trust-remote-code \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3

Key Parameters Explained

Step 5: Launch vLLM on Dual DGX Spark (2× GB10)

For maximum context length (262K tokens) and higher throughput, run tensor-parallel inference across two DGX Spark units connected via NVLink/Ethernet.

Prerequisites for Dual-Node Setup

# On the head node, start the Ray cluster
ray start --head --port=6379

# On the worker node, join the cluster (replace HEAD_IP)
ray start --address=HEAD_IP:6379

Dual-Node vLLM Launch Command

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
  --host 0.0.0.0 \
  --port 8080 \
  --gpu-memory-utilization 0.80 \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 4 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --attention-backend flashinfer \
  --load-format instanttensor \
  --trust-remote-code \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray

Step 6: Verify the Server and Run Your First Inference

Once the server is up (watch for INFO: Application startup complete), test it with curl:

# Basic chat completion (non-thinking mode)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-35B-A3B-FP8",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain Mixture-of-Experts architecture in 3 sentences."}
    ],
    "max_tokens": 512,
    "temperature": 0.7
  }'

To enable the model’s extended thinking/reasoning chain:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.6-35B-A3B-FP8",
    "messages": [
      {"role": "user", "content": "Write a Python function to merge two sorted lists."}
    ],
    "chat_template_kwargs": {"enable_thinking": true},
    "max_tokens": 2048,
    "temperature": 0.6
  }'

Step 7: Agentic Tool Calling

One of Qwen3.6’s headline features is its native tool-calling capability. Here is an example using the OpenAI Python client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["city"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools,
    tool_choice="auto"
)

print(response.choices[0].message.tool_calls)

Step 8: Performance Benchmarks (Community Results from NVIDIA Forum)

The following benchmarks were collected by the DGX Spark community on dual-node (2× GB10) with the configuration from Step 5, using llama-benchy v0.3.5:

• Tool call success rate: 100% at ToolCall-15 benchmark
• pp = prefill phase, tg = token generation, TTFT = time-to-first-token

Step 9: Monitoring and Optimization Tips

Monitor memory usage (since nvidia-smi doesn’t show GPU memory on DGX Spark):

# Install btop for a comprehensive system monitor
sudo apt install btop

# Or use this one-liner to watch memory
watch -n 1 'free -h && cat /proc/meminfo | grep MemAvailable'

• Tune --gpu-memory-utilization: Start at 0.80 and increase to 0.88 if you have headroom. Values above 0.90 can cause OOM crashes under load.
• Use prefix caching for system prompts: If you use a fixed system prompt, --enable-prefix-caching will cache the KV states and dramatically reduce TTFT for subsequent requests.
• FP8 KV cache saves ~40-50% memory: The --kv-cache-dtype fp8 flag is strongly recommended for long context workloads.
• Disable thinking for faster responses: When you don’t need chain-of-thought, pass "enable_thinking": false in chat_template_kwargs for lower latency.

Troubleshooting Common Issues

• AssertionError: VLLM_USE_PRECOMPILED is only supported for CUDA builds — This happens when using the wrong Docker image. Always use the nvcr.io/nvidia/vllm image, not the upstream PyPI vLLM package, on DGX Spark.
• AppImage / FUSE errors — DGX OS does not ship with FUSE by default. Install it with sudo apt install fuse.
• NCCL socket transport failure on dual-node — Ensure both units have correct IP routing. Set NCCL_SOCKET_IFNAME to point to the correct network interface (e.g., export NCCL_SOCKET_IFNAME=enp1s0).
• Model loads slowly — Use --load-format instanttensor which leverages NVIDIA’s InstantTensor format for fast checkpoint loading.

Summary

Running Qwen3.6-35B-A3B-FP8 on the NVIDIA DGX Spark is one of the most impactful local AI deployments available today. With 28K+ community views on the NVIDIA Developer Forums, this setup captures the cutting edge of personal AI supercomputing. A single GB10 delivers 64–77 token/s decode throughput at up to 128K context, while a dual-node setup unlocks 262K context with near-identical decode performance. The combination of FP8 quantization, FlashInfer attention, chunked prefill, and prefix caching makes the DGX Spark a genuinely capable local inference platform for agentic coding, long-document reasoning, and multi-tool orchestration workflows.

Tutorial based on the most-searched thread on the NVIDIA DGX Spark / GB10 User Forum. Source: NVIDIA Developer Forums