Building AI Agents on DGX Spark with Kubernetes: A Complete Tutorial

Table of Contents

The NVIDIA DGX Spark is a compact, personal AI supercomputer powered by the GB10 Grace Blackwell Superchip. With 128GB of unified memory, it can run multiple large language models (LLMs) and vision language models (VLMs) in parallel, making it an ideal platform for building fully local, private, and always-on agentic AI workflows. When you add Kubernetes (K8s) into the picture, you get production-grade orchestration, auto-restart, resource management, and the ability to scale across multiple DGX Spark nodes in a cluster topology.

This tutorial will walk you through setting up the DGX Spark environment, deploying a local LLM inference server, and running a multi-agent system — all orchestrated with Kubernetes manifests.

Prerequisites

A DGX Spark device running NVIDIA DGX OS 7.3.1 (Ubuntu 24.04.3 LTS base)
SSH or local terminal access to the device
Docker with the NVIDIA Container Toolkit installed
kubectl installed on the DGX Spark or a remote management node
A local single-node Kubernetes cluster (e.g., k3s, microk8s, or kubeadm)
A HuggingFace account and API token for downloading model weights
git and curl installed
No other GPU workloads running (nvidia-smi should show GPU memory free)

Part 1 — Set Up Your Kubernetes Cluster on DGX Spark

Step 1.1 — Install k3s (Lightweight K8s)

DGX Spark is ideal for a single-node K8s cluster using k3s, which is production-ready and low overhead.

curl -sfL https://get.k3s.io | sh -

Verify the installation:

sudo kubectl get nodes

Copy the kubeconfig for easier access:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

Step 1.2 — Install the NVIDIA Device Plugin for Kubernetes

For Kubernetes to schedule workloads that use the GPU, install the NVIDIA device plugin:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.4/nvidia-device-plugin.yml

Verify the GPU is visible to K8s:

kubectl get nodes -o json | grep -i nvidia

You should see nvidia.com/gpu: 1 or more listed under allocatable resources.

⚠️ DGX Spark / GB10 Compatibility Note: Device plugin versions earlier than v0.17.4 will crash on DGX Spark hardware with the error error getting device memory: Not Supported. This is a known issue with the GB10 GPU’s unified memory architecture, which does not support the standard NVML memory query (nvidia-smi will show Memory-Usage: Not Supported). Always use device plugin v0.17.4 or later. If you are using the NVIDIA GPU Operator, upgrade to v25.10.0 or later (or pin devicePlugin.version=v0.17.4 in your Helm values). See NVIDIA/gpu-operator#1794 for details.

Part 2 — Deploy a Local LLM Inference Server

The foundation of any agent system on DGX Spark is a locally served LLM. The playbooks on build.nvidia.com/spark recommend vLLM with the nvidia/Qwen3.6-35B-A3B-NVFP4 model — an NVIDIA FP4 precision build tuned specifically for the Blackwell architecture.

Step 2.1 — Create a Kubernetes Secret for HuggingFace

kubectl create secret generic hf-secret   --from-literal=HF_TOKEN=your_huggingface_token

Step 2.2 — Create a Persistent Volume for Model Storage

Save this as model-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: local-path

kubectl apply -f model-pvc.yaml

Step 2.3 — Deploy vLLM as a K8s Deployment

Save this as vllm-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
  labels:
    app: vllm-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-server
  template:
    metadata:
      labels:
        app: vllm-server
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - "--model"
            - "nvidia/Qwen3.6-35B-A3B-NVFP4"
            - "--host"
            - "0.0.0.0"
            - "--port"
            - "8000"
            - "--max-model-len"
            - "32768"
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: HF_TOKEN
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "80Gi"
            requests:
              nvidia.com/gpu: 1
              memory: "40Gi"
          volumeMounts:
            - name: model-cache
              mountPath: /root/.cache/huggingface
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-storage
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm-server
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000
  type: ClusterIP

kubectl apply -f vllm-deployment.yaml

Watch the pod come online — model download can take 30-60 minutes:

kubectl get pods -w
kubectl logs -f deployment/vllm-server

Test the inference endpoint once the pod is Running:

kubectl port-forward svc/vllm-service 8000:8000

curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "nvidia/Qwen3.6-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Hello, what can you do?"}]
  }'

Part 3 — Deploy the Multi-Agent Chatbot System

The DGX Spark multi-agent architecture uses a supervisor agent powered by a large model (like gpt-oss-120B) orchestrating specialized downstream agents: a Coding Agent (Deepseek-Coder:6.7B-Instruct), a RAG Agent with GPU-accelerated document retrieval, and an Image Understanding Agent using a VLM. Each agent is exposed as an MCP (Model Context Protocol) server that the supervisor calls as a tool.

Step 3.1 — Clone the DGX Spark Playbooks Repository

git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/multi-agent-chatbot/assets

Step 3.2 — Download Model Files

The playbook provides a download script that pulls the necessary GGUF model files from HuggingFace. These include gpt-oss-120B (~63 GB) for the supervisor agent, Deepseek-Coder:6.7B-Instruct (~7 GB) for the coding agent, and Qwen3-Embedding-4B (~4 GB) for RAG embeddings.

chmod +x model_download.sh
./model_download.sh

Note: This step can take between 30 minutes to 2 hours depending on your network speed. DGX Spark requires ~120 out of 128GB of unified memory for the default configuration. Ensure no other workloads are running with nvidia-smi before proceeding.

Step 3.3 — Create Kubernetes ConfigMap for Agent Configuration

Save this as agent-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-config
data:
  SUPERVISOR_MODEL: "gpt-oss-120B"
  CODER_MODEL: "deepseek-coder:6.7b-instruct"
  EMBEDDING_MODEL: "Qwen3-Embedding-4B"
  VLLM_ENDPOINT: "http://vllm-service:8000/v1"
  BACKEND_PORT: "8000"
  FRONTEND_PORT: "3000"

kubectl apply -f agent-config.yaml

Step 3.4 — Deploy the Supervisor Agent

Save this as supervisor-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: supervisor-agent
  labels:
    app: supervisor-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: supervisor-agent
  template:
    metadata:
      labels:
        app: supervisor-agent
    spec:
      containers:
        - name: supervisor
          image: nvidia/dgx-spark-supervisor:latest
          envFrom:
            - configMapRef:
                name: agent-config
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: "100Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: supervisor-service
spec:
  selector:
    app: supervisor-agent
  ports:
    - port: 8000
      targetPort: 8000

Step 3.5 — Deploy the Frontend UI

Save this as frontend-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-frontend
  labels:
    app: agent-frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: agent-frontend
  template:
    metadata:
      labels:
        app: agent-frontend
    spec:
      containers:
        - name: frontend
          image: nvidia/dgx-spark-frontend:latest
          env:
            - name: BACKEND_URL
              value: "http://supervisor-service:8000"
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: agent-frontend
  ports:
    - port: 3000
      targetPort: 3000
  type: NodePort

Apply all agent manifests:

kubectl apply -f supervisor-deployment.yaml
kubectl apply -f frontend-deployment.yaml
kubectl get pods

Part 4 — Access the Agent UI

Option A — Local Access

kubectl port-forward svc/frontend-service 3000:3000
kubectl port-forward svc/supervisor-service 8000:8000

Open your browser at http://localhost:3000.

Option B — Remote Access via SSH Tunnel

ssh -L 3000:localhost:3000 -L 8000:localhost:8000 username@dgx-spark-ip

Then open http://localhost:3000 in your browser on your local machine.

Part 5 — Working with the Multi-Agent System

Once the UI is open, you can interact with three specialized agents through the supervisor. The Coding Agent handles code generation and review — try asking it to write a Python script to benchmark matrix multiplication on GPU. The RAG Agent answers questions from uploaded documents — upload a PDF like the NVIDIA Blackwell Whitepaper using the green “Upload Documents” button in the left sidebar, then query it. The Image Understanding Agent runs locally using TensorRT LLM servers and can analyze images you provide.

Part 6 — Running a CLI Coding Agent (Lightweight Alternative)

For a simpler terminal-based agent workflow, the DGX Spark platform supports CLI coding agents with Ollama. This is a great starting point without the full Kubernetes stack.

# Install Ollama (v0.15+ required)
curl -fsSL https://ollama.com/install.sh | sh

# Pull the recommended model for Blackwell (NVIDIA FP4, ~22GB)
ollama pull qwen3.6:35b-a3b-nvfp4

# Launch Claude Code agent
ollama launch claude-code --model qwen3.6:35b-a3b-nvfp4

# Or launch OpenCode (open source)
ollama launch opencode --model qwen3.6:35b-a3b-nvfp4

# Or launch Codex CLI
ollama launch codex --model qwen3.6:35b-a3b-nvfp4

Part 7 — Connecting Multiple DGX Spark Nodes

If you have multiple DGX Spark devices, you can form a Kubernetes cluster for distributed inference and agent orchestration. The platform supports both ring topology (3 nodes) and switch-connected clusters.

# On the primary node — install k3s as control plane
curl -sfL https://get.k3s.io | sh -

# Get the join token
cat /var/lib/rancher/k3s/server/node-token

# On Node 2 and Node 3 — join as workers
curl -sfL https://get.k3s.io | K3S_URL=https://PRIMARY_NODE_IP:6443   K3S_TOKEN=JOIN_TOKEN sh -

# Verify all nodes are in the cluster
kubectl get nodes

# Label nodes for specific workloads
kubectl label nodes node-1 role=inference
kubectl label nodes node-2 role=agent
kubectl label nodes node-3 role=rag

You can then use nodeSelector in your pod specs to pin specific agent workloads to specific DGX Spark units, ensuring optimal memory utilization across the cluster.

Part 8 — Monitoring Your DGX Spark Cluster

# Forward DGX Dashboard to local browser
kubectl port-forward svc/dgx-dashboard 8080:8080

# Quick GPU health check with nvidia-smi
watch -n1 nvidia-smi

# Monitor all pods and their status
kubectl get pods --all-namespaces -w

Cleanup and Rollback

kubectl delete deployment vllm-server supervisor-agent agent-frontend
kubectl delete svc vllm-service supervisor-service frontend-service
kubectl delete pvc model-storage
kubectl delete configmap agent-config
kubectl delete secret hf-secret

Summary

The DGX Spark’s 128GB unified memory, Blackwell GPU architecture, and native Linux environment make it uniquely suited for running large, multi-model agent pipelines entirely on-premise. Kubernetes adds the operational reliability you need to run these workloads in production — with automatic restarts, resource limits, service discovery, and easy multi-node scaling.

Component	Technology	Role
Local inference server	vLLM + Qwen3.6	Serves LLM via OpenAI-compatible API
Supervisor agent	gpt-oss-120B via llama.cpp	Orchestrates downstream agents
Coding agent	Deepseek-Coder 6.7B	Code generation and review
RAG agent	Qwen3-Embedding-4B + GPU retrieval	Document Q&A
Image agent	VLM via TensorRT LLM	Visual understanding
Orchestration	Kubernetes (k3s)	Pod scheduling, health checks, scaling
Frontend	Web UI on port 3000	User interface for agent interaction

Resources

Continue your learning journey with these related Collabnix articles:

NVIDIA DGX Spark + Kubernetes: Run GPU Workloads on the GB10 Grace Blackwell Superchip — Deep-dive into scheduling GPU workloads on Kubernetes with the DGX Spark platform.
Getting Started with NVIDIA DGX Spark: Your Personal AI Supercomputer — Set up your DGX Spark from scratch before deploying your first AI agent.
How to Run GPU Workloads on Kubernetes with NVIDIA: A Step-by-Step Guide — Step-by-step guide for running GPU-accelerated workloads on Kubernetes.
Kubernetes for Generative AI: Complete Guide to Deploying LLMs at Scale — Learn how to deploy and scale LLMs using Kubernetes in production environments.