Setting Up a K3s Kubernetes Cluster on NVIDIA DGX Spark with Full GPU Support

A frequently shared topic on the NVIDIA DGX Spark forum is how to set up a proper local Kubernetes cluster with GPU passthrough. This guide walks you through setting up K3s on your DGX Spark GB10 so you can serve LLMs like Qwen3 or Llama using vLLM.

Prerequisites

• NVIDIA DGX Spark with DGX OS (Ubuntu-based)
• Docker installed and running
• NVIDIA drivers installed (verify with nvidia-smi)
• NGC API Key from ngc.nvidia.com

Step 1: Verify GPU Availability

nvidia-smi
# Should show GB10 Grace Blackwell GPU

Step 2: Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

Step 3: Set NVIDIA as the Default Docker Runtime

This is a critical step. K3s needs Docker to use the NVIDIA runtime by default, otherwise GPU pods will fail to start.

sudo nvidia-ctk runtime configure --runtime=docker --set-as-default
sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.0-base-ubuntu22.04 nvidia-smi

Step 4: Install K3s Using Docker as the Container Runtime

curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 644 --disable traefik

Step 5: Verify K3s is Running

sudo systemctl status k3s
kubectl get nodes
# Should show node in Ready state

Step 6: Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Step 7: Install NVIDIA GPU Operator (Drivers Pre-installed)

Since drivers are already installed on DGX Spark, disable the driver component to avoid conflicts.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace --set driver.enabled=false --set toolkit.enabled=false --wait

Step 8: Verify GPU is Visible to Kubernetes

kubectl get nodes -o json | jq '.items[].status.capacity'
# Look for: "nvidia.com/gpu": "1"

Step 9: Create NGC Pull Secret

kubectl create secret docker-registry ngc-secret --docker-server=nvcr.io --docker-username='$oauthtoken' --docker-password=YOUR_NGC_API_KEY --namespace default

Step 10: Deploy vLLM with Qwen3-4B

kubectl apply -f vllm-deployment.yaml

# vllm-deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-qwen3
  template:
    metadata:
      labels:
        app: vllm-qwen3
    spec:
      imagePullSecrets:
      - name: ngc-secret
      containers:
      - name: vllm
        image: nvcr.io/nvidia/vllm:25.12-py3
        args: ["--model", "Qwen/Qwen3-4B", "--port", "8000"]
        resources:
          limits:
            nvidia.com/gpu: 1

Step 11: Expose and Test

kubectl expose deployment vllm-qwen3 --type=NodePort --port=8000
# Then test with:
curl http://localhost:PORT/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "Qwen/Qwen3-4B", "messages": [{"role": "user", "content": "Hello!"}]}'

Troubleshooting

GPU not in node capacity: Run kubectl describe node and restart the NVIDIA device plugin with kubectl rollout restart daemonset -n gpu-operator.

Pod stuck in ContainerCreating: Ensure Docker uses NVIDIA runtime by default. Re-run: sudo nvidia-ctk runtime configure –runtime=docker –set-as-default

Conclusion

You now have a fully functional K3s Kubernetes cluster on your NVIDIA DGX Spark. Scale to multi-node by adding workers with the K3s join token found at /var/lib/rancher/k3s/server/node-token.