How to Connect Two NVIDIA DGX Spark Nodes as Kubernetes GPU Workers for Distributed AI Inference

Table of Contents

A hot topic in the NVIDIA DGX Spark community is how to connect two Spark nodes for distributed GPU workloads. This guide shows you how to set up two DGX Spark systems as GPU worker nodes using Kubernetes, so your inference and training jobs can span both machines.

Architecture Overview

Node 1 (Spark A) acts as the Kubernetes control plane + worker. Node 2 (Spark B) joins as a GPU worker. Both nodes are connected via standard network (Ethernet or the QSFP direct-connect interface on DGX Spark).

Prerequisites

Two NVIDIA DGX Spark systems on the same network
Docker with NVIDIA runtime installed on both nodes
SSH access between nodes
Static IPs or DNS names for each node

Step 1: Set Up NVIDIA Container Runtime on BOTH Nodes

Run these commands on both Spark A and Spark B:

# On BOTH nodes
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker --set-as-default
sudo systemctl restart docker

# Verify
docker run --rm --gpus all nvidia/cuda:12.0-base-ubuntu22.04 nvidia-smi

Step 2: Install K3s on the Control Plane (Spark A)

# On Spark A (control plane)
curl -sfL https://get.k3s.io | sh -s - \
  --docker \
  --write-kubeconfig-mode 644 \
  --disable traefik \
  --bind-address SPARK_A_IP

# Get the node join token
sudo cat /var/lib/rancher/k3s/server/node-token

Step 3: Join Spark B as a Worker Node

# On Spark B (worker node)
curl -sfL https://get.k3s.io | K3S_URL=https://SPARK_A_IP:6443 \
  K3S_TOKEN=YOUR_NODE_TOKEN \
  sh -s - --docker

Step 4: Verify Both Nodes are Ready

# On Spark A
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get nodes

# Expected output:
# NAME       STATUS   ROLES                  AGE
# spark-a    Ready    control-plane,master   5m
# spark-b    Ready    worker                 2m

Step 5: Install GPU Operator on Both Nodes

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.enabled=false \
  --set toolkit.enabled=false \
  --wait

# Verify GPU on each node
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, gpu: .status.capacity["nvidia.com/gpu"]}'

Step 6: Label Nodes for GPU Workload Targeting

kubectl label node spark-a nvidia.com/gpu.product=DGXB10
kubectl label node spark-b nvidia.com/gpu.product=DGXB10

# Or use custom labels
kubectl label node spark-a workload=inference
kubectl label node spark-b workload=inference

Step 7: Deploy a NIM Container Across Both Nodes

Create a DaemonSet to run NIM on every GPU worker node:

kubectl apply -f nim-daemonset.yaml

# nim-daemonset.yaml:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nim-llm
spec:
  selector:
    matchLabels:
      app: nim-llm
  template:
    metadata:
      labels:
        app: nim-llm
    spec:
      nodeSelector:
        workload: inference
      imagePullSecrets:
      - name: ngc-secret
      containers:
      - name: nim
        image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
        ports:
        - containerPort: 8000
        env:
        - name: NGC_API_KEY
          value: YOUR_NGC_API_KEY
        resources:
          limits:
            nvidia.com/gpu: 1

Step 8: Set Up a Load Balancer with MetalLB

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml

# Configure IP pool (adjust to your subnet)
kubectl apply -f - <<YAML
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: spark-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.200-192.168.1.210
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: spark-l2
  namespace: metallb-system
YAML

Step 9: Expose NIM Service with Load Balancing

kubectl apply -f nim-service.yaml

# nim-service.yaml:
apiVersion: v1
kind: Service
metadata:
  name: nim-lb
spec:
  selector:
    app: nim-llm
  ports:
  - port: 8000
    targetPort: 8000
  type: LoadBalancer
  sessionAffinity: None

# Get the assigned external IP
kubectl get svc nim-lb

Step 10: Test Distributed Inference

# Send 20 requests and watch them distribute across both Sparks
for i in $(seq 1 20); do
  curl -s http://EXTERNAL_IP:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta/llama-3.1-8b-instruct", "prompt": "Request number '$i'", "max_tokens": 20}' &
done
wait
echo "All 20 requests complete"

Step 11: Monitor Both Nodes

# On each node, watch GPU usage
nvidia-smi dmon -s u

# From control plane, watch pod distribution
kubectl get pods -o wide

# Check logs from all NIM pods
kubectl logs -l app=nim-llm --prefix=true

Using the QSFP Direct Connect Interface

DGX Spark has a QSFP port for high-speed direct node-to-node connectivity. Configure it as a secondary network for container traffic:

# On Spark A
sudo ip addr add 10.0.0.1/30 dev enp3s0f0  # QSFP interface name may vary
sudo ip link set enp3s0f0 up

# On Spark B
sudo ip addr add 10.0.0.2/30 dev enp3s0f0
sudo ip link set enp3s0f0 up

# Test connectivity
ping -c 3 10.0.0.2

Troubleshooting

Worker node NotReady: Check firewall rules. K3s requires port 6443 (API server) and 10250 (kubelet) open between nodes.

GPU not showing on worker: The GPU Operator DaemonSet must run on the worker node. Check with: kubectl get pods -n gpu-operator -o wide

Conclusion

Two DGX Spark nodes connected via Kubernetes doubles your inference throughput and gives you true high-availability for AI services. The load balancer automatically distributes requests across both nodes, and the DaemonSet ensures NIM runs on every GPU worker in your cluster.