Docker Compose vs Kubernetes on NVIDIA DGX Spark: Which Setup Should You Choose for NIM Microservices?

One of the most common questions on the NVIDIA Developer Forums is: Should you use Docker Compose or Kubernetes to run NIM microservices on your DGX Spark? This guide cuts through the confusion and gives you practical steps for both setups so you can decide which fits your workload.

Why This Decision Matters

The NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, 128GB unified memory) is a powerful personal AI supercomputer. Choosing the wrong orchestration layer can waste GPU cycles or complicate your deployment. Here is a practical breakdown.

Option 1: Docker Compose Setup

Best for: Single DGX Spark node, quick prototyping, dev/test workflows.

Step 1: Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 2: Authenticate with NGC

docker login nvcr.io
# Username: $oauthtoken
# Password: YOUR_NGC_API_KEY

Step 3: Create docker-compose.yml for NIM

version: "3.8"
services:
  nim-llm:
    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
    runtime: nvidia
    environment:
      - NIM_HTTP_API_PORT=8000
    ports:
      - "8000:8000"
    volumes:
      - ./models:/opt/nim/.cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Step 4: Launch the NIM service

export NGC_API_KEY=nvapi-xxxxxxxxxxxx
docker compose up -d

Step 5: Test the endpoint

curl -X POST http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{"model": "meta/llama-3.1-8b-instruct", "prompt": "What is DGX Spark?", "max_tokens": 100}'

Option 2: Kubernetes Setup (K3s)

Best for: Multi-node DGX Spark clusters, production workloads, GPU scheduling across nodes.

Step 1: Configure NVIDIA runtime as default

sudo nvidia-ctk runtime configure --runtime=docker --set-as-default
sudo systemctl restart docker

Step 2: Install K3s with Docker runtime

curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 644

Step 3: Install NVIDIA GPU Operator

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator   --namespace gpu-operator   --create-namespace   --set driver.enabled=false   --set toolkit.enabled=false

Step 4: Create NGC pull secret

kubectl create secret docker-registry ngc-secret   --docker-server=nvcr.io   --docker-username='$oauthtoken'   --docker-password=YOUR_NGC_API_KEY

Step 5: Deploy NIM as a Kubernetes Deployment

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nim-llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nim-llm
  template:
    metadata:
      labels:
        app: nim-llm
    spec:
      imagePullSecrets:
      - name: ngc-secret
      containers:
      - name: nim-llm
        image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
        env:
        - name: NGC_API_KEY
          value: "YOUR_NGC_API_KEY"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
EOF

Step 6: Expose the service and test

kubectl expose deployment nim-llm --type=NodePort --port=8000
kubectl get svc nim-llm

Quick Comparison

Verdict

If you have a single DGX Spark and want to get NIM running fast, Docker Compose wins. If you have two or more Sparks and need production-grade scheduling, go with K3s or full Kubernetes. The GPU performance of NIM containers is identical in both — the difference is only in orchestration overhead.