Getting Started with NVIDIA DGX Spark: Your Personal AI Supercomputer – A Complete Tutorial Guide

NVIDIA DGX Spark is the world’s first desktop AI supercomputer powered by the NVIDIA GB10 Grace Blackwell Superchip. With 1 petaFLOP of FP4 AI compute, 128GB of coherent unified memory, and the complete NVIDIA AI software stack, DGX Spark lets developers, researchers, and data scientists run frontier-scale models locally ~ no cloud required.

This tutorial guide walks you through everything: unboxing and setup, installing the DGX OS, running your first LLM inference, deploying Jupyter notebooks, building AI agents with LangChain, and scaling with dual-DGX Spark via ConnectX networking.

Prerequisites

• NVIDIA DGX Spark hardware
• Ubuntu 22.04 / DGX OS (pre-installed)
• NVIDIA account for NGC registry access
• Basic Python 3.10+ knowledge

Step 1: Initial System Setup

Once unboxed, power on DGX Spark and connect it to your network. The DGX OS is pre-installed. Verify your system and update packages:

# Verify system info
uname -a
# Output: Linux dgx-spark 5.15.0-... aarch64 GNU/Linux

# Check GPU/Grace Blackwell Superchip
nvidia-smi
# +-----------------------------------------------------------------------------+
# | NVIDIA-SMI 560.x   Driver Version: 560.x   CUDA Version: 12.6             |
# |   0   GB10 Blackwell  Off  | 00000000:01:00.0  0MiB / 96000MiB            |
# +-----------------------------------------------------------------------------+

# Update the system
sudo apt-get update && sudo apt-get upgrade -y

# Check DGX OS version
cat /etc/dgx-release
# DGX_OS_VERSION=6.0
# DGX_PLATFORM=DGX Spark

Step 2: Install NVIDIA Container Toolkit and Docker

DGX Spark uses NVIDIA Containers for AI workloads via NGC (NVIDIA GPU Cloud). Install Docker and the NVIDIA Container Toolkit:

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER
newgrp docker

# Configure NVIDIA Container Toolkit repo
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor   -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |   sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Verify GPU access inside a container
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi

Step 3: Pull a Model from NGC and Run First LLM Inference

DGX Spark can run models up to 200B parameters locally. Pull Llama 3.1 70B using NVIDIA NIM (NVIDIA Inference Microservices):

# Login to NGC (get your API key at https://ngc.nvidia.com)
docker login nvcr.io
# Username: $oauthtoken
# Password: YOUR_NGC_API_KEY

# Pull Llama 3.1 70B NIM container
docker pull nvcr.io/nim/meta/llama-3.1-70b-instruct:latest

# Run the inference container (uses ~70GB of unified memory)
docker run -it --rm   --gpus all   -e NGC_API_KEY=$NGC_API_KEY   -p 8000:8000   -v ~/nim-cache:/opt/nim/.cache   nvcr.io/nim/meta/llama-3.1-70b-instruct:latest

Once the NIM server is running, query it with Python:

# query_nim.py
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed-for-local"
)

response = client.chat.completions.create(
    model="meta/llama-3.1-70b-instruct",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful AI assistant running locally on NVIDIA DGX Spark."
        },
        {
            "role": "user",
            "content": "Explain the NVIDIA Grace Blackwell architecture in 3 sentences."
        }
    ],
    temperature=0.6,
    max_tokens=512
)

print(response.choices[0].message.content)

Step 4: Set Up JupyterLab for AI Development

Launch JupyterLab inside an NVIDIA PyTorch container for interactive AI experimentation:

# Pull the latest NVIDIA PyTorch container
docker pull nvcr.io/nvidia/pytorch:24.10-py3

# Launch JupyterLab with full GPU access
docker run -it --rm   --gpus all   --ipc=host   --ulimit memlock=-1   --ulimit stack=67108864   -p 8888:8888   -v $HOME/notebooks:/workspace/notebooks   nvcr.io/nvidia/pytorch:24.10-py3   jupyter lab --ip=0.0.0.0 --port=8888 --allow-root --no-browser

# Open browser at: http://localhost:8888

Inside your notebook, verify GPU access and benchmark the GB10 Superchip:

import torch
import time

# Confirm GPU device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"Total VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")

# Benchmark: FP16 Matrix Multiplication on DGX Spark
size = 16384
A = torch.randn(size, size, device=device, dtype=torch.float16)
B = torch.randn(size, size, device=device, dtype=torch.float16)

torch.cuda.synchronize()
start = time.perf_counter()
for _ in range(10):
    C = torch.matmul(A, B)
torch.cuda.synchronize()
elapsed = (time.perf_counter() - start) / 10

tflops = 2 * size**3 / elapsed / 1e12
print(f"MatMul {size}x{size} FP16: {elapsed*1000:.2f} ms  |  {tflops:.1f} TFLOPS")

# Expected output on DGX Spark:
# GPU: NVIDIA GB10 Grace Blackwell Superchip
# Total VRAM: 96.0 GB
# MatMul 16384x16384 FP16: 48.3 ms  |  181.6 TFLOPS

Step 5: Fine-tune Llama 3 with PEFT / LoRA

With 128GB of unified memory, DGX Spark is purpose-built for fine-tuning large models locally using Hugging Face PEFT with LoRA:

pip install transformers peft datasets accelerate bitsandbytes trl

# finetune_llama.py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
from datasets import load_dataset

MODEL_ID = "meta-llama/Meta-Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Configure LoRA adapters
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.52%

dataset = load_dataset("timdettmers/openassistant-guanaco", split="train[:5%]")

training_args = TrainingArguments(
    output_dir="./dgx-spark-llama3-lora",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_steps=200,
    optim="adamw_torch_fused",
    report_to="none",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=2048,
)

trainer.train()
trainer.model.save_pretrained("./dgx-spark-llama3-lora-final")

Step 6: Build a Local AI Agent with LangChain and Ollama

DGX Spark is designed to run autonomous agents locally. Here is how to set up a fully local ReAct agent using Ollama for model serving and LangChain for orchestration:

# Install Ollama on DGX Spark
curl -fsSL https://ollama.com/install.sh | sh

# Pull Llama 3.1 8B locally
ollama pull llama3.1:8b

# Test it
ollama run llama3.1:8b "Hello from DGX Spark!"

# Install Python dependencies
pip install langchain langchain-community langchain-ollama duckduckgo-search

# local_agent.py
from langchain_ollama import ChatOllama
from langchain.agents import AgentExecutor, create_react_agent
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.prompts import PromptTemplate

# Initialize local LLM on DGX Spark via Ollama
llm = ChatOllama(
    model="llama3.1:8b",
    base_url="http://localhost:11434",
    temperature=0.1,
)

tools = [DuckDuckGoSearchRun()]

react_prompt = PromptTemplate.from_template("""
Answer the following questions as best you can. You have access to:
{tools}

Use this format:
Question: {input}
Thought: think about what to do
Action: tool name (one of [{tool_names}])
Action Input: the input to the tool
Observation: result
Thought: I now know the final answer
Final Answer: the final answer

Question: {input}
Thought: {agent_scratchpad}
""")

agent = create_react_agent(llm, tools, react_prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    max_iterations=5,
    handle_parsing_errors=True,
)

# 100% local — zero cloud calls
result = agent_executor.invoke({
    "input": "What are the key specs of the NVIDIA DGX Spark?"
})
print(result["output"])

Step 7: Scale to Dual DGX Spark via ConnectX Networking

Two DGX Spark units linked via NVIDIA ConnectX networking doubles memory to 256GB, enabling models up to 405B parameters such as Llama 3.1 405B:

# On both nodes — verify ConnectX link speed
ibstat
# Port 1:
#   State: Active
#   Rate: 400 Gb/sec

# Configure passwordless SSH between nodes
ssh-keygen -t ed25519 -f ~/.ssh/dgx_spark_key
ssh-copy-id -i ~/.ssh/dgx_spark_key.pub user@dgx-spark-node2

# Install vLLM for distributed inference
pip install vllm

# Launch Llama 3.1 405B across two DGX Spark nodes (run on node 1)
python -m vllm.entrypoints.openai.api_server     --model meta-llama/Meta-Llama-3.1-405B-Instruct     --tensor-parallel-size 2     --dtype bfloat16     --max-model-len 32768     --host 0.0.0.0     --port 8000     --distributed-executor-backend ray

# query_405b.py
import requests, json

payload = {
    "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",
    "messages": [
        {
            "role": "user",
            "content": "Write a Python function that implements RAG with FAISS."
        }
    ],
    "max_tokens": 1024,
    "temperature": 0.7
}

resp = requests.post(
    "http://localhost:8000/v1/chat/completions",
    headers={"Content-Type": "application/json"},
    data=json.dumps(payload)
)
print(resp.json()["choices"][0]["message"]["content"])

Step 8: Build a RAG Pipeline with FAISS on GPU

A complete GPU-accelerated Retrieval-Augmented Generation pipeline running entirely on DGX Spark:

pip install faiss-gpu sentence-transformers langchain openai

# rag_pipeline.py
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
from openai import OpenAI

# 1. Build GPU-accelerated vector store
embedder = SentenceTransformer("BAAI/bge-large-en-v1.5", device="cuda")

documents = [
    "NVIDIA DGX Spark is powered by the GB10 Grace Blackwell Superchip.",
    "DGX Spark delivers 1 petaFLOP of FP4 AI performance.",
    "The system has 128GB of coherent unified memory shared between CPU and GPU.",
    "Two DGX Spark units linked via ConnectX run models up to 405B parameters.",
    "DGX Spark runs NIM, NeMo, TensorRT-LLM, and the full NVIDIA AI stack.",
    "The compact desktop form factor runs under 150W TDP.",
]

print("Embedding documents on GPU...")
embeddings = embedder.encode(documents, batch_size=32, device="cuda")
embeddings = np.array(embeddings, dtype=np.float32)

# Create FAISS GPU index
dim = embeddings.shape[1]
res = faiss.StandardGpuResources()
index = faiss.index_cpu_to_gpu(res, 0, faiss.IndexFlatL2(dim))
index.add(embeddings)

# 2. RAG query function
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

def rag_query(question: str, top_k: int = 3) -> str:
    query_vec = np.array(
        embedder.encode([question], device="cuda"), dtype=np.float32
    )
    _, indices = index.search(query_vec, top_k)
    context = "
".join([documents[i] for i in indices[0]])

    resp = client.chat.completions.create(
        model="meta/llama-3.1-70b-instruct",
        messages=[
            {
                "role": "system",
                "content": f"Answer using only this context:
{context}"
            },
            {"role": "user", "content": question}
        ],
        max_tokens=256,
    )
    return resp.choices[0].message.content

# 3. Test the RAG pipeline
print(rag_query("What is the memory capacity of DGX Spark?"))
# Answer: DGX Spark has 128GB of coherent unified memory shared between
# the CPU and GPU.

Step 9: Monitor GPU Performance with DCGM

Use NVIDIA DCGM to monitor GPU utilization, memory, thermals, and power on DGX Spark in real time:

# Install DCGM
sudo apt-get install -y datacenter-gpu-manager
sudo systemctl enable --now nvidia-dcgm

# Monitor all GPU metrics every second
dcgmi dmon -e 203,204,155,150,156,100,110 -d 1000
# 203=GPU Util(%)  204=Mem Util(%)  155=Temp(C)  150=Power(W)
# 156=SM Clock(MHz)  100=FB Mem Used(MiB)  110=FB Mem Free(MiB)

# Sample output during LLM inference:
# Entity  GUTIL  MUTIL  TEMP  POWER  SMCLK   FBUSD   FBFRE
# GPU 0    98     87     72    148   2520   83421   13011

# live_monitor.py — Real-time GPU dashboard
import subprocess, time

def get_gpu_stats():
    result = subprocess.run([
        "nvidia-smi",
        "--query-gpu=name,temperature.gpu,utilization.gpu,"
                     "utilization.memory,power.draw,memory.used,memory.free",
        "--format=csv,noheader,nounits"
    ], capture_output=True, text=True)
    v = result.stdout.strip().split(",")
    return {
        "name":         v[0].strip(),
        "temp_c":       int(v[1]),
        "gpu_util":     int(v[2]),
        "mem_util":     int(v[3]),
        "power_w":      float(v[4]),
        "mem_used_mb":  int(v[5]),
        "mem_free_mb":  int(v[6]),
    }

print("=== DGX Spark GPU Monitor ===")
while True:
    s = get_gpu_stats()
    total = s["mem_used_mb"] + s["mem_free_mb"]
    print(
        f"GPU: {s['gpu_util']}%  |  "
        f"Mem: {s['mem_used_mb']:,}/{total:,} MB  |  "
        f"Temp: {s['temp_c']}C  |  "
        f"Power: {s['power_w']:.1f}W"
    )
    time.sleep(2)

DGX Spark Key Specs at a Glance

What’s Next?

• Explore NVIDIA DGX Spark Playbooks for curated end-to-end recipes
• Try NVIDIA NIM Blueprints for production-ready multimodal and agentic pipelines
• Join the DGX Spark Developer Forum for community support
• Deploy NVIDIA NemoClaw for private, secure autonomous agents with OpenShell
• Scale to DGX Cloud when workloads exceed two nodes

NVIDIA DGX Spark puts a data center worth of AI compute right on your desk. With the tutorials above, you have everything needed to go from unboxing to running frontier-scale AI models — entirely local, entirely private, and blazingly fast. Happy building!

Related Posts

Expand your DGX Spark and AI knowledge with these related Collabnix guides:

• NVIDIA DGX Spark + Kubernetes: Run GPU Workloads on the GB10 Grace Blackwell Superchip — Add Kubernetes orchestration to your DGX Spark to run scalable GPU workloads.
• Building AI Agents on DGX Spark with Kubernetes: A Complete Tutorial — Deploy production-grade AI agents locally on DGX Spark using Kubernetes.
• Deploying an LLM API-Backed Microservice with Docker and CI/CD — Containerize and deploy LLM-powered services with Docker and CI/CD pipelines.
• Kubernetes for Generative AI: Complete Guide to Deploying LLMs at Scale — Scale your AI models beyond a single machine using Kubernetes infrastructure.