Deploying LLM Inference on Kubernetes with the NVIDIA GPU Operator: A Step-by-Step Tutorial

Serving large language models efficiently is one of the dominant themes at recent KubeCon and CloudNativeCon events, and the NVIDIA GPU Operator has become the de-facto way to make GPUs first-class, schedulable resources inside Kubernetes. This tutorial walks through installing the operator and deploying a real LLM inference workload on top of it.

What the GPU Operator actually does

Running GPU workloads on Kubernetes normally means manually installing drivers, the container toolkit, and device plugins on every node. The GPU Operator packages all of that as a set of controllers and DaemonSets, so a bare node with a GPU can be turned into a cluster-ready inference node automatically.

Prerequisites

• A Kubernetes cluster with at least one node that has an NVIDIA GPU
• Helm 3 installed
• Node feature discovery enabled or installable by the operator
• Cluster-admin access to install CRDs and privileged DaemonSets

Step 1: Add the NVIDIA Helm repository

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

Step 2: Install the GPU Operator

kubectl create namespace gpu-operator
helm install gpu-operator nvidia/gpu-operator \
 -n gpu-operator \
 --set driver.enabled=true \
 --set toolkit.enabled=true

Give it a few minutes, then confirm every pod in the namespace reaches Running.

kubectl get pods -n gpu-operator -w

Step 3: Confirm the GPU is schedulable

kubectl describe node <gpu-node-name> | grep -A5 "Allocatable"

You should see nvidia.com/gpu listed as an allocatable resource. That is the signal the operator has wired up drivers, the container runtime, and the device plugin correctly.

Step 4: Deploy an LLM inference server

With GPUs now schedulable, request one explicitly in the pod spec. Here is a minimal deployment running a text-generation-inference style server.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: llm-inference
spec:
 replicas: 1
 selector:
 matchLabels: { app: llm-inference }
 template:
 metadata:
 labels: { app: llm-inference }
 spec:
 containers:
 - name: server
 image: ghcr.io/huggingface/text-generation-inference:latest
 args: ["--model-id", "mistralai/Mistral-7B-Instruct-v0.2"]
 resources:
 limits:
 nvidia.com/gpu: 1
 ports:
 - containerPort: 80

Step 5: Expose and test the model

kubectl expose deployment llm-inference --port=80 --type=ClusterIP
kubectl run tester --rm -it --image=curlimages/curl -- \
 curl -s http://llm-inference/generate \
 -X POST -d '{"inputs":"Explain Kubernetes in one sentence"}'

Step 6: Watch GPU utilization

The operator ships the DCGM exporter, which plugs straight into Prometheus. Once scraped, dashboards can show per-pod GPU utilization, memory, and temperature, which is invaluable for right-sizing inference replicas and catching idle, wasted GPUs.

kubectl port-forward -n gpu-operator svc/nvidia-dcgm-exporter 9400:9400

Wrap-up

You have gone from a bare GPU node to a fully scheduled LLM inference service in a handful of steps, with the GPU Operator handling drivers, runtime, and monitoring. From here, natural next steps include adding a Horizontal Pod Autoscaler driven by GPU utilization metrics and experimenting with time-slicing to pack multiple small models onto a single GPU.

Further Reading & Resources

• NVIDIA GPU Operator documentation
• NVIDIA GPU Operator on GitHub
• Helm documentation – the package manager used to install the operator
• Kubernetes device plugin framework
• DCGM Exporter on GitHub – GPU metrics for Prometheus
• CNCF – Cloud Native Computing Foundation