GPU Monitoring in Kubernetes: A Complete Hands-On Guide with DCGM Exporter, Prometheus & Grafana

Most Kubernetes clusters running GPU workloads have no visibility into whether those expensive GPUs are actually being used. Industry surveys consistently find average GPU utilization sitting well below 50% in shared clusters, yet teams keep provisioning more hardware because they can’t see where the waste is. This hands-on guide sets up full GPU observability on Kubernetes using NVIDIA’s DCGM Exporter, Prometheus, and Grafana, then shows how to turn that data into real cost savings.

Why GPU Monitoring Matters More Than Ever

Whether you’re using plain device plugins, time-slicing, MIG, or HAMi to share GPUs (see our companion tutorial on GPU sharing in Kubernetes), none of those techniques tell you whether the GPU is actually busy. Without per-pod utilization, memory, temperature, and power metrics, you’re flying blind on both performance and cost. DCGM (Data Center GPU Manager) is NVIDIA’s official telemetry agent, and DCGM Exporter packages it as a Prometheus-compatible metrics endpoint that runs as a DaemonSet on every GPU node.

Prerequisites

You’ll need a Kubernetes cluster with GPU nodes (ideally with the NVIDIA GPU Operator or drivers already installed), Helm 3, and an existing Prometheus + Grafana stack (the kube-prometheus-stack Helm chart is the fastest way to get one if you don’t already have it).

Step 1: Install DCGM Exporter

If you’re running the full NVIDIA GPU Operator, DCGM Exporter is already included and you can skip to Step 2. Otherwise, install it standalone:

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  -n gpu-operator --create-namespace

Verify the metrics endpoint

kubectl get pods -n gpu-operator -l app=dcgm-exporter
kubectl port-forward -n gpu-operator svc/dcgm-exporter 9400:9400

Open http://localhost:9400/metrics in a browser or curl it — you should see raw metrics like DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED for each GPU on the node.

Step 2: Configure Prometheus to Scrape DCGM Metrics

If you’re using the Prometheus Operator (kube-prometheus-stack), create a ServiceMonitor so Prometheus auto-discovers the exporter:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  namespace: gpu-operator
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  endpoints:
  - port: metrics
    interval: 15s
  namespaceSelector:
    matchNames:
    - gpu-operator

kubectl apply -f dcgm-servicemonitor.yaml

Confirm Prometheus is scraping

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090

In the Prometheus UI, go to Status > Targets and confirm the dcgm-exporter target shows as UP. Then try a query like DCGM_FI_DEV_GPU_UTIL in the Graph tab to see live per-GPU utilization.

Step 3: Visualize with Grafana

NVIDIA publishes an official community dashboard for DCGM metrics on Grafana’s dashboard library (search for “NVIDIA DCGM Exporter Dashboard”, ID 12239). Import it directly:

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Log into Grafana, go to Dashboards > Import, and enter dashboard ID 12239, then select your Prometheus data source. You’ll immediately get panels for GPU utilization %, framebuffer memory used, temperature, power draw, and SM (streaming multiprocessor) clock speed, broken out per GPU and per node.

Step 4: Alert on Idle and Overheating GPUs

Turn the raw metrics into actionable alerts. A simple idle-GPU alert in a PrometheusRule looks like this:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gpu-alerts
  namespace: monitoring
spec:
  groups:
  - name: gpu.rules
    rules:
    - alert: GPUIdleTooLong
      expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[6h]) < 5
      for: 30m
      labels:
        severity: warning
      annotations:
        summary: "GPU on {{ $labels.instance }} has been under 5% utilization for 6+ hours"
    - alert: GPUOverheating
      expr: DCGM_FI_DEV_GPU_TEMP > 85
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "GPU on {{ $labels.instance }} is running above 85°C"

kubectl apply -f gpu-alerts.yaml

The idle-GPU alert is the one that pays for this whole setup: a GPU sitting under 5% utilization for six hours straight is a very strong signal you’re paying for capacity nobody is using.

Step 5: Turn Utilization Data Into Cost Savings

Once utilization is visible, cost optimization becomes an engineering problem instead of a guessing game. Correlate the Grafana utilization panels against your GPU-hour billing (cloud invoice or on-prem depreciation) to calculate an effective utilization rate. Feed idle-GPU alerts into your scale-down policy so the cluster autoscaler or node pool can reclaim genuinely idle nodes instead of just idle pods. If you’re running shared GPUs via time-slicing or HAMi, use the per-process memory and utilization breakdown to spot noisy-neighbor pods that are starving others on the same physical device. Tools like Kubecost or OpenCost can layer cost-per-namespace reporting on top of these same Prometheus metrics if you need chargeback numbers for finance.

Conclusion

DCGM Exporter, Prometheus, and Grafana together give you the same GPU observability stack that NVIDIA itself recommends, and it plugs directly into whatever GPU-sharing strategy you’ve already deployed. The combination of idle-GPU alerting and per-pod utilization breakdowns is consistently the fastest path teams find to cutting GPU spend without touching a single workload’s code.