Kubernetes GPU Autoscaling: A Complete Hands-On Guide with Karpenter

GPU nodes are the single most expensive line item in most AI infrastructure bills, often 10-20x the cost of a comparable CPU instance. Leaving them running around the clock “just in case” is one of the fastest ways to burn budget. This tutorial walks through autoscaling GPU node pools in Kubernetes with Karpenter, the CNCF project that has become the de facto standard for fast, cost-aware node provisioning, and compares it against the older Cluster Autoscaler along the way.

Karpenter vs. Cluster Autoscaler: The Short Version

Cluster Autoscaler works within pre-defined node groups and scales them up or down based on pending pods, which is reliable but slow and inflexible when it comes to picking the right GPU instance type. Karpenter skips node groups entirely: it provisions right-sized nodes directly from the cloud provider’s API in response to unschedulable pods, and just as aggressively consolidates and terminates underused nodes. For GPU workloads specifically, that means Karpenter can pick the cheapest available instance type and zone that satisfies a pod’s GPU request (including spot capacity) instead of you pre-guessing an instance type per node group.

Prerequisites

You’ll need an EKS (or similar cloud-managed) cluster with IAM permissions configured for node provisioning, Helm 3, and kubectl access. This walkthrough uses AWS terminology, but the same NodePool/NodeClass pattern applies to Karpenter on other supported clouds.

Step 1: Install Karpenter

helm registry logout public.ecr.aws
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --namespace kube-system \
  --set settings.clusterName=my-gpu-cluster \
  --set settings.interruptionQueue=my-gpu-cluster \
  --wait

kubectl get pods -n kube-system -l app.kubernetes.io/name=karpenter

Step 2: Define a GPU NodePool

A NodePool tells Karpenter which instance shapes it’s allowed to launch and under what constraints:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-pool
spec:
  template:
    metadata:
      labels:
        workload-type: gpu
    spec:
      requirements:
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values: ["g5", "p4d", "p5"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-node-class
  limits:
    nvidia.com/gpu: 16
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m

Step 3: Define the EC2NodeClass

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-node-class
spec:
  amiFamily: AL2
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-gpu-cluster
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: my-gpu-cluster
  role: KarpenterNodeRole-my-gpu-cluster

kubectl apply -f gpu-nodepool.yaml
kubectl apply -f gpu-nodeclass.yaml

Step 4: Trigger a Scale-Up With a GPU Workload

apiVersion: v1
kind: Pod
metadata:
  name: gpu-burst-job
spec:
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  containers:
  - name: cuda-test
    image: nvidia/cuda:12.4.0-base-ubuntu22.04
    command: ["sleep", "600"]
    resources:
      limits:
        nvidia.com/gpu: 1

kubectl apply -f gpu-burst-job.yaml
kubectl get nodes -w

Within roughly a minute you should see a brand-new GPU node appear, sized and priced according to whatever fits the NodePool’s constraints, with the pod scheduled onto it as soon as it’s ready.

Step 5: Verify Scale-Down and Consolidation

Delete the job and watch what happens next:

kubectl delete pod gpu-burst-job
kubectl get nodes -w

After the consolidateAfter window (5 minutes in the example above), Karpenter terminates the now-empty GPU node automatically. This is the core of the cost story: you stop paying for GPU capacity the moment nobody needs it, without any manual node group resizing.

Step 6: Combine With GPU Sharing and Monitoring

Autoscaling solves the “too many idle nodes” problem, but it works best alongside the GPU-sharing techniques (time-slicing, MIG, HAMi, DRA) from our GPU sharing tutorial and the observability stack from our GPU monitoring tutorial. Feed the idle-GPU alerts from DCGM Exporter into your consolidation policy, and use fractional sharing to pack more workloads onto fewer nodes before Karpenter ever needs to provision a new one.

Conclusion

Karpenter turns GPU node scaling from a static, over-provisioned node-group problem into a demand-driven one: nodes appear when pods need them and disappear the moment they don’t. Paired with GPU sharing and utilization-based monitoring, it’s currently the most cost-effective combination available for running GPU workloads on Kubernetes.