If you’ve ever added nvidia.com/gpu: 1 to a pod spec and wondered what actually happens behind the scenes, you’re not alone. GPU allocation is one of the most misunderstood parts of Kubernetes, mostly because it works quite differently from how Kubernetes handles CPU and memory. This is a simple, no-jargon explainer of how it actually works.
The Short Answer
Kubernetes doesn’t understand GPUs natively at all. Every bit of GPU awareness in your cluster comes from a small piece of vendor software called a device plugin, which tells the kubelet “this node has some GPUs, here’s how to count and hand them out.” Kubernetes itself just treats a GPU as a whole, indivisible counting unit, the same way it might count “3 available widgets” without knowing or caring what a widget actually is.
How Normal Resources Work (CPU and Memory)
When you request cpu: 500m or memory: 256Mi, Kubernetes has built-in, native understanding of what that means. It can slice a CPU core into fractions, compress memory limits, and enforce all of it directly through Linux cgroups. This is why you can ask for half a CPU core and it just works.
Why GPUs Are Different
GPUs have no such native support. Kubernetes has no built-in concept of “half a GPU” or “a GPU with 4GB of memory.” Instead, GPUs are exposed through a mechanism called extended resources, a generic system that lets any hardware vendor advertise custom countable resources (like nvidia.com/gpu) without Kubernetes needing to understand what they actually are. To the scheduler, one GPU is just an opaque integer it can add or subtract, nothing more.
The Three Moving Parts
Getting a GPU into a pod involves three cooperating pieces, each doing one specific job:
1. The device plugin runs as a DaemonSet on every GPU node (NVIDIA’s is the most common one). On startup, it detects the physical GPUs on that machine and registers them with the local kubelet, saying “I have 2 units of nvidia.com/gpu available here.”
2. The kubelet takes that information and reports it up as part of the node’s allocatable capacity, exactly like it reports available CPU and memory. Now the node “has” GPUs as far as the rest of the cluster is concerned.
3. The scheduler sees your pod requesting nvidia.com/gpu: 1, looks for a node with at least 1 unit of that resource still free, and places the pod there. Once scheduled, the kubelet asks the device plugin which specific physical GPU to hand over, injects the right device files and drivers into the container, and only then does your workload actually get GPU access.
What This Looks Like in a Pod Spec
apiVersion: v1
kind: Pod
metadata:
name: simple-gpu-pod
spec:
containers:
- name: cuda-container
image: nvidia/cuda:12.4.0-base-ubuntu22.04
resources:
limits:
nvidia.com/gpu: 1Notice there’s no requests field needed separately here — for extended resources like GPUs, the limit is also treated as the request. You can’t ask for “some” GPU and get more later; it’s whole-number allocation only, by design.
The Catch: No Fractional Requests by Default
Because a GPU is just an opaque counting unit, you cannot natively ask for half a GPU, a percentage of compute, or a specific slice of memory. If a pod requests 1 GPU, it gets exclusive use of one entire physical device, even if that workload only needs 10% of its capacity. This all-or-nothing model is exactly why the Kubernetes and CNCF communities built extra layers on top: time-slicing, MIG, and projects like HAMi exist specifically to work around this limitation and let multiple pods share a single physical GPU. We cover all of them hands-on in our GPU sharing in Kubernetes tutorial.
Putting It All Together
So the full journey of a GPU request looks like this: the device plugin tells the kubelet what GPUs exist, the kubelet reports them as extended resource capacity, the scheduler matches your pod’s request against that capacity and picks a node, and finally the kubelet and device plugin hand the actual device over to your container at startup. Kubernetes itself never inspects, understands, or manages the GPU directly, it just counts.
Where to Go From Here
Once this basic model clicks, the more advanced topics make a lot more sense. Check out our guides on sharing a single GPU across multiple pods, monitoring real GPU utilization, and autoscaling GPU nodes automatically to take this from “how it works” to “how to run it well in production.”
