Here is a situation that plays out in almost every cluster running AI workloads. You have eight GPUs. Three teams share them. Someone submits a training job that asks for six GPUs, another team submits one that asks for four, and the Kubernetes scheduler places pods from both. Now each job holds some GPUs, neither has enough to start, and the hardware sits there doing nothing while both jobs wait.
The default scheduler is not at fault. It schedules pods, one at a time, and it has no concept of a job that needs all of its pods or none of them. It also has no idea that the platform team promised the research group four GPUs and the inference group four GPUs.
Kueue is the Kubernetes SIG project that fills that gap. It sits above the scheduler and decides which jobs are allowed to start, based on quotas you define. A job that cannot get everything it needs waits in a queue instead of grabbing half the cluster.
In this tutorial we will install Kueue, set up quotas for two teams, watch jobs queue and start in order, and then set up quota borrowing so idle capacity does not go to waste. Everything runs on a local cluster with no GPUs required, and there is a section showing the exact changes for real GPU nodes.
How Kueue thinks about work
The key idea is job level admission. Normally you create a Job and its pods go straight to the scheduler. With Kueue, the Job is created suspended. Kueue looks at the total resources the Job needs, checks whether the relevant quota has room, and only then unsuspends it. All the pods start, or none of them do.
There are four objects to learn.
- ResourceFlavor describes a kind of hardware. One flavor for A100 nodes, another for L4 nodes, another for plain CPU nodes. It maps to node labels and can carry taints.
- ClusterQueue holds the quota. It says how much of each resource, in each flavor, is available to the workloads that use it. This is a cluster scoped object owned by the platform team.
- LocalQueue lives in a namespace and points at a ClusterQueue. Application teams submit to their LocalQueue and never touch the ClusterQueue.
- Workload is created by Kueue for each Job it manages. You mostly read these rather than write them, and they are where you look when something is stuck.
That split matters. The platform team controls capacity in one place, and teams get a queue name to put in their manifests. Nobody needs cluster wide permissions to submit a training job.
What you need
- A Kubernetes cluster on 1.32 or newer. Kind works fine.
- kubectl
- About 30 minutes
- No GPUs. We will use CPU quota to demonstrate the behaviour, then show the GPU version.
Step 1: Install Kueue
kind create cluster --name kueue-lab
kubectl apply --server-side -f \
https://github.com/kubernetes-sigs/kueue/releases/download/v0.18.2/manifests.yaml
kubectl -n kueue-system wait --for=condition=Available \
deployment/kueue-controller-manager --timeout=300sConfirm the API version your install serves, since Kueue moved from v1beta1 to v1beta2:
kubectl api-resources | grep kueueThe examples below use kueue.x-k8s.io/v1beta2. If your version differs, adjust the apiVersion field.
Step 2: Define a ResourceFlavor
Start with the simplest possible flavor, one that matches any node. We will add a GPU specific flavor later.
cat > flavor.yaml <<'EOF'
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: default-flavor
EOF
kubectl apply -f flavor.yamlA flavor with no nodeLabels matches everything. That is fine for a lab, but in a real cluster you almost always want flavors tied to hardware so that a job asking for A100s cannot be admitted against L4 quota.
Step 3: Create ClusterQueues for two teams
Now the quota. We will give a research team and an inference team their own ClusterQueue, and put both in the same cohort so they can borrow from each other later.
cat > queues.yaml <<'EOF'
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: research-cq
spec:
namespaceSelector: {}
cohort: ai-platform
queueingStrategy: BestEffortFIFO
resourceGroups:
- coveredResources: ["cpu", "memory"]
flavors:
- name: default-flavor
resources:
- name: cpu
nominalQuota: 4
borrowingLimit: 4
- name: memory
nominalQuota: 4Gi
borrowingLimit: 4Gi
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: inference-cq
spec:
namespaceSelector: {}
cohort: ai-platform
queueingStrategy: BestEffortFIFO
resourceGroups:
- coveredResources: ["cpu", "memory"]
flavors:
- name: default-flavor
resources:
- name: cpu
nominalQuota: 4
borrowingLimit: 4
- name: memory
nominalQuota: 4Gi
borrowingLimit: 4Gi
EOF
kubectl apply -f queues.yaml
kubectl get clusterqueueThree fields are doing the interesting work.
nominalQuotais the guaranteed share. This team can always get this much.cohortputs both queues in a shared pool. Unused nominal quota in one queue becomes borrowable by the other.borrowingLimitcaps how much extra a queue can take from the cohort. Without it, one team can consume everything the moment the other is idle, and then the second team waits for jobs to finish rather than getting its guaranteed share back.
queueingStrategy is worth understanding too. BestEffortFIFO, the default, lets a small job jump ahead of a large one that does not fit yet, which keeps the cluster busy. StrictFIFO makes everything wait behind the head of the queue, which is fairer but leaves capacity idle. Most teams want BestEffortFIFO with priorities layered on top.
Step 4: Give each team a LocalQueue
kubectl create namespace research
kubectl create namespace inference
cat > localqueues.yaml <<'EOF'
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
name: team-queue
namespace: research
spec:
clusterQueue: research-cq
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
name: team-queue
namespace: inference
spec:
clusterQueue: inference-cq
EOF
kubectl apply -f localqueues.yamlBoth are called team-queue on purpose. Teams can use the same label value in their manifests regardless of which namespace they deploy to, and the platform team decides behind the scenes which ClusterQueue that maps to.
Step 5: Submit a job and watch it get admitted
A Job joins a queue through one label: kueue.x-k8s.io/queue-name. That is the whole integration.
cat > job.yaml <<'EOF'
apiVersion: batch/v1
kind: Job
metadata:
generateName: train-
namespace: research
labels:
kueue.x-k8s.io/queue-name: team-queue
spec:
parallelism: 2
completions: 2
suspend: true
template:
spec:
containers:
- name: trainer
image: registry.k8s.io/e2e-test-images/agnhost:2.53
command: ["sleep"]
args: ["60"]
resources:
requests:
cpu: "1"
memory: "512Mi"
restartPolicy: Never
EOF
kubectl create -f job.yamlNote suspend: true. Kueue will flip it to false once quota is available. If you forget it, Kueue sets it for you, but being explicit makes the intent obvious to anyone reading the manifest.
kubectl -n research get jobs
kubectl -n research get workloads
kubectl -n research get podsThe Workload object is the one to watch. It carries the admission decision and, when a job is not running, the reason why.
kubectl -n research get workloads -o wide
kubectl -n research describe workloadStep 6: Fill the quota and see queueing work
The research queue has 4 CPU of nominal quota. Submit three more jobs at 2 CPU each and the fourth will not fit.
for i in 1 2 3; do kubectl create -f job.yaml; done
kubectl -n research get workloads \
-o custom-columns='NAME:.metadata.name,QUEUE:.spec.queueName,ADMITTED:.status.conditions[?(@.type=="Admitted")].status'Some workloads show Admitted: True and are running. The rest are pending. Crucially, the pending ones have created no pods at all. They are not sitting in Pending taking up scheduler attention or partially holding resources. That is the behaviour that fixes the deadlock from the opening of this article.
Check what the queue thinks its usage is:
kubectl get clusterqueue research-cq -o yaml | grep -A20 "flavorsUsage"
kubectl get clusterqueue \
-o custom-columns='NAME:.metadata.name,PENDING:.status.pendingWorkloads,ADMITTED:.status.admittedWorkloads'Now delete a running job and watch a queued one start within a few seconds. No manual intervention, no cron job, no retry loop in your CI system.
kubectl -n research delete job $(kubectl -n research get jobs -o name | head -1)
kubectl -n research get workloads -wStep 7: Borrowing between teams
Both queues are in the ai-platform cohort, and the inference namespace is empty. So the research queue can borrow inference’s unused 4 CPU, up to its borrowingLimit.
Submit more research jobs and you will see admitted usage climb past the nominal 4 CPU. Then submit an inference job:
sed 's/namespace: research/namespace: inference/' job.yaml | kubectl create -f -
kubectl -n inference get workloadsBy default the inference job waits for borrowed capacity to be released as research jobs finish. If you want it to take its guaranteed share back immediately, add reclaim preemption to the inference queue:
kubectl patch clusterqueue inference-cq --type=merge -p '
spec:
preemption:
reclaimWithinCohort: Any
withinClusterQueue: LowerPriority'reclaimWithinCohort: Any means the inference queue can evict borrowed workloads from other queues to get back down to its nominal quota. This is the setting that makes borrowing safe to enable. Without it, “you can use idle GPUs” quietly turns into “you can be blocked for six hours by someone else’s training run”.
Preemption means eviction, so your training jobs need to checkpoint. A job that loses eight hours of work because it got preempted is worse than a job that waited. Set reclaimWithinCohort only for workloads that can restart cleanly.
Doing this with real GPUs
The changes are small. Give the flavor node labels that match your GPU nodes, and add the GPU resource to the covered resources.
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: a100-flavor
spec:
nodeLabels:
nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: research-gpu-cq
spec:
namespaceSelector: {}
cohort: ai-platform
resourceGroups:
- coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
flavors:
- name: a100-flavor
resources:
- name: cpu
nominalQuota: 96
- name: memory
nominalQuota: 768Gi
- name: "nvidia.com/gpu"
nominalQuota: 8
borrowingLimit: 4Jobs then request nvidia.com/gpu in their container resources as usual, and Kueue counts it against the quota. Separate flavors per GPU model are worth the extra objects. A job that needs 80GB of GPU memory should not be admitted against quota that is backed by L4s.
If you are on Kubernetes 1.34 or newer and using Dynamic Resource Allocation for GPUs, recent Kueue releases account for DRA claims in quota as well, so the two work together rather than competing.
Beyond batch Jobs
Kueue is not limited to batch/v1 Jobs. It has integrations for JobSet, Kubeflow training operators such as PyTorchJob and TFJob, RayJob and RayCluster, MPIJob, and plain Pods and Deployments through the pod integration. The queue-name label is the same in every case, so the mental model does not change as you add frameworks.
For multi cluster setups, MultiKueue lets a management cluster dispatch jobs to worker clusters, which is how larger organisations pool GPU capacity across regions or clouds.
Troubleshooting
Job stays suspended forever
Describe the Workload and read the conditions. The usual causes are a queue-name label pointing at a LocalQueue that does not exist, a LocalQueue pointing at a missing ClusterQueue, or a job requesting a resource that is not in coveredResources. That last one is easy to miss. If your ClusterQueue covers cpu and memory but the job requests nvidia.com/gpu, it will never be admitted.
kubectl -n research describe workload
kubectl get clusterqueue research-cq -o jsonpath='{.status.conditions}' | jqClusterQueue shows Active: False
Almost always a ResourceFlavor named in the queue does not exist. Check spelling against kubectl get resourceflavor.
Job is admitted but pods stay Pending
Kueue admitted the job against quota, but the scheduler cannot place the pods. Quota and actual capacity are different things. If you set nominalQuota higher than the hardware you really have, Kueue will happily admit work the cluster cannot run. Keep quota totals at or below real capacity.
Jobs bypass Kueue entirely
A Job without the queue-name label runs normally and is invisible to Kueue’s accounting. Once you are relying on quotas, enforce the label with an admission policy so nobody can quietly opt out of the system.
Cleanup
kind delete cluster --name kueue-lab
rm -f flavor.yaml queues.yaml localqueues.yaml job.yamlWrapping up
GPUs are the most expensive thing in most clusters, and the default scheduler was never designed to ration them between teams. Kueue adds the missing layer: jobs are admitted as a whole, quotas are explicit, and idle capacity can be lent out without giving up your guaranteed share.
If you are starting out, the useful first move is small. Create one ClusterQueue that reflects your actual GPU count, put every team’s training jobs behind it, and watch the queue depth for a couple of weeks. That number alone tends to settle a lot of arguments about whether you need to buy more hardware or just schedule what you have more carefully.
