Gateway API’s Inference Extension was one of the standout topics at recent KubeCon events, tackling a very specific pain point: generic HTTP load balancing algorithms are a poor fit for LLM traffic, where request cost varies wildly and a single long-running generation can pin a backend for seconds. This tutorial sets up the Inference Extension so routing decisions account for model identity and backend load instead of blind round robin.
Why regular load balancing falls short for LLMs
A typical Service or Ingress balances connections evenly, but an inference server serving a 4k-token completion is doing far more work than one serving a two-word answer. The Gateway API Inference Extension introduces an InferencePool and InferenceModel API so the proxy can route based on model name, priority, and real-time backend saturation.
Prerequisites
- A Kubernetes cluster running Gateway API v1.1 or later
- A Gateway API implementation that supports the extension, such as Envoy Gateway
- One or more LLM inference deployments already running (see our GPU Operator tutorial for that part)
Step 1: Install the Inference Extension CRDs
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/latest/download/manifests.yamlStep 2: Define an InferencePool
An InferencePool groups backend pods that serve the same model family and exposes a single routing target.
apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferencePool
metadata:
name: mistral-pool
spec:
selector:
app: llm-inference
targetPortNumber: 80
extensionRef:
name: mistral-endpoint-pickerStep 3: Declare an InferenceModel
The InferenceModel maps a logical model name to a pool and sets scheduling priority, which matters when multiple model versions share capacity.
apiVersion: inference.networking.x-k8s.io/v1alpha1
kind: InferenceModel
metadata:
name: mistral-7b
spec:
modelName: mistral-7b-instruct
poolRef:
name: mistral-pool
criticality: StandardStep 4: Point an HTTPRoute at the pool
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm-route
spec:
parentRefs:
- name: my-gateway
rules:
- backendRefs:
- group: inference.networking.x-k8s.io
kind: InferencePool
name: mistral-poolStep 5: Send traffic and observe routing
curl -s http://my-gateway/v1/completions \
-H "Content-Type: application/json" \
-d '{"model":"mistral-7b-instruct","prompt":"Hello"}'Behind the scenes, the extension’s endpoint picker inspects real-time queue depth and KV cache usage on each backend pod, then steers the request to the instance best able to serve it, instead of the next pod in a round-robin list.
Step 6: Tune for your workload
Adjust criticality per InferenceModel so latency-sensitive chat traffic can preempt best-effort batch jobs sharing the same pool, and watch the extension’s metrics endpoint to confirm queueing behaves as expected under load.
Wrap-up
Standard Gateway API routing treats every request as equal work, which breaks down fast for generative workloads. The Inference Extension closes that gap with model-aware, load-aware routing, and it is quickly becoming the reference pattern the CNCF community points to for running LLM traffic on Kubernetes.
