Table of Contents

Karpenter is an open-source provisioning tool for Kubernetes that helps manage the creation and scaling of worker nodes in a cluster. It provides a flexible and customizable way to define the constraints for node creation and pod scheduling. One of the key components of Karpenter is the Provisioner, which allows you to set specific requirements and configurations for node provisioning. In this blog post, we’ll explore how to use Karpenter Provisioner and provide examples in YAML and code.

Installing Karpenter

To install Karpenter and Karpenter Provisioner, you’ll need to follow a few steps. Here’s a general overview of the installation process:

Set up a Kubernetes cluster

Ensure that you have a functioning Kubernetes cluster available. You can use a managed Kubernetes service like Amazon EKS, Google Kubernetes Engine (GKE), or deploy your own cluster using tools like kops or kubeadm.

Install Karpenter controller

Karpenter consists of a controller that runs as a Kubernetes deployment. The controller is responsible for managing the provisioning and scaling of worker nodes. To install the Karpenter controller, you can use the following kubectl command:

kubectl apply -f https://github.com/awslabs/karpenter/releases/latest/download/releases.yaml

This command will deploy the Karpenter controller in the karpenter namespace of your Kubernetes cluster.

Configure the cloud provider

Karpenter requires integration with your cloud provider to provision worker nodes. Depending on the cloud provider you are using, you’ll need to configure the appropriate cloud provider-specific resources. Refer to the Karpenter documentation for detailed instructions on configuring the cloud provider.

Create and configure Karpenter Provisioners

Karpenter Provisioners allow you to define specific constraints and configurations for node provisioning. You can create multiple Provisioners with different settings for different workloads. To create a Provisioner, you’ll need to define a Kubernetes custom resource of kind Provisioner. You can use YAML configuration files to define the Provisioners. Here’s an example YAML for a Provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: my-provisioner
spec:
  # Configuration options for the Provisioner

Save the YAML configuration in a file (e.g., provisioner.yaml) and apply it to your cluster using the following command:

kubectl apply -f provisioner.yaml

Repeat this step for each Provisioner you want to create.

Use the Provisioners in pod specifications

To utilize the Provisioners you’ve created, you can include the provisioner field in your pod specifications. Specify the name of the Provisioner you want to use for each pod. Here’s an example pod specification using a specific Provisioner:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  provisioner:
    name: my-provisioner
    namespace: karpenter
  containers:
    - name: my-container
      image: nginx
      resources:
        requests:
          cpu: "1"
          memory: "1Gi"

In the above example, the pod will use the Provisioner named “my-provisioner” that was created earlier.

Verify and monitor Karpenter: You can monitor the status of Karpenter using standard Kubernetes tools like kubectl or by checking the Karpenter logs. You can also monitor the cluster and observe the provisioning and scaling of worker nodes.

Setting up the Default Provisioner

When you first install Karpenter, you’ll need to set up a default Provisioner. This Provisioner will be used if no other Provisioners are configured or matched for a specific pod. The default Provisioner sets constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes. Let’s take a look at an example YAML configuration for the default Provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  # References cloud provider-specific custom resource, see your cloud provider specific documentation
  providerRef:
    name: default

  # Provisioned nodes will have these taints
  taints:
    - key: example.com/special-taint
      effect: NoSchedule

  # Provisioned nodes will have these taints, but pods do not need to tolerate these taints
  startupTaints:
    - key: example.com/another-taint
      effect: NoSchedule

  # Labels are arbitrary key-values that are applied to all nodes
  labels:
    billing-team: my-team

  # Annotations are arbitrary key-values that are applied to all nodes
  annotations:
    example.com/owner: "my-team"

  # Requirements that constrain the parameters of provisioned nodes
  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["c", "m", "r"]
    - key: "karpenter.k8s.aws/instance-cpu"
      operator: In
      values: ["4", "8", "16", "32"]
    # ... other requirements ...

  # Kubelet configuration options
  kubeletConfiguration:
    clusterDNS: ["10.0.1.100"]
    containerRuntime: containerd
    # ... other kubelet configuration options ...

  # Resource limits for the cluster
  limits:
    resources:
      cpu: "1000"
      memory: 1000Gi

  # Other features and settings
  consolidation:
    enabled: true
  ttlSecondsUntilExpired: 2592000
  ttlSecondsAfterEmpty: 30
  weight: 10

In the above example, we define the default Provisioner with various settings and configurations. Let’s break down the different sections:

providerRef: This references the cloud provider-specific custom resource. You’ll need to consult your cloud provider’s documentation for the specific details.
taints: These are the taints applied to the nodes that are provisioned by Karpenter. Pods need to tolerate these taints in order to be scheduled on the nodes.
startupTaints: These taints are applied to the nodes initially, but pods do not need to tolerate them. They are assumed to be temporary and will be removed by some other entity.
labels and annotations: These are arbitrary key-value pairs that are applied to all nodes created by Karpenter. They can be used for organizing and identifying nodes.
requirements: These are the requirements that constrain the parameters of provisioned nodes. In the example, we specify constraints on instance categories and CPU values. You can add other requirements as needed.
kubeletConfiguration: This section allows you to configure various options for the kubelet running on the provisioned nodes, such as cluster DNS and the container runtime.
limits: These are the resource limits for the cluster as a whole. In the example, we set limits on CPU and memory resources.
consolidation, ttlSecondsUntilExpired, ttlSecondsAfterEmpty, and weight: These are additional features and settings that can be configured for the Provisioner.

Creating Custom Provisioners

In addition to the default Provisioner, Karpenter allows you to create custom Provisioners for specific pods or workloads. Custom Provisioners can have their own unique requirements, taints, labels, annotations, and other configurations. Let’s see an example YAML configuration for a custom Provisioner:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: custom-provisioner
spec:
  # Constraints specific to this Provisioner
  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["c4", "c5"]

  # Other configurations specific to this Provisioner
  taints:
    - key: example.com/custom-taint
      effect: NoSchedule

  labels:
    environment: production

In this example, we define a custom Provisioner named “custom-provisioner” with specific constraints and configurations. The requirements constrain the provisioned nodes to instance categories “c4” and “c5”. The Provisioner also applies a custom taint, “example.com/custom-taint”, to the nodes, requiring pods to tolerate it in order to be scheduled. Additionally, we add a label “environment” with the value “production” to the provisioned nodes.

Using the Provisioners

To use the Provisioners defined above, you can include them in your pod specifications. Here’s an example YAML for a pod that uses a specific Provisioner:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  provisioner:
    name: custom-provisioner
    namespace: karpenter
  containers:
    - name: my-container
      image: nginx
      resources:
        requests:
          cpu: "1"
          memory: "1Gi"

In the above example, we specify the Provisioner for the pod using the provisioner field. We set the name of the Provisioner to “custom-provisioner” and optionally specify the namespace where the Provisioner is defined. The rest of the pod specification follows the standard Kubernetes pod format.

Conclusion

Karpenter Provisioner provides a powerful way to manage the creation and scaling of worker nodes in a Kubernetes cluster. By defining specific constraints and configurations, you can control the characteristics of the nodes and the pods that run on them. In this blog post, we explored how to set up the default Provisioner and create custom Provisioners, along with examples in YAML. With Karpenter Provisioner, you can optimize resource allocation and improve the efficiency of your Kubernetes clusters.

Introduction to Karpenter Provisioner