Join our Discord Server
Collabnix
Adesoji Alu Adesoji brings a proven ability to apply machine learning(ML) and data science techniques to solve real-world problems. He has experience working with a variety of cloud platforms, including AWS, Azure, and Google Cloud Platform. He has a strong skills in software engineering, data science, and machine learning. He is passionate about using technology to make a positive impact on the world.

Kube-green: An Ultimate Method to Automatically Reduce Kubernetes Resource Waste

6 min read

How many of your development or preview pods stay running during weekends or at night? It’s a waste of resources and money! But fear not, kube-green is here to the rescue.

kube-greenis a simple Kubernetes addon that automatically shuts down (some of) your resources when you don’t need them. If you’re already using kube-green, consider adding yourself as an adopter!

Kube-green Image

Table of Contents

  • Getting Started
    • Prerequisites
    • Installation
  • Running Tests
    • Unit Tests
    • Integration Tests
  • Deployment
    • Local Deployment with KinD
  • Usage
    • Configuring the SleepInfo CRD
    • CRD Examples
  • How kube-green Works
    • Resources Handled
    • Sleep and Wake Up Mechanism
    • State Preservation
  • Installation Guide
    • Prerequisites
    • kubectl Apply
    • Customization with kustomize
    • Operator Lifecycle Manager (OLM)
  • Cloud Compatibility
    • GKE
    • AWS EKS
  • Configuration Details
    • Examples
  • Metrics and Savings
    • Small Cluster Example
    • Numeric Summary
  • Conclusion
  • References

    • Getting Started

    These instructions will help you set up kube-green on your local machine for development and testing purposes. For installation on a live system, refer to our documentation.

    Empower

    Prerequisites

    • Go: Ensure you have Go installed (version 1.19 or higher). Download Go

    Installation

    Clone the Kube-green repository and install the dependencies: 
    
git clone https://github.com/kube-green/kube-green.git

cd kube-green

go get

    Running Tests

    kube-green comes with a suite of tests to ensure everything runs smoothly.

    Unit Tests

    Run all unit tests with: 
    

make test

    Integration Tests

    To run integration tests: 
    
make e2e-test

    Deployment

    For development purposes, you can deploy kube-green locally using KinD and ko.

    Local Deployment with KinD

    Start a KinD Cluster: 
    
kind create cluster --name kube-green-development
    Deploy kube-green using ko: 
    
make local-run clusterName=kube-green-development

    Usage

    Using kube-green is straightforward. Once installed on the cluster, you configure it using the SleepInfo Custom Resource Definition (CRD).

    Configuring the SleepInfo CRD

    The SleepInfo CRD allows you to specify sleep and wake-up times for namespaces. Here’s how you can define it.

    CRD Examples

    Example 1: Pods running during working hours with Europe/Rome timezone, suspend CronJobs, and exclude a deployment named api-gateway: 
            

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: working-hours
spec:
  weekdays: "1-5"
  sleepAt: "20:00"
  wakeUpAt: "08:00"
  timeZone: "Europe/Rome"
  suspendCronJobs: true
  excludeRef:
    - apiVersion: "apps/v1"
      kind: Deployment
      name: api-gateway
    Example 2: Pods sleep every night without restore: 
            

apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: nightly-shutdown
spec:
  sleepAt: "20:00"
  timeZone: "Europe/Rome"
  weekdays: "*"
    For more examples, refer to the kube green documentation.

    How kube-green Works

    Kube-green Works
    Once installed, kube-green uses the SleepInfo CRD to manage the sleep and wake-up cycles of your Kubernetes resources.

    Resources Handled

    kube-green currently handles:
    • Deployments
    • CronJobs
    By default, kube-green stops Deployment resources during sleep (unless excluded). To suspend CronJobs, set suspendCronJobs to true in your SleepInfo configuration.

    Sleep and Wake Up Mechanism

    • Sleep:
      • Deployments: Replicas are set to 0.
      • CronJobs: The suspend field is set to true.
    • Wake Up:
      • Deployments: Replicas are restored to their original count before sleep.
      • CronJobs: The suspend field is restored to its original value.

    State Preservation

    The state of resources before sleep is saved in a Secret with the name of the SleepInfo in the respective namespace. Understanding Secrets in Kubernetes

    Installation Guide

    Prerequisites

    • Kubernetes Version: Supports versions >= 1.23 and <= 1.30.
    • OpenShift: Compatible with OpenShift Container Platform v4.
    • Architectures: Supports amd64 and arm64.
    • Certificates: Requires certificates for webhooks. Using cert-manager is recommended.

    kubectl Apply

    To install the latest release with default settings: 
            

kubectl apply -f https://github.com/kube-green/kube-green/releases/latest/download/kube-green.yaml
    To install a specific release version: 
            

kubectl apply -f https://github.com/kube-green/kube-green/releases/download/${RELEASE_TAG}/kube-green.yaml
    Replace ${RELEASE_TAG} with the desired version tag. Find the latest kube-green releases here

    Customization with kustomize

    For custom configurations, clone the repository and modify the config files as needed. For example, to deploy kube-green in a different namespace, edit the kustomization.yaml file. Deploy with: 
            
make deploy

    Operator Lifecycle Manager (OLM)

    Install from OperatorHub

    kube-green is available on OperatorHub.io. Click the Install button and follow the instructions.

    Install Kube-green Operator on OpenShift

    Red Hat® OpenShift® is a critical component of creating a secure cloud-native development environment. OpenShift 4 is the best enterprise platform for building production-ready applications today and for the decade ahead. On OpenShift 4, you can install kube-green from the OperatorHub under the community operators catalog. Follow the OpenShift guide for Installation.

    Cloud Compatibility

    GKE

    For GKE private clusters, you need to configure an additional firewall rule to allow the GKE control plane to access your webhook pod (exposed on port 9443 by default). This is necessary for the webhook component to function correctly. Learn how to add firewall rules to GKE.

    AWS EKS

    When using a custom CNI (like Cilium) on EKS, set hostNetwork: true in the webhook deployment to ensure the control plane can reach the webhook.

    To apply this setting, uncomment the host_network_patch.yaml line in the kustomization file.

    Configuration Details

    Apply the SleepInfo CRD in the namespace where you want to enable kube-green. The SleepInfo spec includes:
    • weekdays: Days of the week (for every day, 1 for Monday, 1-5 for Monday to Friday).
    • sleepAt: Time (HH:mm) when the namespace goes to sleep.
    • wakeUpAt (optional): Time (HH:mm) when the namespace wakes up.
    • timeZone (optional): Time zone in IANA format (e.g., Europe/Rome).
    • suspendDeployments (optional, default true): Whether to suspend Deployments.
    • suspendCronJobs (optional, default false): Whether to suspend CronJobs.
    • excludeRef (optional): Resources to exclude from sleep, specified by apiVersion, kind, name, or matchLabels.

    Examples

    Simplest SleepInfo Resource 
            
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: basic-schedule
spec:
  weekdays: "1-5"
  sleepAt: "20:00"
  wakeUpAt: "08:00"
  timeZone: "Europe/Rome"
    Complete SleepInfo Resource 
            
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: full-schedule
spec:
  weekdays: "1-5"
  sleepAt: "20:00"
  wakeUpAt: "08:00"
  timeZone: "Europe/Rome"
  suspendCronJobs: true
  excludeRef:
    - apiVersion: "apps/v1"
      kind: Deployment
      name: api-gateway
    SleepInfo Without Wake Up 
            
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: sleep-only
spec:
  weekdays: "1-5"
  sleepAt: "20:00"
  timeZone: "Europe/Rome"
  suspendCronJobs: true
  excludeRef:
    - apiVersion: "apps/v1"
      kind: Deployment
      name: api-gateway
    Suspend Only CronJobs 
            
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: cronjob-suspend
spec:
  weekdays: "*"
  sleepAt: "20:00"
  wakeUpAt: "08:00"
  suspendCronJobs: true
  suspendDeployments: false
  excludeRef:
    - apiVersion: "batch/v1"
      kind: CronJob
      name: do-not-suspend
    Exclude Resources by Labels 
            
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: label-exclusion
spec:
  weekdays: "*"
  sleepAt: "20:00"
  wakeUpAt: "08:00"
  suspendCronJobs: true
  excludeRef:
    - matchLabels:
        kube-green.dev/exclude: "true"

    Metrics and Savings

    Small Cluster Example

    In a small development cluster with 15 namespaces (kube-green enabled in the 3 largest), we used the following configuration: 
            
apiVersion: kube-green.com/v1alpha1
kind: SleepInfo
metadata:
  name: working-hours
spec:
  weekdays: "1-5"
  sleepAt: "20:00"
  wakeUpAt: "08:00"
  timeZone: "Europe/Rome"
  excludeRef:
    - apiVersion: "apps/v1"
      kind: Deployment
      name: api-gateway
    Memory Usage Before and After kube-green
    Memory usage before
    CPU Usage Before and After kube-green
    After

    Numeric Summary

    Total With kube-green Difference
    Number of Pods 446 205 241
    Memory Consumed [GB] 28 16 12
    CPU Consumed [cores] 2 1.3 0.7
    Memory Allocated [GB] 34 22 12
    CPU Allocated [cores] 14 12 2
    CO₂e Savings per Week [kg] 94 66 28

    By using kube-green, we saved 28 kg of CO₂e per week for this cluster. Over a year (52 weeks), that’s a reduction of 1,456 kg CO₂e.

    Conclusion

    As demonstrated, kube-green significantly reduces resource usage in Kubernetes clusters by automatically shutting down resources when they’re not needed. This not only saves costs but also reduces the environmental impact of running Kubernetes workloads. For larger clusters, the savings can lead to node downscaling, further amplifying the benefits.

    Note: kube-green exposes Prometheus metrics available on port 8080 at the /metrics path. You can change the metrics port using the metrics-bind-address startup variable. Exposed Metrics:
    • current_sleepinfo: Gauge metric providing information about the SleepInfo resources, including their names and namespaces.
    Ready to start saving resources and money? Install kube-green today and contribute to a greener Kubernetes ecosystem!

    References

    Have Queries? Join https://launchpass.com/collabnix

    Adesoji Alu Adesoji brings a proven ability to apply machine learning(ML) and data science techniques to solve real-world problems. He has experience working with a variety of cloud platforms, including AWS, Azure, and Google Cloud Platform. He has a strong skills in software engineering, data science, and machine learning. He is passionate about using technology to make a positive impact on the world.
    «
    »

    Understanding DevOps, its benefits, and best practices

    Tanvir Kour
    Oct 26, 2024 3 min read

    How to Integrate Kubernetes with Google Cloud Platform(GCP)

    Abraham Dahunsi
    Oct 23, 2024 2 min read

    Mastering Kubernetes Scaling: From Manual Adjustments to Intelligent Automation…

    Adesoji Alu
    Oct 16, 2024 7 min read
    Join our Discord Server
    Table of Contents
    → Index