Kubernetes is a popular container orchestration platform used to deploy, scale and manage containerized applications. Kubestalk is a tool that integrates with Kubernetes to help you manage Kubernetes resources using GitOps principles. In this blog, we will provide an introduction to Kubestalk, discuss how to install it, and explore real-world use-cases with code snippets in YAML.
Introduction to Kubestalk
Kubestalk is an open-source GitOps tool for Kubernetes, developed by the Kubestalk team. It allows you to manage Kubernetes resources using GitOps principles, which means that all configuration changes are made through code stored in Git. This approach provides a number of benefits, including version control, collaboration, and automated deployments.
Source ~ https://github.com/redhuntlabs/KubeStalk
Kubestalk works by watching a Git repository for changes to configuration files, and then applying those changes to a Kubernetes cluster. This allows you to use Git as the single source of truth for your Kubernetes resources, and makes it easy to roll back changes if necessary.
Mapping the Attack Surface
Working Architecture
First things first, let us get a quick understanding of the working architecture of Kubernetes.
Kubernetes Core Architecture
A Kubernetes cluster consists of a set of worker machines, called nodes that run containerized applications. A pod is the smallest unit in Kubernetes, hosting one or more containers. A control plane/master node manages the worker nodes and the pods in the cluster.
The control plane includes components such as:
- Kube API Server: A frontend for Kubernetes that takes RESTful requests from clients and translates the requests to operations on containers, pods or nodes.
- etcd: A key-value database used as a backing store for all cluster data.
- Kube Scheduler: A watcher for newly created pods.
- Kube Controller Manager: Runs controller processes.
A few components on each node maintain the running pods and provide a runtime environment to it:
- Kubelet: Makes sure that the containers are running properly.
- Kube Proxy: Maintains network rules on nodes.
- Container runtime: Software that runs on containers (e.g. Docker).
Kubestalk Methodology
The initial idea was to cover the basic components of Kubernetes instances. However, the attack surface gradually widened during the course of research.
The scanning timeline spanned over a period of 59 hours distributed across many nodes spread across the internet.
Broken down our findings into two major parts.
- The first part covers the core components of Kubernetes, and
- Second one covers the additional attack surface which is external tools used to manage k8s clusters 🙁
The United States alone accounted for almost 57% of the total exposures, followed by Germany, Belgium, Ireland, Singapore and Japan. Findings showed around 61% of the exposures were hosted in public clouds, with Google Cloud Platform (GCP) leading the most exposure counts
Most affected versions – v1.21 and v1.22
An architectural overview of the tool is depicted in the flowchart below:
Additional Attack Surface – Kubernetes Management Components
Apart from the regular default components, we discovered developers using additional software to manage their Kubernetes infrastructure, which also poses a significant security risk if misconfigured
Let’s take a look at one component .
cAdvisor
cAdvisor is a container metric engine that provides developers with an overview of their resource usage and the performance of their containers.
The software exposes a Web UI for viewing the resource metrics. It includes system processes, CPU, memory, network and filesystem usage statistics.
Found a total of 23,101 unauthenticated cAdvisor dashboards exposed.
Deep Analysis of Compromised Clusters
The Deep Analysis of Compromised Clusters revealed the presence of adversaries deploying malicious cryptominers inside the compromised clusters.
It was discovered that many instances had traces and IoCs of the Hildegard malware. Furthermore, a script attributed to TeamTNT was found on one of the instances, which had a detection rate of zero on VirusTotal.
How script works
- The script works by initially using masscan to discover unsecured Kubernetes clusters across IP ranges by scanning for open ports.
- Once an unsecured cluster has been discovered, the script makes use of zgrab to probe for HTTP services serving the path /v1.16/version.
- If an unsecured cluster that allows anonymous access to its APIs is discovered, launching a malicious process in a hijacked container is more stealthy as it doesn’t need to pull new images or run new containers.
A Hacker’s Point of View
As an attacker, gaining code execution (RCE) in a running container through unsecured API servers or Kubelets is possible.
- Both the exec API and run APIs allow running arbitrary commands in containers.
- Once an unsecured cluster that allows anonymous access to its APIs has been discovered, launching a malicious process in a hijacked container is more stealthy as it doesn’t need to pull new images or run new containers.
- In case Kubelet’s APIs running on port 10250 allow anonymous access, an attacker can discover the pods running inside the node by visiting the /pods endpoint.
- One of the first steps would be to discover the pods running inside the node.
- A list of pods inside the container can be retrieved by visiting the /pods endpoint.
curl -sk 'https://<ip>:10250/pods' | jq -r '.items[] | "Pod: \(.metadata.name)\tNamespace: \(.metadata.namespace)\tContainer: \(.spec.containers[].name)"'
Installation of Kubestalk
KubeStalk is written in Python and requires the requests library.To install the tool, you can clone the repository to any directory:
git clone https://github.com/redhuntlabs/kubestalk
Once cloned, you need to install the requests library using python3 -m pip install requests or:
python3 -m pip install -r requirements.txt
Everything is setup and you can use the tool directly.
Command-line Arguments
A list of command line arguments supported by the tool can be displayed using the -h flag.
$ python3 kubestalk.py -h
+---------------------+
| K U B E S T A L K |
+---------------------+ v0.1
[!] KubeStalk by RedHunt Labs - A Modern Attack Surface (ASM) Management Company
[!] Author: 0xInfection (RHL Research Team)
[!] Continuously Track Your Attack Surface using https://redhuntlabs.com/nvadr.
usage: ./kubestalk.py <url(s)>/<cidr>
Required Arguments:
urls List of hosts to scan
Optional Arguments:
-o OUTPUT, --output OUTPUT
Output path to write the CSV file to
-f SIG_FILE, --sig-dir SIG_FILE
Signature directory path to load
-t TIMEOUT, --timeout TIMEOUT
HTTP timeout value in seconds
-ua USER_AGENT, --user-agent USER_AGENT
User agent header to set in HTTP requests
--concurrency CONCURRENCY
No. of hosts to process simultaneously
--verify-ssl Verify SSL certificates
--version Display the version of KubeStalk and exit.
Basic Usage
To use the tool, you can pass one or more hosts to the script.
A basic usage is as below:
$ python3 kubestalk.py https://███.██.██.███:10250
+---------------------+
| K U B E S T A L K |
+---------------------+ v0.1
[!] KubeStalk by RedHunt Labs - A Modern Attack Surface (ASM) Management Company
[!] Author: 0xInfection (RHL Research Team)
[!] Continuously Track Your Attack Surface using https://redhuntlabs.com/nvadr.
[+] Loaded 10 signatures to scan.
[*] Processing host: https://███.██.██.██:10250
[!] Found potential issue on https://███.██.██.██:10250: Kubernetes Pod List Exposure
[*] Writing results to output file.
[+] Done.
Conclusion
Kubestalk is a powerful tool for identifying security vulnerabilities in Kubernetes clusters and the external tools used to manage them.
Its various modules enable security teams to assess different areas of a Kubernetes environment, including the nodes, containers, and network.
Additionally, Kubestalk can detect various types of threats, such as misconfigurations, privilege escalations, and malware.
By using Kubestalk, organizations can proactively identify and remediate security risks in their Kubernetes clusters, ultimately improving the overall security posture of their infrastructure.
It is important to note that Kubestalk should not be the only security tool used and that regular security assessments should be conducted to ensure the ongoing protection of Kubernetes environments.
