Kubernetes is a powerful container orchestration platform that allows users to deploy and manage containerized applications efficiently. However, the health of the nodes in a Kubernetes cluster is crucial for the overall stability and reliability of the applications running on it. Node problems, such as hardware failures, kernel issues, or container runtime problems, can impact the availability of pods and disrupt the entire cluster.
To address this, Kubernetes offers a tool called node-problem-detector, which aims to detect and report various node problems to the cluster management stack. In this blog, we will explore node-problem-detector, its features, how to deploy it in a Kubernetes cluster, and real-world use-cases with code snippets.
What is node-problem-detector?
Background and Motivation
Node problems in a Kubernetes cluster can lead to application disruptions and impact user experience. Issues like hardware failures, kernel panics, or unresponsive container runtimes are challenging to detect early and remediate. The node-problem-detector tool aims to address this problem by making various node problems visible to the upstream layers in the cluster management stack.
Problem API
node-problem-detector uses two mechanisms to report problems to the Kubernetes API server: Event and NodeCondition. Permanent problems that make the node unavailable for pods are reported as NodeConditions, while temporary problems that have limited impact on pods but are informative are reported as Events.
Supported Problem Daemons
node-problem-detector consists of multiple problem daemons, each responsible for monitoring specific kinds of node problems. The supported problem daemon types include System Log Monitor, System Stats Monitor, Custom Plugin Monitor, and Health Checker.
How node-problem-detector Works?
System Log Monitor
The System Log Monitor is a crucial component of node-problem-detector that monitors system logs and reports problems and metrics according to predefined rules. It collects log data from various sources, including kernel logs, system logs, and container runtime logs.
Code Snippet: Configuring System Log Monitor
node-problem-detector --config.system-log-monitor=config/kernel-monitor.json,config/system-monitor.json
System Stats Monitor
The System Stats Monitor collects various health-related system stats as metrics to provide insights into the node’s health status. Although it is not fully supported yet, it’s a promising feature for future releases.
Custom Plugin Monitor
The Custom Plugin Monitor allows users to define and check various node problems using custom check scripts. This flexibility enables users to address node problems specific to their use-cases.
Health Checker
The Health Checker verifies the health of essential components in the node, such as the kubelet and container runtime. It ensures these components are functioning correctly and reports any issues detected.
Exporter
The Exporter is responsible for reporting node problems and metrics to certain backends. Supported exporters include the Kubernetes exporter, Prometheus exporter, and Stackdriver exporter.
Building and Deploying node-problem-detector
Deploying with Helm
Helm simplifies the deployment of node-problem-detector in a Kubernetes cluster.
Code Snippet: Deploying with Helm
helm repo add deliveryhero https://charts.deliveryhero.io/
helm install --generate-name deliveryhero/node-problem-detector
Manual Installation
For manual installation, you can use YAML manifests to deploy node-problem-detector in your cluster.
Code Snippet: Manual Installation
- Edit node-problem-detector.yaml to fit your environment. Set log volume to your system log directory (used by SystemLogMonitor). You can use a ConfigMap to overwrite the config directory inside the pod.
- Edit node-problem-detector-config.yaml to configure node-problem-detector.
- Edit rbac.yaml to fit your environment.
Create the ServiceAccount and ClusterRoleBinding with:
kubectl create -f rbac.yaml
- Create the ConfigMap with:
kubectl create -f node-problem-detector-config.yaml
- Create the DaemonSet with:
kubectl create -f node-problem-detector.yaml
- Apply required manifests
kubectl create -f node-problem-detector-config.yaml
kubectl create -f rbac.yaml
kubectl create -f node-problem-detector.yaml
Configuration and Usage:
Command Line Flags
node-problem-detector provides various command line flags to configure its behavior.
Code Snippet: Using Command Line Flags
node-problem-detector --hostname-override=my-node --enable-k8s-exporter
Configuring System Log Monitor
You can specify the paths to system log monitor configuration files using the –config.system-log-monitor flag.
Code Snippet: Configuring System Log Monitor
node-problem-detector --config.system-log-monitor=config/kernel-monitor.json,config/filelog-monitor.json
Configuring System Stats Monitor
System Stats Monitor is still under development, but it will allow you to collect various health-related system stats as metrics.
Configuring Custom Plugin Monitor
The Custom Plugin Monitor can be configured with a list of paths to custom plugin monitor configuration files.
Code Snippet: Configuring Custom Plugin Monitor
node-problem-detector --config.custom-plugin-monitor=config/custom-plugin-monitor.json
Enabling Kubernetes Exporter
By default, node-problem-detector exports node problems to the Kubernetes API server. You can disable it using the –enable-k8s-exporter=false flag.
Code Snippet: Enabling Kubernetes Exporter
node-problem-detector --enable-k8s-exporter=false
Prometheus Exporter Configuration
The Prometheus exporter reports node problems and metrics locally as Prometheus metrics.
Code Snippet: Prometheus Exporter Configuration
node-problem-detector --prometheus-port=20257
Stackdriver Exporter Configuration
The Stackdriver exporter reports node problems and metrics to the Stackdriver Monitoring API.
Code Snippet: Stackdriver Exporter Configuration
node-problem-detector --exporter.stackdriver=config/stackdriver-exporter.json
Conclusion
Node-problem-detector is a valuable tool for monitoring node health in Kubernetes clusters. By making node problems visible to the cluster management stack, it enables administrators to detect and address issues before they impact applications. In this blog, we explored the features of node-problem-detector, how to deploy it, and real-world use-cases. Armed with this knowledge, you can enhance the reliability and stability of your Kubernetes clusters and ensure seamless application deployment.
