Join our Discord Server
Abraham Dahunsi Web Developer 🌐 | Technical Writer ✍️| DevOps Enthusiast👨‍💻 | Python🐍 |

Monitoring Containerd

5 min read

Containerd is an open-source container runtime used in Kubernetes to manage low-level tasks like creating and managing sandboxes, image transfers, and container execution. Containerd plays a crucial role in efficiently creating, managing, and running containers in production environments.

Containerd is very paramount for simplifying the management of containers, and that is why you must monitor Containerd. Monitoring Containerd involves tracking and analyzing different important metrics such as runtime performance, resource utilization, and health so that you can ensure efficient operation and mitigate any issues that may come up.

Next, we’ll discuss the importance of monitoring Containerd, understand its architecture, and explore the different tools for monitoring.

Understanding Containerd Architecture

Containerd architecture is made up of a core runtime engine responsible for managing containers and processes. It exposes a GRPC API for interaction with clients, facilitating container and image operations. Containerd ensures compatibility with OCI specifications for seamless integration with other OCI-compliant runtimes and container formats.

Its modular design allows for other additional functionalities by integrating plugins. Pulling and pushing container images from registries is allowed because built-in support for image distribution is enabled.

Containerd Runtimes

Containerd can be integrated with container runtimes like runc as part of its architecture. These runtimes help execute and manage containers based on the specifications defined by the Open Container Initiative (OCI).

Since Containerd supports multiple runtimes, you can choose a runtime that best fits your requirements. Runtimes like runc are OCI-compliant and provide a lightweight and secure environment for running containers, making them a popular choice for many deployments.

Since Containerd runtime integration allows seamless interaction between Containerd and the chosen runtime, containers can be managed efficiently without a shortage of resources and functionalities provided by your chosen runtime.

Containerd Plugins

As stated earlier, Containerd’s architecture is designed to be modular and extensible, allowing you to enhance its capabilities with plugins. One of the key benefits of using plugins is that they allow you to customize Containerd for specific use cases, certain requirements, and unique environments and workflows.

Here are some types of Containerd plugins:

“containers” io.containerd.grpc.v1 plugin

The “containers” plugin exposes Containerd’s container management functionality via the gRPC (Google Remote Procedure Call) API. It allows you to create, start, stop, and delete containers. This plugin also enables you to build custom tools, interfaces, or automation workflows that leverage Containerd’s container management capabilities.

Network Plugins

Network plugins can be used to boost your Containerd’s networking capabilities, allowing you to configure custom network environments for containers. Some examples are:

  • flannel: It offers backing for overlay networks through the Flannel networking solution, which aids in the communication between containers on various hosts.
  • Calico: integrates with the Calico network policy engine, enabling fine-grained network access controls and network segmentation for containers.

Authentication Plugins

Authentication plugins in Containerd are designed to help control access to Containerd’s resources and APIs. They allow you to specify credentials (like a username and password) that Containerd should use when pulling images from a registry.

This in particular can be very useful when you’re working with private registries that require authentication or when you want to increase your rate limit on public registries by authenticating with your account.

Some examples are:

  • JWT (JSON Web Tokens): Allows you to authenticate with Containerd using JWT tokens, which can be issued by identity providers such as OAuth providers.
  • LDAP (Lightweight Directory Access Protocol): It connects with LDAP servers to authenticate users, which allows for unified management of users and control over their access.

“bolt” io.containerd.metadata.v1 plugin

The “bolt” plugin provides a metadata storage backend for Containerd, using the BoltDB embedded key-value store. The “bolt” plugin is crucial for Containerd’s internal operations, and it stores metadata related to containers, images, snapshots, and other Containerd objects, providing fast and efficient access to this information.

Introduction to monitoring Containerd

Monitoring Containerd is key to ensuring the health, efficiency, and security of your containerized environments. By keeping track of vital metrics and events, you can optimize resources, spot problems, and troubleshoot them efficiently. Let’s take a look at how to monitor Containerd, including the important metrics and what makes a monitoring system good:

Key Metrics to Look at when Monitoring Containerd

Resource Utilization Metrics:

  • CPU Usage: Monitor CPU utilization of Containerd processes to ensure efficient resource allocation.
  • Memory Usage: Track memory consumption to identify potential memory leaks or inefficient memory usage.
  • Disk I/O: Monitor Disk read or write operations to assess storage performance and detect bottlenecks.

Image Management Metrics:

  • Image Pulls/Pushes: Monitor image transfer rates to assess image distribution performance and optimize caching strategies.
  • Image Sizes: Track the sizes of container images to manage storage usage efficiently.

Network Traffic:

  • Incoming/Outgoing Traffic: Monitor network bandwidth utilization to detect abnormal traffic patterns and troubleshoot network issues.
  • Network Errors: Track network errors and packet loss to identify potential network issues affecting container communication.

Characteristics of a Good Container Monitoring System:

A good monitoring system needs to provide an overview of your entire application as well as relevant information on each component. Here’s what to consider when selecting a container monitoring solution:

  1. Scalability: The monitoring system should scale effortlessly to handle large-scale container deployments with thousands of containers and nodes.
  2. Customizable Alerts: The system should support customizable alerting mechanisms based on predefined thresholds or anomaly detection algorithms to notify administrators of critical events.
  3. Historical Data Analysis: It should allow for the storage and analysis of historical data, enabling trend analysis, capacity planning, and performance optimization.
  4. Ease of Use: The monitoring system should be easy to deploy, configure, and use, with intuitive interfaces and comprehensive documentation.
  5. Security: It should ensure the security and privacy of monitoring data through encryption, access controls, and compliance with regulatory requirements.

When you focus on these key metrics and characteristics, you can build an effective monitoring strategy for Containerd-based environments, making sure that they provide optimal performance, reliability, and security of containerized workloads.

Extracting Data From Containerd Metrics

Containerd’s metrics are exposed in Prometheus format. The following step will show you how to do it:

Enable Prometheus Metrics in Containerd Configuration

Step 1: Locate the Containerd configuration file

Find the configuration file located at etc/containerd/config.toml

Step 2: Edit the configuration file to enable Prometheus metrics endpoint.

[metrics]
  address = "0.0.0.0:9323"

Step 3: Verify Prometheus Metrics Endpoint

Using a browser such as chrome access the metrics endpoint available at http://localhost:9323/metrics.

Checking Rates of Requests

PromQL provides powerful functions for querying and manipulating time-series data collected by Prometheus. For this metric, you will use the rate() function. The rate() function calculates the per-second average rate of increase of a time series over a specified time window.

Suppose you want to get the rate of requests to the Containerd metrics endpoint over the last 5 minutes. You can use the rate() function as follows:

rate(containerd_http_requests_total[5m])

This query will calculate the rate of HTTP requests to the Containerd metrics endpoint (containerd_http_requests_total) over the last 5 minutes.

Checking Errors

For efficient error retrieval from Containerd, you can employ a range of strategies, including using Containerd’s inherent error reporting systems, monitoring utilities, and log management solutions. Here is a way to go about it:

sum by(grpc_code, instance) (rate(grpc_server_handled_total{job="containerd",grpc_code=~"Unknown|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded", grpc_method="RunPodSandbox"}[$__rate_interval]))

Checking Logs

Containerd is configured to log messages at the “info” level and send them to standard output by default. However, you can modify the level or format of these messages using the [debug] configuration as follows:

Option 1: Accessing the Logs Manually

You can access Containerd logs locally by examining the stdout and stderr streams of individual containers. Here’s an example using the ctr command-line tool:

ctr logs <container_id>
# Replace `<container_id>` with the ID of the container you want to view

Option 2: Integrating with External Logging Solutions

You can integrate Containerd with external logging solutions like Splunk or Fluentd to collect logs. Here’s an example configuration for Fluentd to collect logs from Containerd:

<source>
  @type docker
  tag containerd.*
  ...
</source>
<match containerd.**>
  @type elasticsearch
  ...
</match>

Option 3: Using Containerd’s Logging Endpoints

Another option available to you is to expose containerized logging endpoints that can be queried to retrieve logs programmatically. Here is an example of using Python’s requests library:

import requests
container_id = "<container_id>"
response = requests.get(f"http://localhost:9323/containers/{container_id}/logs")
if response.status_code == 200:
    print(response.text)
else:
    print("Failed to retrieve logs:", response.status_code)

Conclusion

In this article, you have learned that effectively monitoring the essential containerd metrics is essential for ensuring the reliability, performance, and security of your containerized environments. When you track key metrics like resource utilization, container lifecycle events, and network activity, you can understand Containerd’s behavior and mitigate issues before they escalate.

Keep in mind that by adopting a proactive approach to monitoring containers, efficiency, stability, and containerized infrastructures can be maximized.

References

Have Queries? Join https://launchpass.com/collabnix

Abraham Dahunsi Web Developer 🌐 | Technical Writer ✍️| DevOps Enthusiast👨‍💻 | Python🐍 |

“sh: next: command not found” in Next.js Development? Here’s…

Struggling with the "sh: next: command not found" error in your Next.js project? Learn how to fix it by running Next.js locally or globally....
Avinash Bendigeri
1 min read
Join our Discord Server
Index