Ajeet Raina Docker Captain, ARM Innovator & Docker Bangalore Community Leader.

How To Use & Manage Kubernetes DaemonSets? – KubeLabs Glossary

3 min read


Imagine that you want to run a process on all the nodes of the cluster. One of the easiest solution could be running cron job that runs on machine boot or reboot. Also, alternatively one can use the /etc/init.local file to ensure that a specific process or command gets executed as soon as the server gets started. Though it looks to be viable solution, using the node itself to control the daemons that run on it (especially within a Kubernetes cluster) suffers some drawbacks:

  • We need the process to remain running on the node as long as it is part of the cluster. It should be terminated when the node is evicted.
  • The process may need a particular runtime environment that may or may not be available on the node (for example, a specific JDK version, a required kernel library, a specific Linux distro…etc.). So, the process should run inside a container. Kubernetes uses Pods to run containers. This daemon should be aware that it is running within Kubernetes. Hence, it has access to other pods in the cluster and is part of the network.


Enter DaemonSets

DaemonSets are used to ensure that some or all of your K8S nodes run a copy of a pod, which allows you to run a daemon on every node.

When you add a new node to the cluster, a pod gets added to match the nodes. Similarly, when you remove a node from your cluster, the pod is put into the trash. Deleting a DaemonSet cleans up the pods that it previously created.

A Daemonset is another controller that manages pods like Deployments, ReplicaSets, and StatefulSets. It was created for one particular purpose: ensuring that the pods it manages to run on all the cluster nodes. As soon as a node joins the cluster, the DaemonSet ensures that it has the necessary pods running on it. When the node leaves the cluster, those pods are garbage collected.

DaemonSets are used in Kubernetes when you need to run one or more pods on all (or a subset of) the nodes in a cluster. The typical use case for a DaemonSet is logging and monitoring for the hosts. For example, a node needs a service (daemon) that collects health or log data and pushes them to a central system or database (like ELK stack). DaemonSets can be deployed to specific nodes either by the nodes’ user-defined labels or using values provided by Kubernetes like the node hostname.

Why use DaemonSets?

  • Now that we understand DaemonSets, here are some examples of why and how to use it:

  • To run a daemon for cluster storage on each node, such as: – glusterd – ceph
  • To run a daemon for logs collection on each node, such as: – fluentd – logstash
  • To run a daemon for node monitoring on ever note, such as: – Prometheus Node Exporter – collectd – Datadog agent

  • As your use case gets more complex, you can deploy multiple DaemonSets for one kind of daemon, using a variety of flags or memory and CPU requests for various hardware types.

Creating your first DeamonSet Deployment

git clone https://github.com/collabnix/kubelabs
cd kubelabs/DaemonSet101
kubectl apply -f daemonset.yml

The other way to do this:

$ kubectl create -f daemonset.yml --record 

The –record flag will track changes made through each revision.

Getting the basic details about daemonsets:

$ kubectl get daemonsets/prometheus-daemonset

Further Details

kubectl describe daemonset/prometheus-daemonset
[node1 DaemonSet101]$ kubectl describe daemonset/prometheus-daemonset
Name:           prometheus-daemonset
Selector:       name=prometheus-exporter,tier=monitoring
Node-Selector:  <none>
Labels:         name=prometheus-exporter
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 1Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  name=prometheus-exporter
    Image:        prom/node-exporter
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
  Type    Reason            Age    From                  Message
  ----    ------            ----   ----                  -------
  Normal  SuccessfulCreate  3m21s  daemonset-controller  Created pod: prometheus-daemonset-nsjwx

Getting pods in daemonset:

$ kubectl get pods -lname=prometheus-exporter
[node1 DaemonSet101]$ kubectl get pods -lname=prometheus-exporterNAME                         
prometheus-daemonset-nsjwx   1/1     Running   0          4m12s
[node1 DaemonSet101]$

Delete a daemonset:

$ kubectl delete -f daemonset.yml

Restrict DaemonSets To Run On Specific Nodes

By default, a DaemonSet schedules its pods on all the cluster nodes. But sometimes you may need to run specific processes on specific nodes. For example, nodes that host database pods need different monitoring or logging rules. DaemonSets allow you to select which nodes you want to run the pods on. You can do this by using nodeSelector. With nodeSelector, you can select nodes by their labels the same way you do with pods. However, Kubernetes also allows you to select nodes based on some already-defined node properties. For example, kubernetes.io/hostname matches the node name. So, our example cluster has two nodes. We can modify the DaemonSet definition to run only on the first node. Lets’ first get the node names:

$kubectl get nodes
node1   Ready    master   17m   v1.14.9
node2   Ready    <none>   17m   v1.14.9

You need to add the below entry in the above YAML file:

    	  kubernetes.io/hostname: node1

How To Reach a DaemonSet Pod

  • There are several design patterns DaemonSet-pods communication in the cluster:

  • The Push pattern: pods do not receive traffic. Instead, they push data to other services like ElasticSearch, for example.
  • NodeIP and known port pattern: in this design, pods use the hostPort to acquire the node’s IP address. Clients can use the node IP and the known port (for example, port 80 if the DaemonSet has a web server) to connect to the pod.
  • DNS pattern: create a Headless Service that selects the DaemonSet pods. Use Endpoints to discover DaemonSet pods.
  • Service pattern: create a traditional service that selects the DaemonSet pods. Use NodePort to expose the pods using a random port. The drawback of this approach is that there is no way to choose a specific pod.



Join Collabnix Community Slack

Have Queries? Join https://launchpass.com/collabnix

Ajeet Raina Docker Captain, ARM Innovator & Docker Bangalore Community Leader.

26 Replies to “How To Use & Manage Kubernetes DaemonSets? – KubeLabs…”

  1. It is perfect time to make some plans for the long run and it’s time to be happy. I’ve read this submit and if I may I want to suggest you some fascinating issues or suggestions. Maybe you could write next articles relating to this article. I want to learn more things about it!

  2. Just want to say your article is as surprising. The clarity for your post is just excellent and i could think you are an expert on this subject. Fine with your permission allow me to seize your feed to keep updated with approaching post. Thanks a million and please keep up the gratifying work.

  3. A lot of of what you point out is astonishingly precise and that makes me ponder why I hadn’t looked at this with this light previously. This piece really did turn the light on for me personally as far as this specific subject matter goes. However at this time there is actually 1 point I am not really too comfy with so whilst I attempt to reconcile that with the central idea of your point, permit me observe just what the rest of the subscribers have to point out.Nicely done.

  4. Thanks for another informative website. Where else could I get that kind of information written in such a perfect way? I’ve a project that I am just now working on, and I’ve been on the look out for such info.

  5. Hello there, just became alert to your blog through Google, and found that it’s really informative. I am going to watch out for brussels. I will appreciate if you continue this in future. Lots of people will be benefited from your writing. Cheers!

  6. I haven’t checked in here for a while because I thought it was getting boring, but the last few posts are great quality so I guess I’ll add you back to my everyday bloglist. You deserve it my friend 🙂

  7. I’ll immediately grasp your rss as I can’t to find your e-mail subscription hyperlink or e-newsletter service. Do you have any? Please let me recognize in order that I could subscribe. Thanks.

  8. Great amazing things here. I?¦m very glad to see your article. Thank you a lot and i’m taking a look ahead to contact you. Will you please drop me a mail?

  9. I was very pleased to find this web-site.I wanted to thanks for your time for this wonderful read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you blog post.

  10. Aw, this was a very nice post. In thought I want to put in writing like this moreover – taking time and precise effort to make an excellent article… but what can I say… I procrastinate alot and on no account appear to get something done.

Leave a Reply

Your email address will not be published.