Spread the love


Today Docker Inc. released Engine 1.12 Release Candidate 4 with numerous improvements and added security features. With an optional “Swarm Mode” feature rightly integrated into core Docker Engine, a native management of a cluster of Docker Engines, orchestration, decentralized design, service and application deployment, scaling, desired state reconciliation, multi-host networking, service discovery and routing mesh implementation is just a matter of few liner commands.

In the previous posts, we introduced Swarm Mode, implemented a simple service applications and went through 1.12 networking model. Under this post, we will deep dive into Swarm Mode and study what kind of communication gets generated between master and worker nodes in the Swarm cluster.

Setting up Swarm Master Node

Let’s start setting up Swarm Mode cluster and see how underlying communication takes place. I will be using docker-machine to setup master and worker nodes on my Google Cloud Engine.

$docker-machine create -d google –google-project <project-id> –engine-url https://test.docker.com test-master1

If you have less time setting up Swarm Cluster, do refer https://github.com/ajeetraina/google-cloud-swarm. I have forked it from here.

As you see below, Docker Hosts machines gets created through docker-machine with all the nodes running Docker Engine 1.12-rc4.

Let’s initialize the swarm mode on the first master node as shown below:

I have used one liner docker-machine command to keep it clean and simple. The docker-machine command will SSH to the master node and initialize the swarm mode.

The newly released RC4 version holds improvement in terms of security which is enabled by default. In earlier release, one has to pass –secret parameter to secure and control which worker node can join and which can’t. But going forward, the swarm mode automatically generates random secret key. This is just awesome !!!

[Under the hood] – Whenever we do “docker swarm init”, a TLS root CA (Certificate Authority) gets created as shown below.

Then a key-pair is issued for the first node and signed by root CA.

Let’s add the first worker node as shown below:

Looking at inotify output:

When further nodes joins the swarm, they are issues their own keypair, signed by the root CA, and they also receive the root CA public key and certificate. All the communication is encrypted over TLS.

The node keys and certificates are automatically renewed on regular intervals (by default 90 days) but one can tune it with docker swarm update command.

Let us spend some time understanding the master and worker architecture in detail.


Every node in Swarm Mode has a role which can be categorized as  Manager and Worker. Manager node has responsibility to actually orchestrate the cluster, perform the health-check, running containers serving the API and so on. The worker node just execute the tasks which are actually containers. It can-not decide to schedule the containers on the different machine. It can-not change the desired state. The workers only takes work and report back the status. You can enable node promotion or demotion easily through one-liner command.

Managers and Workers uses two different communication models. Managers have built-in RAFT system that allows them to share information for new leader election. At one time, only manager is actually performing the scaling and they use a leader follower model to figure out which one is supposed to be what. No External K-V store is required as built-in internal distributed state store is available.

Workers, on the other side, uses GOSSIP network protocol which is quite fast and consistent. Whenever any new container/tasks gets generated in the cluster, the gossip is going to broadcast it to all the other containers in a specific overlay network that this new container has started. Please remember that ONLY the containers which are running in the specific overlay network will be communicated and NOT globally. Gossip is optimized for heavy traffic.

Let us go one level more deeper to understand how the underlying service is created and dispatched to the worker nodes. Before creating the service, let us first create a new overlay network called mynetwork.

The inotify triggers the relevant output accordingly:

Let’s create our first service:

$sudo docker-machine ssh test-master1 ‘sudo docker service create –name collabnix –replicas 3 \
   –network mynetwork dockercloud/hello-world

Once you run the above command, 3 replicas of services gets generated and distributed across the cluster nodes.

[Under the hood] – Let’s understand what happens whenever a new service is created.


Whenever we create overlay network through “docker network create -d overlay” command, it basically goes to manager. Manager is built up of multiple pipeline stages. One of them is Allocator. Allocator takes the network creation request and choose particular pre-defined sub network that is available. Allocation purely happen in the memory and hence it goes quick. Once network is created, it’s time to connect service to that network. Say, you start with service creation, orchestrator is involved and try to generate the requisite number of tasks which is nothing but containers in real world. But the tasks needs IP address, VXLAN ids as the overlay network needs that too. The allocation happens in the manager nodes. Once allocation gets completed, tasks are created and the state is preserved in the raft store. Once allocation is done, only then the scheduler will be able to move that particular task into the assigned state which is then dispatched to one of the worker node. Manager can also be worker. Every task goes through multiple stages – New, Allocated, Assigned etc. if the task has not been moved to allocator stage, it will not be assigned to worker nodes. With the help of network control plane(gossip protocol), multiple tasks distributed across the multiple worker node is taken care and managed effectively.

I hope you liked reading this deep-dive article. In future blog post, I will try to cover deep dive session into Docker network and volume aspects. Till then, Happy Swarming !!!





Spread the love

Ajeet Raina

My name is Ajeet Singh Raina and I am an author of this blogging site. I am a Docker Captain, ARM Innovator & Docker Bangalore Community Leader. I bagged 2 special awards last year(2019): Firstly, “The Tip of Captain’s Hat Award” at Dockercon 2019, San Francisco, and secondly, “2019 Docker Community Award“. I run Collabnix Community Slack with over 5300+ audience . I have built popular GITHUB repositories like DockerLabs, KubeLabs, Kubetools, RedisPlanet Terraform etc. with the support of Collabnix Community. Currently working as Developer Relations Manager at Redis Labs where I help customers and community members adopt Redis. With over 12,000+ followers over LinkedIn & close to 5100+ twitter followers, I like sharing Docker and Kubernetes related content . You can follow me on Twitter(@ajeetsraina) & GitHub(@ajeetraina)


Jona · 18th July 2016 at 7:54 am

Wow! This blog looks exactly like my old one! It’s
on a entirely different topic but it has pretty much the same
layout and design. Outstanding choice of colors! http://Www.yahoo.net/

Christian · 21st July 2016 at 5:41 pm

Thank you, great blog.

Where can I find log there yo ushowed (where the TLS root CA gets created)


    ajeetraina · 27th July 2016 at 3:41 pm

    I have used inotify tool to get that information.

Sky Moon · 27th July 2016 at 2:50 pm

This website is amazing. I will tell about it to my friends and anybody that could be interested in this subject. Great work guys!

delhi escort · 5th August 2016 at 5:58 pm

Your mode of telling all in this article is genuinely pleasant,
every one be able to easily be aware of it, Thanks a lot.

Kiersten · 6th August 2016 at 11:45 am

I have been browsing on-line more than 3 hours these days, yet I never discovered any attention-grabbing article like yours.
It is lovely price enough for me. In my opinion, if all website owners and
bloggers made excellent content as you did, the internet can be a lot more useful than ever before. http://bing.org

    ajeetraina · 6th August 2016 at 11:51 am

    Thanks for these encouraging words. Will keep posting interesting articles !!!

百家樂遊戲 · 13th September 2016 at 11:49 pm

I am extremely impressed with your writing skills and also with the layout on your weblog. Is this a paid theme or did you customize it yourself? Either way keep up the nice quality writing, it rare to see a great blog like this one nowadays..

Franklyn Stoneburner · 19th September 2016 at 11:49 pm

Wow! This could be one particular of the most helpful blogs We’ve ever arrive across on this subject. Actually Fantastic. I’m also an expert in this topic therefore I can understand your hard work.

Leslie Ina · 24th September 2016 at 7:08 am

blog. A great read.

Mohamed · 25th September 2016 at 4:48 am


John · 21st October 2016 at 2:33 am

I just couldn’t depart your site before suggesting that I actually enjoyed the standard information a person provide for your visitors? Is gonna be back often in order to check up on new posts

Blogs.rediff.Com · 24th October 2016 at 7:38 pm

Excellent post! Continue the good work.

DirkKSleeter · 18th December 2016 at 12:36 pm

Hello to every one, the contents present at this web page are really remarkable for
people experience, well, keep up the nice work fellows.

Itamar Serpa Fernandes · 7th March 2017 at 6:23 pm

“Very informative article post.Really looking forward to read more. Keep writing.”

new era beanies · 1st April 2017 at 3:07 am

“Wow, fantastic blog layout! How long have you been blogging for? you made blogging look easy. The overall look of your website is great, as well as the content!”

new era beanies · 1st April 2017 at 3:37 am

“Hey! I’m at work surfing around your blog from my new iphone! Just wanted to say I love reading your blog and look forward to all your posts! Carry on the excellent work!”

thajsko · 1st April 2017 at 3:42 am

“I see something genuinely interesting about your site so I saved to favorites .”

yealink kenya · 1st April 2017 at 3:48 am

“I would really love to guest post on your blog.:.”~*”

call center solutions kenya · 2nd April 2017 at 4:04 am

“I am so grateful for your blog.Much thanks again. Keep writing.”

thajsko dovolena · 2nd April 2017 at 4:33 am

“Thank you ever so for you blog.Really thank you! Really Great.”

Understanding Node Failure Handling under Docker 1.12 Swarm Mode – Collabnix · 15th July 2016 at 9:30 am

[…] PrevDocker 1.12 Swarm Mode – Under the hood […]

docker笔记(12)——docker 1.12集成docker swarm功能 | 我的站点 · 21st July 2016 at 9:25 am

[…] 参考资料: The relation between “docker/swarm” and “docker/swarmkit”; Comparing Swarm, Swarmkit and Swarm Mode; Docker 1.12 Swarm Mode – Under the hood。 […]

Leave a Reply

Your email address will not be published. Required fields are marked *