Sysctl Support for Docker Swarm Cluster for the first time in Docker 19.03.0 Pre-Release

Estimated Reading Time: 7 minutes

Docker CE 19.03.0 Beta 1 went public 2 week back. It was the first release which arrived with sysctl support for Docker Swarm Mode for the first time. This is definitely a great news for popular communities like Elastic Stack, Redis etc. as they rely on tuning the kernel parameter to get rid of memory exceptions. For example, Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions, hence one need to  increase the limits everytime using sysctl tool. Great to see that Docker Inc. acknowledged the fact that kernel tuning is required sometimes and provides explicit support under Docker 19.03.0 Pre-Release. Great Job !

Wait..Do I really need sysctl?

Say, you have deployed your application on Docker Swarm. it’s pretty simple and it’s working great. Your application is growing day by day and now you just need to scale it. How are you going to do it? The simple answer is: docker service scale app=<number of tasks>.Surely, it is possible today but your containers can quickly hit kernel limits. One of the most popular kernel parameter is net.core.somaxconn. This parameter represents the maximum number of connections that can be queued for acceptance. The default value on Linux is 128, which is rather low.

The Linux kernel is flexible, and you can even modify the way it works on the fly by dynamically changing some of its parameters, thanks to the sysctl command. The sysctl programs allow to limit system-wide resource use. This can help a lot in system administration, e.g. when a user starts too many processes and therefore makes the system unresponsive for other users. Sysctl basically provides an interface that allows you to examine and change several hundred kernel parameters in Linux or BSD. Changes take effect immediately, and there’s even a way to make them persist after a reboot. By using sysctl judiciously, you can optimize your box without having to recompile the kernel and get the results immediately.

Please note that Not all sysctls are namespaced as of Docker 19.03.0 CE Pre-Release. Docker does not support changing sysctls inside of a container that also modify the host system.

Docker does support setting namespaced kernel parameters at runtime & runc honors this. Have a look:

$ docker run --runtime=runc --sysctl net.ipv4.ip_forward=1 -it alpine sh
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
bdf0201b3a05: Pull complete 
Digest: sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913
Status: Downloaded newer image for alpine:latest
/ # sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
/ # 

It is important to note that sysctl support is not new to Docker. The support for sysctl in Docker Compose all started during compose file format v2.1.

For example, to set Kernel parameters in the container, you can use either an array or a dictionary.

  net.core.somaxconn: 1024
  net.ipv4.tcp_syncookies: 0

  - net.core.somaxconn=1024
  - net.ipv4.tcp_syncookies=0

Under this blog post, I will showcase how to use sysctl under 2-Node Docker Swarm Cluster. Let us get started –

Installing a Node with Docker 19.03.0 Beta 1 Test Build on Ubuntu 18.10


Downloading the static binary archive. Go to (or change stable to nightly or test), choose your hardware platform, and download the .tgz file relating to the version of Docker CE you want to install.

--2019-04-10 09:20:01--
Resolving (,,, ...
Connecting to (||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62701372 (60M) [application/x-tar]
Saving to: ‘docker-19.03.0-beta1.tgz’
docker-19.03.0-beta1.tgz  100%[=====================================>]  59.80M  10.7MB/s    in 7.1s    
2019-04-10 09:20:09 (8.38 MB/s) - ‘docker-19.03.0-beta1.tgz’ saved [62701372/62701372]
Extract the archive
You can use the tar utility. The dockerd and docker binaries are extracted.

Extract the tar file

Captain'sBay==>tar xzvf docker-19.03.0-beta1.tgz 

Move the binaries to executable path

Move the binaries to a directory on your executable path It could be such as /usr/bin/. If you skip this step, you must provide the path to the executable when you invoke docker or dockerd commands.

Captain'sBay==>sudo cp -rf docker/* /usr/local/bin/

Start the Docker daemon:

$ sudo dockerd &
Client: Docker Engine - Community
 Version:           19.03.0-beta1
 API version:       1.40
 Go version:        go1.12.1
 Git commit:        62240a9
 Built:             Thu Apr  4 19:15:07 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
  Version:          19.03.0-beta1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.1
  Git commit:       62240a9
  Built:            Thu Apr  4 19:22:34 2019
  OS/Arch:          linux/amd64
  Experimental:     false
  Version:          v1.2.5
  GitCommit:        bb71b10fd8f58240ca47fbb579b9d1028eea7c84
  Version:          1.0.0-rc6+dev
  GitCommit:        2b18fe1d885ee5083ef9f0838fee39b62d653e30
  Version:          0.18.0
  GitCommit:        fec3683

Testing with hello-world

Captain'sBay==>sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete 
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest
INFO[2019-04-10T09:26:23.338596029Z] shim containerd-shim started                  address="/containerd-shim/m
oby/5b23a7045ca683d888c9d1026451af743b7bf4005c6b8dd92b9e95e125e68134/shim.sock" debug=false pid=2953
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
For more examples and ideas, visit:
## Verifying the new `docker context` command

Verifying Docker version

root@DebianBuster:~# docker version
 Version:           19.03.0-beta1
 API version:       1.40
 Go version:        go1.11.5
 Git commit:        62240a9
 Built:             Thu Apr  4 19:18:53 2019
 OS/Arch:           linux/amd64
 Experimental:      false

  Version:          19.03.0-beta1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.11.5
  Git commit:       62240a9
  Built:            Thu Apr  4 19:17:35 2019
  OS/Arch:          linux/amd64
  Experimental:     false
  Version:          1.2.5
  GitCommit:        bb71b10fd8f58240ca47fbb579b9d1028eea7c84
  Version:          1.0.0-rc6+dev
  GitCommit:        2b18fe1d885ee5083ef9f0838fee39b62d653e30
  Version:          0.18.0
  GitCommit:        fec3683

Creating a 2-Node Docker Swarm Mode Cluster

swarm-node-1:~$ sudo docker swarm init --advertise-addr --listen-addr 10.140.0
Swarm initialized: current node (c78wm1g99q1a1g2sxiuawqyps) is now a manager.
To add a worker to this swarm, run the following command:
    docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxfztwgj2r443337mja-cmhuu258lu0327
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Run the below command on worker node:

swarm-node2:~$ sudo docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxf
This node joined a swarm as a worker.

Listing the Swarm Mode CLuster

$ sudo docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
rf3xns913p4tlprmu98z2o8hi     swarm-node2         Ready               Active                            
isbcijzlrft3ahpbzhgipwr9a *   swarm-node-1        Ready               Active              Leader        

Running Multi-service Docker Compose for Redis

Redis is an open source, in-memory data structure store, used as a database, cache and message broker. Redis Commander is an application that allows users to explore a Redis instance through a browser. Let us look at the below Docker compose file for Redis as well as Redis Commander shown below:

version: '3'
    hostname: redis
    image: redis

    hostname: redis-commander
    image: rediscommander/redis-commander:latest
    restart: always
    - REDIS_HOSTS=local:redis:6379
    - "8081:8081"

Ensure that Docker Compose is installed on your system using the below commands:

curl -L`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Run the below command to bring up Redis application running on Docker Swarm Mode:

sudo docker stack deploy -c docker-compose.yml myapp

$ sudo docker stack deploy -c docker-compose.yml myapp
Ignoring unsupported options: restart
Creating network myapp_default
Creating service myapp_redis-commander
Creating service myapp_redis

Verifying if the services are up and running:

~$ sudo docker stack ls
NAME                SERVICES            ORCHESTRATOR
myapp               2                   Swarm

~$ sudo docker service ls
ID                  NAME                    MODE                REPLICAS            IMAGE               
ucakpqi7ozg1        myapp_redis             replicated          1/1                 redis:latest        
fxor8v90a4m0        myapp_redis-commander   replicated          0/1                 rediscommander/redis
-commander:latest   *:8081->8081/tcp

Checking the service logs:

$ docker service logs -f myapp3_redis
myapp3_redis.1.7jpnbigi8kek@manager1    | 1:C 17 Apr 2019 06:26:08.006 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
myapp3_redis.1.7jpnbigi8kek@manager1    | 1:C 17 Apr 2019 06:26:08.006 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
myapp3_redis.1.7jpnbigi8kek@manager1    | 1:C 17 Apr 2019 06:26:08.006 # Warning: no config file specified, using the default config. In order to specify a configfile use redis-server /path/to/redis.conf
myapp3_redis.1.7jpnbigi8kek@manager1    | 1:M 17 Apr 2019 06:26:08.009 * Running mode=standalone, port=6379.
myapp3_redis.1.7jpnbigi8kek@manager1    | 1:M 17 Apr 2019 06:26:08.009 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

As you see above, there is a warning around /proc/sys/net/core/somaxconn lower value set to 128.

Building Docker Compose File using Sysctl parameter

Let us try to build a new Docker compose file with sysctl parameter specified:

Copy the below content and save it as a docker-compose.yml file.

version: '3'
    hostname: redis
    image: redis
    net.core.somaxconn: 1024
    image: rediscommander/redis-commander:latest
    restart: always
    - REDIS_HOSTS=local:redis:6379
    - "8081:8081"

Running Your Redis application

$ sudo docker stack deploy -c docker-compose.yml myapp
Ignoring unsupported options: restart
Creating network myapp_default
Creating service myapp_redis
Creating service myapp_redis-commander

$ sudo docker service ls
ID                  NAME                    MODE                REPLICAS            IMAGE               
2oxhaychob7s        myapp_redis             replicated          1/1                 redis:latest        
pjdwti7hkg1q        myapp_redis-commander   replicated          1/1                 rediscommander/redis
-commander:latest   *:80->8081/tcp

Verifying the Redis service logs

$ sudo docker service logs -f myapp_redis
myapp_redis.1.mp57syo3okka@swarm-node-1    | 1:C 17 Apr 2019 06:59:44.510 # oO0OoO0OoO0Oo Redis is start
ing oO0OoO0OoO0Oo
myapp_redis.1.mp57syo3okka@swarm-node-1    | 1:C 17 Apr 2019 06:59:44.510 # Redis version=5.0.4, bits=64
, commit=00000000, modified=0, pid=1, just started
myapp_redis.1.mp57syo3okka@swarm-node-1    | 1:M 17 Apr 2019 06:59:44.511 * Running mode=standalone, port=6379.

You can see that the warning around /proc/sys/net/core/somaxconn is no longer being displayed which shows that the sysctls parameter has really worked.

In my next blog post, I will talk around rootless Docker and how to get it tested. Stay tuned !

Switching Docker 18.09 Community Edition to Enterprise Engine with no downtime

Estimated Reading Time: 3 minutes 

Under the newer Docker Engine 18.09 release,  a new feature called CE-EE Node Activate has been introduced. It allows a user to perform an in-place seamless activation of the Enterprise engine feature set on an existing Community Edition (CE) node through the Docker command line.CE-EE Node Activate applies a license, and switch the Docker engine to the Enterprise engine binary.


  • Docker Community Edition (CE) version must be 18.09 or higher.
  • All of the Docker packages must be installed: docker-cli, docker-server, and containerd.
  • Node-level Engine activation between CE and EE is only supported in the same version of Docker Enterprise Engine for Docker

Tested Infrastructure

Platform Number of Instance Reading Time
Google Cloud Platform 1 5 min


  • Create an account with Google Cloud Engine (Free Tier)
  • Pick up Ubuntu 18.10 as OS instance

Installing Docker Community Editon 18.09

Verifying Ubuntu 18.10 release

$ cat /etc/os-release
VERSION="18.10 (Cosmic Cuttlefish)"
PRETTY_NAME="Ubuntu 18.10"

Installing Docker 18.09 Release

sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] bionic test"
sudo apt install docker-ce
~$ sudo docker version
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false
Server: Docker Engine - Community
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Running Nginx Docker container

$ sudo docker run -d -p 80:80 nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
a5a6f2f73cd8: Pull complete 
67da5fbcb7a0: Pull complete 
e82455fa5628: Pull complete 
Digest: sha256:31b8e90a349d1fce7621f5a5a08e4fc519b634f7d3feb09d53fac9b12aa4d991
Status: Downloaded newer image for nginx:latest
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
ba4a5822d7c9        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 3 seconds>80/tcp   peaceful_swanson

Verifying Nginx Docker container Up and Running

~$ sudo curl localhost:80
<!DOCTYPE html>
<title>Welcome to nginx!</title>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href=""></a>.<br/>
Commercial support is available at
<a href=""></a>.</p>
<p><em>Thank you for using nginx.</em></p>

Connect your system to DockerHub Account

$sudo docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to to create one.
Username: ajeetraina
WARNING! Your password will be stored unencrypted in /home/joginderkour1950/.docker/config.json.
Configure a credential helper to remove this warning. See
Login Succeeded

Downloading Your Docker Enterprise License

  • Go to site.
  • Login with your Docker ID.
  • Under your profile page, click on “My Content”
  • Click on “Setup” to get your Docker Enterprise License
  • Download your Docker Enterprise License in your system
  • Copy the content of .lic file
  • Create a file called mylicense.lic on your Ubuntu sytem and save it in some location.

Activate the EE license. You must use sudo even if your user is part of the docker group.

$ sudo docker engine activate --license mylicense.lic
License: Quantity: 10 Nodes     Expiration date: 2018-12-10     License is currently active
18.09.0: resolved 
267a9a121ee1: done 
4365cd59d876: done [==================================================>]  1.161kB/1.161kB
7ec4ee35c404: done [==================================================>]   4.55MB/4.55MB
3c60d2c9ddf3: done [==================================================>]  25.71MB/25.71MB
55fa4079a8ab: done [==================================================>]  1.122MB/1.122MB
c5a93cbd4679: done [==================================================>]  333.9kB/333.9kB
e661b0f8ba29: done [==================================================>]   4.82kB/4.82kB
Successfully activated engine.
Restart docker with 'systemctl restart docker' to complete the activation.

Restarting the Docker service

$ sudo systemctl restart docker

Verifying Docker Enterprise Version

$ sudo docker version
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false
Server: Docker Engine - Enterprise
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       33a45cd
  Built:            Wed Nov  7 00:17:07 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Verifying if Nginx container is still running

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
ba4a5822d7c9        nginx               "nginx -g 'daemon of…"   6 minutes ago       Up 6 minutes>80/tcp   peaceful_swanson


5 Minutes to Bootstrap Kubernetes Cluster on GKE using Docker for Mac 18.03.0

Estimated Reading Time: 6 minutesDocker 18.03.0 CE Release is now available under Docker for Mac Platform. Docker for Mac 18.03.0 CE is now shipped with Docker Compose version 1.20.1, Kubernetes v1.9.2,  Docker Machine 0.14.0 & Notary 0.6.0.  Few of the promising features included under this release are listed below-

  • Changing VM Swap size under settings
  • Linux Kernel 4.9.87
  • Support of NFS Volume sharing under Kubernetes.
  • Revert the default disk format to qcow2 for users running macOS 10.13 (High Sierra).
  • DNS name `host.docker.internal` used for host resolution from containers.
  • Improvement over Kubernetes Load balanced services (No longer marked as `Pending`)
  • Fixed hostPath mounts in Kubernetes`.
  • Fix support for AUFS.
  • Fix synchronisation between CLI `docker login` and GUI login.
  • Updated Compose on Kubernetes to v0.3.0. Existing Kubernetes stacks will be removed during migration and need to be re-deployed on the cluster… and many more

In my last blog, I talked about context switching and showcased how one can switch the context from docker-for-desktop to Minikube under Docker for Mac Platform. A context element in a kubeconfig file is used to group access parameters under a convenient name. Each context has three parameters: cluster, namespace, and user. By default, the kubectl command-line tool uses parameters from the current context to communicate with the cluster. Under  .kube/config file, you can see the list of context specified a shown below –

– cluster:
name: gke_spheric-temple-187614_asia-east1-a_k8s-lab1
– cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOOd2..LQo=
server: https://localhost:6443
name: kubernetes
– cluster:
certificate-authority: /Users/ajeetraina/.minikube/ca.crt
name: minikube
– context:
cluster: docker-for-desktop-cluster
user: docker-for-desktop
name: docker-for-desktop

Under this blog, I will showcase how you can bootstrap Kubernetes Cluster on GKE Platform using context switching functionality under Docker for Mac Platform.


  • Install/Upgrade Docker for Mac 18.03 CE Edition
  • Install google-cloud-sdk
  • Enable Google Cloud Engine API
  • Authenticate Your Google Cloud using gcloud auth

Installing Docker for Mac 18.03 CE Edition

Installing Google Cloud SDK on your macOS

  • Make sure that Python 2.7 is installed on your system:

Ajeets-MacBook-Air:~ ajeetraina$ python -V
Python 2.7.10
  • Download the below package based on your system.
Platform Package Size SHA256 Checksum
macOS 64-bit(x86_64) google-cloud-sdk-195.0.0-darwin-x86_64.tar.gz 15.0 MB 56d72895dfc6c4208ca6599292aff629e357ad517e6979203a68a3a8ca5f6cc8
macOS 32-bit(x86) google-cloud-sdk-195.0.0-darwin-x86.tar.gz 15.0 MB e389ec98b65a0dbfc3f2c2637b9e3a375913b39d50e668fecb07cd04474fc080
  • Extract the archive to any location on your file system.
  • Restart your terminal for the changes to take effect.

Initializing the SDK

gcloud init

In your browser, log in to your Google user account when prompted and click Allow to grant permission to access Google Cloud Platform resources.

Enabling Kubernetes Engine API

You need to enable K8s engine API to bootstrap K8s cluster on Google Cloud Platform. To do so, open up this link.

Authenticate Your Google Cloud

Next, you need to authenticate your Google Cloud using glcloud auth

gcloud auth login

Done. We are all set to bootstrap K8s cluster…

Creating GKE Cluster Node

Ajeets-MacBook-Air:~ ajeetraina$ gcloud container clusters create k8s-lab1 --disk-size 10 --zone asia-east1-a --machine-type n1-standard-2 --num-nodes 3 --scopes compute-rw
WARNING: The behavior of --scopes will change in a future gcloud release: service-control and service-management scopes will no longer be added to what is specified in --scopes. To use these scopes, add them explicitly to --scopes. To use the new behavior, set container/new_scopes_behavior property (gcloud config set container/new_scopes_behavior true).
WARNING: Starting in Kubernetes v1.10, new clusters will no longer get compute-rw and storage-ro scopes added to what is specified in --scopes (though the latter will remain included in the default --scopes). To use these scopes, add them explicitly to --scopes. To use the new behavior, set container/new_scopes_behavior property (gcloud config set container/new_scopes_behavior true).
Creating cluster k8s-lab1...done.
Created [].
To inspect the contents of your cluster, go to:
kubeconfig entry generated for k8s-lab1.
k8s-lab1  asia-east1-a  1.7.11-gke.1  n1-standard-2  1.7.11-gke.1  3          RUNNING

Viewing it on Docker for Mac UI

Click on Whale icon on the top right of Docker for Mac and by now, you must be able to see the new Context getting appeared.


Listing the Nodes


Ajeets-MacBook-Air:~ ajeetraina$ kubectl get nodes
NAME                                      STATUS    ROLES     AGE       VERSION
gke-k8s-lab1-default-pool-042d2598-591g   Ready     <none>    7m        v1.7.11-gke.1
gke-k8s-lab1-default-pool-042d2598-c633   Ready     <none>    7m        v1.7.11-gke.1
gke-k8s-lab1-default-pool-042d2598-q603   Ready     <none>    7m        v1.7.11-gke.1

Viewing it directly under GCP Platform



Connecting to Your GKE Cluster

There are 2 ways to do this:

Method-1: Click on “Connection” button to see how to connect to K8s-lab1.



You can connect to your cluster via command-line or using a dashboard.

Ajeets-MacBook-Air:~ ajeetraina$gcloud container clusters get-credentials k8s-lab1 --zone asia-east1-a --project captain-199803
Fetching cluster endpoint and auth data.
kubeconfig entry generated for k8s-lab1. 

Fetching cluster endpoint and auth data. kubeconfig entry generated for k8s-lab1.

Listing the Nodes under Google Cloud Platform


Deploy Nginx on GKE Cluster

Let us see how to deploy Nginx on remote GKE cluster using Docker for Mac. This requires two commands. deploy and expose.

Step 1: Deploy nginx

$ kubectl run nginx --image=nginx --replicas=3

deployment "nginx" created

This will create a replication controller to spin up 3 pods, each pod runs the nginx container.

Step 2: Verify that the pods are running.

You can see the status of deployment by running:

kubectl get pods -owide
NAME                    READY     STATUS    RESTARTS   AGE       IP          NODE
nginx-7c87f569d-glczj   1/1       Running   0          8s   gke-k8s-lab1-default-pool-b2aaa29b-w904
nginx-7c87f569d-pll76   1/1       Running   0          8s   gke-k8s-lab1-default-pool-b2aaa29b-2gzh
nginx-7c87f569d-sf8z9   1/1       Running   0          8s   gke-k8s-lab1-default-pool-b2aaa29b-qpc7

Youcan see that each nginx pod is now running in a different node (virtual machine).

Once all pods have the Running status, you can then expose the nginx cluster as an external service.

Step 3: Expose the nginx cluster as an external service.

$ kubectl expose deployment nginx --port=80 --target-port=80 \

service "nginx" exposed

This command will create a network load balancer to load balance traffic to the three nginx instances.

Step 4: Find the network load balancer address:

kubectl get service nginx
NAME      TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
nginx     LoadBalancer   <pending>     80:30253/TCP   12s

It may take several minutes to see the value of EXTERNAL_IP. If you don’t see it the first time with the above command, retry every minute or so until the value of EXTERNAL_IP is displayed.

You can then visit http://EXTERNAL_IP/ to see the server being served through network load balancing.

GKE provides amazing platform to view workloads & Load-balancer as shown below:

GKE also provides UI for displaying Loadbalancer:

In my upcoming blog post, I will showcase how context switching can help you in switching your project between Dev, QA & Production environment flawlessly.

Did you find this blog helpful? Feel free to share your experience. Get in touch with me on Twitter –  @ajeetsraina

If you are looking out for contribution, join me at Docker Community Slack Channel.

LinuxKit 101: Getting Started with LinuxKit for Google Cloud Platform

Estimated Reading Time: 7 minutes 

“…LinuxKit? A New Beast?

     What problem does it solve for us?..”



In case you missed out Dockercon 2017 and have no idea what is LinuxKit all about, then you have arrived at the right place. For the next 30 minutes of your time, I will be talking about an open source container toolkit which Docker Inc. has recently made to the public & will help you get started with it in very easy and precise way.

What is LinuxKit?

LinuxKit is just like Docker’s other open-source container toolkits such as InfraKit and VPNkit. It is essentially a container-native toolkit that allows organizations to build their own containerized operating systems that are secure, lean, modular and portable. Essentially, it is more of a developer kit than an end-user product.This project is completely open source and  is hosted on GitHub, under an Apache 2 licence.

What problem does it solve?

Last year Docker Inc. started shipping Docker for Mac, Docker for Windows, Docker for Azure & Docker for GCP and that brought a Docker-native experience to these various platforms. One of the common problem which the community faced was non-standard Linux OS running on all those platform.  Esp. Cloud platform do not ship with a standard Linux which brought lots of concerns around portability, security and  incompatibility. This lead Docker Inc. to bundle Linux into the Docker platform to run on all of these places uniformly.

Talking about portability, Docker Inc. has always focused on product which should run anywhere. Hence, they worked with partners like HP, Intel, ARM and Microsoft to ensure that LinuxKit toolkit should flawlessly run on the desktop, server, cloud ARM, x86, virtual environment and on bare metal. LinuxKit was built with an intention of  an optimized tooling for portability which can accommodate a new architecture, a new system in very easier way.

What does LinuxKit hold?

LinuxKit includes the tooling to allow building custom Linux subsystems that only include exactly the components the runtime platform requires. All system services are containers that can be replaced, and everything that is not required can be removed.The toolkit works with Docker’s containerd. All components can be substituted with ones that match specific needs.You can optimize LinuxKit images for specific hardware platforms and host operating systems with just the drivers and other dependencies you need, and nothing more, rather than use a full-fat generic base. The toolkit basically tries to help you create your own slimline containerized operating system as painlessly as possible. The size of a LinuxKit image is in MBs ( around 35-50MB).


The above shown is YAML file which specifies a kernel and base init system, a set of containers that are built into the generated image and started at boot time. It also specifies what formats to output(shown at the last line), such as bootable ISOs and images for various platforms. Interestingly, system services are sandboxed in containers, with only the privileges they need. The configuration is designed for the container use case. The whole system is built to be used as immutable infrastructure, so it can be built and tested in your CI pipeline, deployed, and new versions are redeployed when you wish to upgrade. To know more about YAML specification, check this out.

What tool does LinuxKit uses?

There are two basic tools which LinuxKit uses – Linuxkit & Moby.


In short, the moby tool converts the yaml specification into one or more bootable images.

Let us get started with LinuxKit to understand how it builds customized ISO images and run uniformly across various platform. Under this blog post, I have chosen Google Cloud Platform. We will build LinuxKit based customized ISO image locally on my Macbook Air and push it to Google Cloud Platform to run as VM instance. I will be using forked linuxkit repository which I have built around and runs Docker container(ex. running Portainer docker container) inside VM instance too.


  1. Install LinuxKit & Moby tool on macOS
  2. Building a LinuxKit ISO Image with Moby 
  3. Create a bucket under Google Cloud Platform
  4. Upload the LinuxKit ISO image to a GCP bucket using LinuxKit tool
  5. Initiate the GCP instance from the LinuxKit ISO image placed under GCP bucket
  6. Verifying Docker running inside LinuxKitOS 
  7. Running Portainer as Docker container



– Install Google Cloud SDK on your macOS system through this link. You will need to verify your google account using the below command:

$gcloud auth login

– Ensure that the build essential tools like make are perfectly working

– Ensure that GO packages are installed on macOS..


  1. Clone the repository:


sudo git clone


2.  Change directory to linuxkit and run make which builds “moby” and “linuxkit” for us

cd linuxkit && sudo make


3.  Verify that these tools are built and placed under /bin:

cd bin/
moby         linuxkit

4.  Copy these tools into system PATH:

sudo cp bin/* /usr/local/bin/

5. Use moby tool to build the customized ISO image:


cd examples/
sudo moby build gcpwithdocker.yml




[Update: 6/21/2017 – With the latest release of LinuxKit, Output section is no longer allowed inside YAML file. It means that whenever you use moby build command to build an image, specify -output gcp to build an image in a format that GCP will understand. For example:

moby build -output gcp example/gcpwithdocker.yml

This will create a local gcpwithdocker.img.tar.gz compressed image file.]


6.  Create a GCE bucket “mygcp” under your Google Cloud Platform:


7. Run  linuxkit push command  to push it to GCP:


sudo linuxkit push gcp -project synthetic-diode-161714 -bucket mygcp gcpwithdocker.img.tar.gz



[Note: “synthetic-diode-161714” is my GCP project name and “mygcp” is the bucket name which I created in earlier step. Please input as per your environment.]

Please note that you might need to enable Google Cloud API using this link in case you encounter “unable to connect GCP”  error. 

8.  You can execute the image you created and this will should show up under VM instance on Google  Cloud Platform:



This will build up a LinuxKit OS which you can verify below:


You can also verify if this brings up VM instance on GCP platform:


9. You can use runc command to list out all the services which were defined under gcpwithdocker.yml file:


10. As shown above, one of the service which I am interested is called “docker”. You can use the below command to enter into docker service:


runc exec -t docker sh

Wow ! It is running the latest Docker 17.04.0-ce version.

11.  Let us try to run Portainer application and check if it works good.


You can verify the IP address running ifconfig for that specific container which in my case is


Now this is what I call ” a coolest stuff on earth”. Linuxkit allows you to build your own secure, modular, portable, lean and mean containerized OS and that too in just minutes. I am currently exploring LinuxKit in terms of bare metal OS and will share it under my next blog post.

Did you find this blog helpful? Are you planning to explore LinuxKit? Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution, join me at Docker Community Slack Channel.

Introducing new RexRay 0.8 with Docker 17.03 Managed Plugin System for Persistent Storage on Cloud Platforms

Estimated Reading Time: 5 minutesDellEMC Rex-Ray 0.8 Final Release was announced last week. Graduated as top-level project within {code} community, RexRay 0.8 release has been considered as one of the largest releases till date. The new release introduced support for long lists of new storage platforms like S3FS, EBS, EFS, GCEPD & ScaleIO shown below:


Public cloud storage is one of the fastest growing sector in storage with leaders like Amazon AWS, Google Cloud Storage and Microsoft Azure. With the release of RexRay 0.8, {code} community took the right approach in targeting the first community-contributed driver starting with Amazon EFS driver and then quickly adding additional community-contributed drivers like Digital Ocean, FittedCloud, Google Cloud Engine (GCEPD) & Microsoft Azure Unmanaged Disk driver.

Introducing New Docker 17.03 Volume Plugin System

With Docker 17.03 release, a new managed plugin system has been introduced. This is quite different from the old Docker plugin system. Plugins are now distributed as Docker images and can be hosted on Docker Hub or on a private registry.A volume plugin enables Docker volumes to persist across multiple Docker hosts.


In case you are very new to Docker Plugins, they basically extend Docker’s functionality.  A plugin is a process running on the same or a different host as the docker daemon, which registers itself by placing a file on the same docker host in one of the plugin directories described either as a.sock files( UNIX domain sockets placed under /run/docker/plugins), .spec files( text files containing a URL, such as unix:///other.sock or tcp://localhost:8080 placed under /etc/docker/plugins or /usr/lib/docker/plugins)or.json files ( text files containing a full json specification for the plugin placed under /etc/docker/plugin). You can refer this in case you want to develop your own Docker Volume Plugin.

Running RexRay inside Docker container

Yes, you read it correct ! With the introduction of Docker 17.03 New Plugin Managed System, you can now run RexRay inside Docker container flawlessly. Rex-Ray Volume Plugin is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC.

You can list out available Docker Volume Plugin for various storage platforms using docker search rexray as shown below:


Let us test-drive RexRay Volume Plugin for Swarm Mode cluster for the first time. I have 4-node Swarm Mode cluster running on Google Cloud Platform as shown below:


Verify that all the cluster nodes are running the latest 17.03.0-ce (Community Edition).

Installing the RexRay Volume plugin is just one-liner command:



You can inspect the Rex-Ray Volume plugin using docker plugin inspect command:

rexray_inspect    rexray_inspect22

It’s time to create a volume using docker volume create utility :

$sudo docker volume create –driver rexray/gcepd –name storage1 –opt=size=32


You can verify if it is visible under GCE Console window:


Let us try running few application which uses RexRay volume plugin as shown:

{code}==>docker run -dit –name mydb -e MYSQL_ROOT_PASSWORD=wordpress -e MYSQL_USER=wordpress -e MYSQL_PASSWORD:wordpress –volume-driver=rexray/gcepd -v dbdata:/var/lib/mysql mysql:5.7

Verify that MySQL service is up and running using docker logs <container-id> command as shown below:


By now, we should be able to see new volume called “dbdata” created and shown under docker volume ls command:


Under GCE console, it should get displayed too:


Using Rex-Ray Volume Plugin under Docker 17.03 Swarm Mode

This is the most interesting section of this blog post. RexRay volume plugin worked great for us till now, especially for a single Docker host running multiple number of services.But what if I want to enable RexRay Volume to persist across multiple Docker Hosts(Swarm Mode cluster)? Yes, there is one possible way to achieve this – using Swarm Executor. It executes docker command across the swarm cluster. Credits to Madhu Venugopal @ Docker Team for assisting me testing with this tool.


Please remember that this is UNOFFICIAL way of achieving Volume Plugin implementation across swarm cluster.  I found this tool really cool and hope that it gets integrated within Docker official repository.

First, we need to clone this repository:

$git clone

Run the below command to push the plugin across the swarm cluster:

$cd swarm-exec

$./ docker plugin install –grant-all-permissions rexray/gcepd GCEPD_TAG=rexray


First, let’s quickly  verify the plugin on the master node  as shown below:


While verifying it on the worker nodes:



The docker volume inspect <volname> should display this particular volume as rexray volume driver as shown below:


Creating a MySQL service which uses Rex-Ray volume under Swarm Mode cluster:


Verifying that the service is up and running:


To conclude, the new version of RexRay looks promising and brings support for various Cloud storage platform. It continues to be leading open source container orchestration engine and now with inclusion of Docker 17.03 Managed Plugin architecture, it will definitely reduce the pain for implementing persistent storage solution.

What’s new in Docker 1.12.0 Load-Balancing feature?

Estimated Reading Time: 8 minutesIn the previous blog post, we deep-dived into Service Discovery aspects of Docker. A service is now a first class citizen in Docker 1.12.0 which allows replication, update of images and dynamic load-balancing. With Docker 1.12, services can be exposed on ports on all Swarm nodes and load balanced internally by Docker using either a virtual IP(VIP) based or DNS round robin(RR) based Load-Balancing method or both.


In case you are very new to Load-balancing concept, the load balancer assigns workload to a set of networked computer servers or components in such a manner that the computing resources are used in an optimal manner. A load balancer provides high availability by detecting server or component failure and re-configuring the system appropriately. Under this post, I will try to answer the following queries:


Let’s get started –

Is Load-Balancing new to Docker?

Load-balancing(LB) feature is not at all new for Docker. It was firstly introduced under Docker 1.10 release where Docker Engine implements an embedded DNS server for containers in user-defined networks.In particular, containers that are run with a network alias ( — net-alias) were resolved by this embedded DNS with the IP address of the container when the alias is used.

No doubt, DNS Round robin is extremely simple to implement and is an excellent mechanism to increment capacity in certain scenarios, provided that you take into account the default address selection bias but it possess certain limitations and issues like some applications cache the DNS host name to IP address mapping and  this causes applications to timeout when the mapping gets changed.Also, having non-zero DNS TTL value causes delay in DNS entries reflecting the latest detail. DNS based load balancing does not do proper load balancing based on the client implementation. To learn more about DNS RR which is sometimes called as poor man’s protocol, you can refer here.

What’s new in Load-balancing feature under Docker 1.12.0?

  • Docker 1.12.0 comes with built-in Load Balancing feature now.LB is designed as an integral part of Container Network Model (rightly called as CNM) and works on top of CNM constructs like network, endpoints and sandbox. Docker 1.12 comes with VIP-based Load-balancing.VIP based services use Linux IPVS load balancing to route to the backend containers
  • No more centralized Load-Balancer, it’s distributed and hence scalable. LB is plumbed into individual container. Whenever container wants to talk to another service, LB is actually embedded into the container where it happens. LB is more powerful  now and just works out of the box.


  • New Docker 1.12.0 swarm mode uses IPVS(kernel module called “ip_vs”) for load balancing. It’s a load balancing module integrated into the Linux kernel
  • Docker 1.12 introduces Routing Mesh for the first time.With IPVS routing packets inside the kernel, swarm’s routing mesh delivers high performance container-aware load-balancing.Docker Swarm Mode includes a Routing Mesh that enables multi-host networking. It allows containers on two different hosts to communicate as if they are on the same host. It does this by creating a Virtual Extensible LAN (VXLAN), designed for cloud-based networking. we will talk more on Routing Mesh at the end of this post.

Whenever you create a new service in Swarm cluster, the service gets Virtual IP(VIP) address. Whenever you try to make a request to the particular VIP, the swarm Load-balancer will distribute that request to one of the container of that specified service. Actually the built-in service discovery resolves service name to Virtual-IP. Lastly, the service VIP to container IP load-balancing is achieved using IPVS. It is important to note here that VIP is only useful within the cluster. It has no meaning outside the cluster because it is a private non-routable IP.

I have 6 node cluster running Docker 1.12.0 in Google Cloud Engine. Let’s examine the  VIP address through the below steps:

  1. Create a new overlay network:

    $docker network create –driver overlay \

     –subnet \

     –opt encrypted \




2. Let’s create a new service called collabweb which is a simple Nginx server as shown:

        $ docker service create \

       —replicas 3 \

       —name collabweb \

       —network collabnet \


3. As shown below, there are 3 nodes where 3 replicas of containers are running the service under the swarm overlay network called “collabnet”.


4. Use docker inspect command to look into the service internally as shown below:



It shows “VIP” address added to each service. There is a single command which can help us in getting the Virtual IP address as shown in the diagram below:


5. You can use nsenter utility to enter into its sandbox to check the iptables configuration:


In any iptables, usually a packets enters the Mangle Table chains first and then the NAT Table chains.Mangling refers to modifying the IP Packet whereas NAT refers to only address translation. As shown above in the mangle table, service IP gets marking of 0x10c using iptables OUTPUT chain. IPVS uses this marking and load balances it to containers, and as shown:


As shown above, you can use  ipvsadm  to set up, maintain or inspect the IP virtual server table in the Linux kernel.This tool can be installed on any of Linux machine through apt or yum based on the Linux distribution.

A typical DNS RR and IPVS LB can be differentiated as shown in the below diagram where DNS RR shows subsequent list of IP addresses when we try to access the service each time(either through curl or dig) while VIP  load balances it to containers(i.e., and



6. Let’s create  a new service called collab-box under the same network. As shown in the diagram, a new Virtual-IP ( will be automatically attached to this service as shown below:


Also, the service discovery works as expected,



IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching. It’s a load balancing module integrated into the linux kernel. It is based on Netfilter.It supports TCP, SCTP & UDP, v4 and v7. IPVS running on a host acts as a load balancer before a cluster of real servers, it can direct requests for TCP/UDP based services to the real servers, and makes services of the real servers to appear as a virtual service on a single IP address.

It is important to note that IPVS is not a proxy — it’s a forwarder that runs on Layer 4. IPVS forwards traffic from clients to back-ends, meaning you can load balance anything, even DNS! Modes it can use include:

  • UDP support
  • Dynamically configurable
  • 8+ balancing methods
  • Health checking

IPVS holds lots of interesting features and has been in kernel for more than 15 years. Below chart differentiate IPVS from other LB tools:


Is Routing Mesh a Load-balancer?

Routing Mesh is not Load-Balancer. It makes use of LB concepts.It provides global publish port for a given service. The routing mesh uses port based service discovery and load balancing. So to reach any service from outside the cluster you need to expose ports and reach them via the Published Port.

In simple words, if you had 3 swarm nodes, A, B and C, and a service which is running on nodes A and C and assigned node port 30000, this would be accessible via any of the 3 swarm nodes on port 30000 regardless of whether the service is running on that machine and automatically load balanced between the 2 running containers. I will talk about Routing Mesh in separate blog if time permits.

It is important to note that Docker 1.12 Engine creates “ingress” overlay network to achieve the routing mesh. Usually the frontend web service and sandbox are part of “ingress” network and take care in routing mesh.All nodes become part of “ingress” overlay network by default using the sandbox network namespace created inside each node. You can refer this link to learn more about the internals of Routing Mesh.

Is it possible to integrate an external LB to the services in the cluster.Can I use HA-proxy in Docker Swarm Mode?

You can expose the ports for services to an external load balancer. Internally, the swarm lets you specify how to distribute service containers between nodes.If you would like to use an L7 LB you either need to point them to any (or all or some) node IPs and PublishedPort. This is only if your L7 LB cannot be made part of the cluster. If the L7 LB can be made of the cluster by running the L7 LB itself as a service then they can just point to the service name itself (which will resolve to a VIP). A typical architecture would look like this:


In my next blog, I am going to elaborate on External Load balancer primarily. Keep Reading !

Docker 1.12 Swarm Mode – Under the hood

Estimated Reading Time: 6 minutesToday Docker Inc. released Engine 1.12 Release Candidate 4 with numerous improvements and added security features. With an optional “Swarm Mode” feature rightly integrated into core Docker Engine, a native management of a cluster of Docker Engines, orchestration, decentralized design, service and application deployment, scaling, desired state reconciliation, multi-host networking, service discovery and routing mesh implementation is just a matter of few liner commands.

In the previous posts, we introduced Swarm Mode, implemented a simple service applications and went through 1.12 networking model. Under this post, we will deep dive into Swarm Mode and study what kind of communication gets generated between master and worker nodes in the Swarm cluster.

Setting up Swarm Master Node

Let’s start setting up Swarm Mode cluster and see how underlying communication takes place. I will be using docker-machine to setup master and worker nodes on my Google Cloud Engine.

$docker-machine create -d google –google-project <project-id> –engine-url test-master1

If you have less time setting up Swarm Cluster, do refer I have forked it from here.

As you see below, Docker Hosts machines gets created through docker-machine with all the nodes running Docker Engine 1.12-rc4.

Let’s initialize the swarm mode on the first master node as shown below:

I have used one liner docker-machine command to keep it clean and simple. The docker-machine command will SSH to the master node and initialize the swarm mode.

The newly released RC4 version holds improvement in terms of security which is enabled by default. In earlier release, one has to pass –secret parameter to secure and control which worker node can join and which can’t. But going forward, the swarm mode automatically generates random secret key. This is just awesome !!!

[Under the hood] – Whenever we do “docker swarm init”, a TLS root CA (Certificate Authority) gets created as shown below.

Then a key-pair is issued for the first node and signed by root CA.

Let’s add the first worker node as shown below:

Looking at inotify output:

When further nodes joins the swarm, they are issues their own keypair, signed by the root CA, and they also receive the root CA public key and certificate. All the communication is encrypted over TLS.

The node keys and certificates are automatically renewed on regular intervals (by default 90 days) but one can tune it with docker swarm update command.

Let us spend some time understanding the master and worker architecture in detail.


Every node in Swarm Mode has a role which can be categorized as  Manager and Worker. Manager node has responsibility to actually orchestrate the cluster, perform the health-check, running containers serving the API and so on. The worker node just execute the tasks which are actually containers. It can-not decide to schedule the containers on the different machine. It can-not change the desired state. The workers only takes work and report back the status. You can enable node promotion or demotion easily through one-liner command.

Managers and Workers uses two different communication models. Managers have built-in RAFT system that allows them to share information for new leader election. At one time, only manager is actually performing the scaling and they use a leader follower model to figure out which one is supposed to be what. No External K-V store is required as built-in internal distributed state store is available.

Workers, on the other side, uses GOSSIP network protocol which is quite fast and consistent. Whenever any new container/tasks gets generated in the cluster, the gossip is going to broadcast it to all the other containers in a specific overlay network that this new container has started. Please remember that ONLY the containers which are running in the specific overlay network will be communicated and NOT globally. Gossip is optimized for heavy traffic.

Let us go one level more deeper to understand how the underlying service is created and dispatched to the worker nodes. Before creating the service, let us first create a new overlay network called mynetwork.

The inotify triggers the relevant output accordingly:

Let’s create our first service:

$sudo docker-machine ssh test-master1 ‘sudo docker service create –name collabnix –replicas 3 \
   –network mynetwork dockercloud/hello-world

Once you run the above command, 3 replicas of services gets generated and distributed across the cluster nodes.

[Under the hood] – Let’s understand what happens whenever a new service is created.


Whenever we create overlay network through “docker network create -d overlay” command, it basically goes to manager. Manager is built up of multiple pipeline stages. One of them is Allocator. Allocator takes the network creation request and choose particular pre-defined sub network that is available. Allocation purely happen in the memory and hence it goes quick. Once network is created, it’s time to connect service to that network. Say, you start with service creation, orchestrator is involved and try to generate the requisite number of tasks which is nothing but containers in real world. But the tasks needs IP address, VXLAN ids as the overlay network needs that too. The allocation happens in the manager nodes. Once allocation gets completed, tasks are created and the state is preserved in the raft store. Once allocation is done, only then the scheduler will be able to move that particular task into the assigned state which is then dispatched to one of the worker node. Manager can also be worker. Every task goes through multiple stages – New, Allocated, Assigned etc. if the task has not been moved to allocator stage, it will not be assigned to worker nodes. With the help of network control plane(gossip protocol), multiple tasks distributed across the multiple worker node is taken care and managed effectively.

I hope you liked reading this deep-dive article. In future blog post, I will try to cover deep dive session into Docker network and volume aspects. Till then, Happy Swarming !!!




Docker 1.12 Networking Model Overview

Estimated Reading Time: 8 minutes“The Best way to orchestrate Docker is Docker”

In our previous post, we talked about Swarm Mode’s built-in orchestration and distribution engine.Docker’s new deployment API objects like Service and Node , built-in multi-host and multi-container cluster management integrated with Docker Engine, decentralized design,declarative service model, scaling and resiliency services , desired state conciliation, service discovery,  load balancing,  out-of-box security by default and rolling updates etc. just makes Docker 1.12  all-in-all automated deployment and management tool for Dockerized distributed applications and microservices at scale in production.



Under this post, we are going to deep-dive into Docker 1.12 Networking model. I have 5 node swarm cluster test environment as shown below:


If you SSH to test-master1 and check the default network layout:


Every container has an IP address on three overlay networks:

  1. Ingress
  2. docker_gwbridge
  3. user-defined overlay

Ingress Networking:

The swarm manager uses ingress load balancing to expose the services you want to make available externally to the swarm. The swarm manager can automatically assign the service a PublishedPort or you can configure a PublishedPort for the service in the 30000-32767 range.  What it actually means is Network ingress into the cluster is based on a node port model in which each service is randomly assigned a cluster-wide reserved port between 30000 and 32000 (default range). This means that every node in the cluster listens on this port and routes traffic for that service to it. This is true irrespective of whether a particular worker node is actually running the specified service.

It is important to note that only those services that has a port published (using the -p option) require the ingress network. But for those backend services which doesn’t publish ports, the corresponding containers are NOT attached to the ingress network.

External components, such as cloud load balancers, can access the service on the PublishedPort of any node in the cluster whether or not the node is currently running the task for the service. All nodes in the swarm cluster route ingress connections to a running task instance. Hence, ingress  follow a node port model in which each service has the same port on every node in the cluster.


The `default_gwbridge` network is added only for non-internal networks. Internal networks can be created with “–internal” option.. Containers connected to the multi-host network are automatically connected to the docker_gwbridge network. This network allows the containers to have external connectivity outside of their cluster, and is created on each worker node.

Docker Engine provides you flexibility to create this default_gwbridge by hand instead of letting the daemon create it automatically. In case you want docker to create a docker_gwbridge network in desired subnet, you can tweak it as shown below:

$docker network create –subnet={Your prefered subnet } -o -o docker_gwbridge.

User-defined Overlay:

This is the overlay network that the user has specified that the container should be on. In our upcoming example , we will call it mynet. A container can be on multiple user-defined overlays.


Enough with the theoretical aspects !! Let’s try hands on the networking model practically.

As shown below, I have 3 nodes swarm cluster with 1 master and 2 worker nodes.


I created a user-defined overlay through the below command:

$ sudo docker network create -d overlay mynet

I can see that the new overlay network gets listed under “swarm” scope(as we are already into swarm mode)  as shown below:


I have a service “frontier” running tasks on node1, node2 and master1 as shown below:



We can check the container running under node-1 and node-2 respectively:



Meanwhile, I added a new Node-3 and then scaled it to 10.


Now I can see that the containers are scaled across the swarm cluster.

To look into how overlay networking works, let us target the 4th Node and add it to the Swarm Cluster.


Now, the node list gets updated as shown below:


Whenever you add any node to the swarm cluster, it doesn’t automatically reflect the mynet overlay network as shown below:


The overlay network only gets reflected whenever the new task is assigned and this happens on-demand.

Let us try to scale our old service and see if node-4 network layout gets reflected with mynet network.

Earlier, we had 10 replicas running which is scaled across master1, node1, node2 and node4. Once we scaled it to 20, the swarm cluster engine will scale it across all the nodes as shown below:


Let us now check the network layout at node-4:

Hence, the overlay network gets created on-demand whenever the new task is assigned to this node.


Swarm nodes are “self-organizing and self-healing. What it means? Whenever any node or container goes crash or sudden unplanned shutdown, the scaling swarm engine attempts to correct to make things right again. Let us look into this aspect in detail:

As we see above, here is an example of nodes and running tasks:

Master-1 running 4 tasks

Node-1 running 4 tasks

Node-2 running 4 tasks

Node-3 running 4 tasks

Node-4 running  4 instances

Now let’s bring down node-4.


As soon as all the containers running on node-4 is stopped, it tries to start another 4 containers with different IDs on the same node.


So this shows self-healing aspect of Docker Swarm Engine.


Let’s try bringing down node-4 completely. As soon as you bring down the node-4, the containers running on node-4 gets started on other node automatically.



Master-1 running 5 tasks

Node-1 running 5 tasks

Node-2 running 5 tasks

Node-3 running 5 tasks

Hence, this is how it organizes 20 tasks which is now scaled across master1, node1, node2 and node3.

Global Services:

This option enables service tasks to sit on all the nodes. You can create a service with –mode-global option to enable this functionality as shown below:



There are scenerios when you sometime want to segregate workloads on your cluster where you specifically want workloads to go to only a certain set of nodes .One example I have pulled from DockerCon slide which shows SSD based constraints which can be applied as shown below:



Routing Mesh 

We have reached the last topic of this blog post and that’s not complete without talking about Routing mesh. I have pulled out the presentation demonstrated at DockerCon 2016 which clearly shows us how routing mesh works.

                                                                                                                                                                                                        ~ Source: DockerCon 2016

To understand how Routing mesh works, suppose that there is one manager node and 3 worker nodes which is serving myapp:80. Whenever an operator try to access myapp:80 on the exposed port, he probably because of an external load-balancer might happen to hit worker-2 and it sounds good because worker-2 is having 2 copies of the frontend containers and it is ready to serve it without any issue. Imagine a scenario when user access myapp:80 and gets redirected to worker-3 which currently has no copies of the containers. This is where Routing Mesh technology comes into the picture. Even though the worker-3 has no copies of this container, the docker swarm engine is going to re-route the traffic to worker-2 which has necessary copies to serve it.Your external Load-balancer don’t need to understand where is the container running. Routing mesh takes care of that automatically. In short, Container-aware routing mesh is capable of transparent rerouting the traffic from node-3 to a node that is running container(which is node-2) shown above. Internally, Docker Engine  allocates a cluster wide port and it maps that port to the containers of a service and the routing mesh will take care of routing traffic to the containers of that service by exposing a port on every node in the swarm.

In our next post, we will talk about routing mesh in detail and cover volume aspect of Swarm Mode. Till then, Happy Swarming !!!

Docker Engine 1.12 comes with built-in Distribution & Orchestration System

Estimated Reading Time: 7 minutesDocker Engine 1.12 can be rightly called ” A Next Generation Docker Clustering & Distributed System”. Though Docker Engine 1.12 Final Release is around corner but the recent RC2 brings lots of improvements and exciting features. One of the major highlight of this release is Docker Swarm Mode which provides powerful yet optional ability to create coordinated groups of decentralized Docker Engines. Swarm Mode combines your engine in swarms of any scale. It’s self-organizing and self-healing. It enables infrastructure-agnostic topology.The newer version democratizes orchestration with out-of-box capabilities for multi-container on multi-host app deployments as shown below:

Built on Engine as a uniform building block for self organizing and healing group of Engines, Docker ensures that orchestration is accessible for every developer and operation user. The new Swarm Mode adopts  the de-centralized architecture rather than centralized one (key-value store) as seen in the earlier  Swarm releases. Swarm Mode uses the Raft consensus algorithm  to perform leader selection, and maintain the cluster’s states.

In Swarm Mode, all Docker Engine will unite into a cluster with management tier. It is basically master – slave system but all Docker Engine will be united and they will maintain a cluster state. Instead of running a single container, you will declare a desired state for your application which means multiple container and then engine themselves will maintain that state. Additionally, a new “docker service” feature has been added under the new release. The “docker service create” is expected to be an evolution of “docker run”. Docker run is imperative command and all it helps you  to get container up and running.The new “docker service create” command declare that you have to setup a server which can run one or more containers and those container will run , provided the state you declare for the service will be maintained in Engine / inside the distributed store based on raft consensus protocol.That brings the notion of desired state reconciliation. Whenever any node in the cluster goes down, the swarm itself will recognize that there has been deviation between the desired state and it will bring up  new instance to reconstruct the reconciliation. I highly recommend visualizing to understand what does it mean.

Docker Swarm Node is used  for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and much more. Let’s see what features does Docker Swarm Mode adds to Docker Cluster functionality:Pic-6

Looking at the above features, Docker Swarm mode brings the following benefits :

  • Distributed: Swarm Mode uses the Raft Consensus Algorithm in order to coordinate and does not rely on a single point of failure to perform decisions.
  • Secure: Node communication and membership within a Swarm are secure out of the box. Swarm Mode uses mutual TLS for node authentication, role authorization and transport encryption, automating both certificate issuance and rotation.
  • Simple: Swarm Mode is operationally simple and minimizes infrastructure dependencies. It does not need an external database to operate. It uses internal distributed State store.

Below picture depicts Swarm Mode cluster architecture. Fundamentally its a master and slave architecture. Every node in a swarm is Docker Host running Docker Engine. Some of the node has privilege role called Manager.The manager node participate in “raft consensus” group. As shown below, components in blue color are sharing Internal Distributed State
store of the cluster while the green colored components/boxes are worker Nodes. The worker node receive work instructions from the manager group and this is clearly shown in dash lines.Pic-3

Below picture shows how Docker Engine Swarm Mode nodes works together:


For operation team, it might be relief-tablet as there is no need of any external key-value store like etcd and consul.Docker Engine 1.12 has internal distributed state store to coordinate and hence no longer single point of failure. Additionally, Docker security is no longer an additional implementation,the secure mode is enabled by default.

Getting started with Docker Engine 1.12

Under this blog post, I will cover the following aspects:

  1. Initializing the Swarm Mode
  2. Creating the services and Tasks
  3. Scaling the Service
  4. Rolling Updates
  5. Promoting the node to Manager group

To test drive Docker Mode, I used 4 node cluster in Google Cloud Engine all running the latest stable Ubuntu 16.04 system as shown below:


Setting up docker 1.12-rc2 on all the nodes should be simple enough with the below command:

                                                      #curl -fsSL | sh

Run the below command to initialize Swarm Mode under the master node:


Let’s look at docker info command:


Listing the Docker Swarm Master node:


Let us add the first Swarm agent node(worker node) as shown below:


Let’s go back to Swarm Master Node to see the latest Swarm Mode status:


Similarly, we can add the 2nd Swarm agent node to Swarm Mode list:


Finally, we see all the nodes listed:


Let’s add 3rd Swarm Agent node in the similar fashion as shown above:


Finally, the list of worker and master nodes gets displayed as shown below:

Let’s try creating a single service:


As of now, we dont have any service created. Let’s start creating a service called collab which uses busybox image from Dockerhub and all it does is ping website.


Verifying and inspecting the services is done through the below command:




Quick Look at Scaling !!!

Task is an atomic unit of service.We actually create a task whenever we add a new service. For example, as shown below we created a task called collab.


Let’s scale this service to 5:


Now you can see that the service has been scaled to 5.


Rolling Updates Made Easy

Updating a service is pretty simple. The “docker service update” is feature rich and provides loads of options to play around with the service.


Let’s try updating redis container from 3.0.6 to 3.0.7 with 10s delay and parallelism count of 2.


Wow !!! Rolling updates just went flawless.

Time to promote the Agent Node to Manager Node

Let’s try to promote Swarm Agent Node-1 to Manager group as shown below:


In short, Swarm Mode is definitely a neat and powerful feature which provides an easy way to orchestrate Docker containers and replication of services. In our next post, we will look at how overlay networking works under Swarm Mode.

Setting up Docker Hosts on Google Compute Engine using Docker Machine

Estimated Reading Time: 3 minutesDocker Machine enables a simplified approach to set up Docker hosts on supported platforms, including Linux, Windows, OS X, and various cloud providers, in a standard way. As per Docker Inc.  “It automatically creates hosts, installs Docker on them, then configures the docker client to talk to them. A “machine” is the combination of a Docker host and a configured client.”

Docker Inc., in their official documentations speaks about Docker Machine compatibility with the following providers:

1. AWS
2.  Digital Ocean
3. Google Compute Engine
4. IBM Softlayer
5. Microsoft Azure && Hyper-V
6. OpenStack
7. VirtualBox
8. Rackspace
9. Ubuntu server over SSH – generic driver
10. VMware Fusion/vCloud Air/vSphere

You can play around with different releases and features of Docker Machine at

I had Windows 8.1 machine where I wanted to setup Docker Machine to work with my Google Cloud instances. Rather than logging into Google Cloud console and then trying to install manually, I preferred getting stuffs handled automatically through Docker Machine. Just few commands and I am ready to create Docker Host instances on the fly. Here is how I got Docker Machine working with Google Compute Engines.

Setting up Google Cloud SDK Platform

1. Download Google Cloud SDK from
2. As pre-requisite, you will need Python 2.7 to install it. Select All Users to get this installed automatically.




As shown above, python 2.7 is required to be installed.


Let the installation take care of python 2.7 installation.



Great !! The Google Cloud SDK is successfully installed.

Next, its time to authenticate your local Windows machine to the remote Google Cloud Compute Engine.


As stated, GCE_8

Ensure that the right project ID is entered while providing the following command: gcloud config set project PROJECT_ID.

Setting up New Container Host through Docker Machine:

Finally one can use Docker Machine to setup a new docker container Host through the following command:



You can easily see the Google Cloud Engine instances as shown below:


That’s all. You can easily setup multiple Docker hosts on Google Cloud Engine sitting on your local Windows machine through Docker Machine.