Docker Engine 1.12 comes with built-in Distribution & Orchestration System

Docker Engine 1.12 can be rightly called ” A Next Generation Docker Clustering & Distributed System”. Though Docker Engine 1.12 Final Release is around corner but the recent RC2 brings lots of improvements and exciting features. One of the major highlight of this release is Docker Swarm Mode which provides powerful yet optional ability to create coordinated groups of decentralized Docker Engines. Swarm Mode combines your engine in swarms of any scale. It’s self-organizing and self-healing. It enables infrastructure-agnostic topology.The newer version democratizes orchestration with out-of-box capabilities for multi-container on multi-host app deployments as shown below:

Built on Engine as a uniform building block for self organizing and healing group of Engines, Docker ensures that orchestration is accessible for every developer and operation user. The new Swarm Mode adopts  the de-centralized architecture rather than centralized one (key-value store) as seen in the earlier  Swarm releases. Swarm Mode uses the Raft consensus algorithm  to perform leader selection, and maintain the cluster’s states.

In Swarm Mode, all Docker Engine will unite into a cluster with management tier. It is basically master – slave system but all Docker Engine will be united and they will maintain a cluster state. Instead of running a single container, you will declare a desired state for your application which means multiple container and then engine themselves will maintain that state. Additionally, a new “docker service” feature has been added under the new release. The “docker service create” is expected to be an evolution of “docker run”. Docker run is imperative command and all it helps you  to get container up and running.The new “docker service create” command declare that you have to setup a server which can run one or more containers and those container will run , provided the state you declare for the service will be maintained in Engine / inside the distributed store based on raft consensus protocol.That brings the notion of desired state reconciliation. Whenever any node in the cluster goes down, the swarm itself will recognize that there has been deviation between the desired state and it will bring up  new instance to reconstruct the reconciliation. I highly recommend visualizing http://thesecretlivesofdata.com/raft/ to understand what does it mean.

Docker Swarm Node is used  for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and much more. Let’s see what features does Docker Swarm Mode adds to Docker Cluster functionality:Pic-6

Looking at the above features, Docker Swarm mode brings the following benefits :

  • Distributed: Swarm Mode uses the Raft Consensus Algorithm in order to coordinate and does not rely on a single point of failure to perform decisions.
  • Secure: Node communication and membership within a Swarm are secure out of the box. Swarm Mode uses mutual TLS for node authentication, role authorization and transport encryption, automating both certificate issuance and rotation.
  • Simple: Swarm Mode is operationally simple and minimizes infrastructure dependencies. It does not need an external database to operate. It uses internal distributed State store.

Below picture depicts Swarm Mode cluster architecture. Fundamentally its a master and slave architecture. Every node in a swarm is Docker Host running Docker Engine. Some of the node has privilege role called Manager.The manager node participate in “raft consensus” group. As shown below, components in blue color are sharing Internal Distributed State
store of the cluster while the green colored components/boxes are worker Nodes. The worker node receive work instructions from the manager group and this is clearly shown in dash lines.Pic-3

Below picture shows how Docker Engine Swarm Mode nodes works together:
Pic-9

 

For operation team, it might be relief-tablet as there is no need of any external key-value store like etcd and consul.Docker Engine 1.12 has internal distributed state store to coordinate and hence no longer single point of failure. Additionally, Docker security is no longer an additional implementation,the secure mode is enabled by default.

Getting started with Docker Engine 1.12

Under this blog post, I will cover the following aspects:

  1. Initializing the Swarm Mode
  2. Creating the services and Tasks
  3. Scaling the Service
  4. Rolling Updates
  5. Promoting the node to Manager group

To test drive Docker Mode, I used 4 node cluster in Google Cloud Engine all running the latest stable Ubuntu 16.04 system as shown below:

GCE-0

Setting up docker 1.12-rc2 on all the nodes should be simple enough with the below command:

                                                      #curl -fsSL https://test.docker.com/ | sh

Run the below command to initialize Swarm Mode under the master node:

Snap-1

Let’s look at docker info command:

Snp-2

Listing the Docker Swarm Master node:

Snap-3

Let us add the first Swarm agent node(worker node) as shown below:

Snap-4

Let’s go back to Swarm Master Node to see the latest Swarm Mode status:

Snap-5

Similarly, we can add the 2nd Swarm agent node to Swarm Mode list:

agent-2

Finally, we see all the nodes listed:

Snap-7

Let’s add 3rd Swarm Agent node in the similar fashion as shown above:

Snap-8

Finally, the list of worker and master nodes gets displayed as shown below:

Let’s try creating a single service:

Snap-11

As of now, we dont have any service created. Let’s start creating a service called collab which uses busybox image from Dockerhub and all it does is ping collabnix.com website.

Snap-12

Verifying and inspecting the services is done through the below command:

Snap12

 

 

Quick Look at Scaling !!!

Task is an atomic unit of service.We actually create a task whenever we add a new service. For example, as shown below we created a task called collab.

Snap15

Let’s scale this service to 5:

Snap16

Now you can see that the service has been scaled to 5.

scal-listing

Rolling Updates Made Easy

Updating a service is pretty simple. The “docker service update” is feature rich and provides loads of options to play around with the service.

snap-4

Let’s try updating redis container from 3.0.6 to 3.0.7 with 10s delay and parallelism count of 2.

try-1

Wow !!! Rolling updates just went flawless.

Time to promote the Agent Node to Manager Node

Let’s try to promote Swarm Agent Node-1 to Manager group as shown below:

Promotion

In short, Swarm Mode is definitely a neat and powerful feature which provides an easy way to orchestrate Docker containers and replication of services. In our next post, we will look at how overlay networking works under Swarm Mode.

23 thoughts on “Docker Engine 1.12 comes with built-in Distribution & Orchestration System

  1. Oscar

    A very good introduction and tutorial. If I have you permission, I would like to translate the series (including networking model overview) into chinese and put it at wechat page (with source URL on the page). I will appreciate it if you could authorize me to do that. Thank you

  2. You have made some decent points there. I looked on the internet to find out more
    about the issue and found most individuals will go along with your views on this website. http://www.yahoo.net

  3. Touche. Outstanding arguments. Maintain the great effort.

  4. Thanks a bunch for sharing this with all people you actually
    know what you are talking approximately! Bookmarked.
    Please additionally discuss with my site =). We could have a link exchange contract between us

  5. It is appropriatᥱ time to maкe ѕome plans fоr the longer term and it iѕ time
    to bee ɦappy. І have rеad this post аnd if Ӏ may just
    І desire tо recommend ʏoᥙ fеw interesting issues oг suggestions.
    Mɑybe you could ᴡrite subsequent articles гegarding thіs article.
    I want to rᥱad even morе issues аpproximately it!

  6. Hmm it appears like your blog ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well am an aspiring blog blogger but I’m still new to everything. Do you have any recommendations for rookie blog writers? I’d really appreciate it.

  7. It’s really a cool and useful piece of information. I am glad that you shared this useful information with us. Please keep us up to date like this. Thanks for sharing.

  8. Howdy! Do you use Twitter? I’d like to follow you if that would be okay.

    I’m absolutely enjoying your blog and look forward to new
    updates.

  9. To start with I would like to say fantastic blog! I needed a
    fast question which I’d prefer to ask should you not mind.
    I used to be curious to find out the way you center yourself and clear your ideas prior to writing.
    I have had a tricky time clearing my mind in getting my thoughts available.
    I truly do take pleasure in writing nevertheless it just appears like the first ten or fifteen minutes are generally wasted just trying to
    puzzle out how to begin. Any recommendations or hints?

    Thanks!

  10. Hello, I like reading all your post. I wanted to write a
    little comment to support you.

  11. This is really interesting, You’re a very skilled blogger.
    I have joined your rss feed and look forward to seeking more of your wonderful post.
    Also, I have shared your website in my social networks!

    1. Thanks for your time. Feel free to provide your feedback to improve this website.

  12. Hello there! Do you use Twitter? I’d like to follow
    you if that would be okay. I’m undoubtedly enjoying your blog and look forward to new
    updates.

  13. Hi there! This is kind of off topic but I need some help from an established blog.

    Is it tough to set up your own blog? I’m not very techincal but I can figure things out pretty fast.
    I’m thinking about creating my own but I’m not sure where to start.

    Do you have any ideas or suggestions? Thank you

  14. Usually I do not rerad aricle on blogs, butt I wish
    to saay that this write-up very compelled me to try and do it!

    Your writing style has bedn amazed me. Thank you, very great post. http://mocarny.eu/dive1

  15. Hey! This is my first visit to your blog! We
    are a team of volunteers and starting a whole new initiative within a community from the
    same niche. Your site provided us useful information to work on. You
    possess done a outstanding job!

Leave a Reply

Your email address will not be published. Required fields are marked *