What’s New in Docker Engine 1.13 Swarm Mode?

Docker Engine 1.13.0 Final Release has been officially announced . With over 1050 commits, 1025 file changes and 175 days since Engine 1.12,Docker team has put a major effort to extend the Swarm Mode functionality and bug fixes/improvements. Docker 1.13.0 brings dozens of new features, a major highlights of this release includes – Centralized Logging, New Docker Management CLI, impressive New Secret API , Deploy Stack directly from Docker Compose, Auto Locking, Plugins Management and many more. With this release,Docker Inc. added support for building docker DEBs for Ubuntu 16.04 Xenial & 16.10 Yakkety Yak PPC64LE & s390x platform. There is now an inclusion of RPM builder for VMware Photon OS, Fedora 25 and DEB builder for Ubuntu 16.10 Yakkety Yak.

Under this blog, let us explore few of important new features of Docker 1.13 Swarm Mode:

Upgrading Docker Engine from 1.12 to 1.13

curl -sSL https://get.docker.com/ | sh

Docker 1.13 includes New System Management CLIs (which we will discuss later under this blog):

Experimental & Stable Release ~ all in a single binary

Experimental features are now included in the standard Docker binaries as of version 1.13.0.This is a great improvement since 1.12 release. Under Docker 1.12, there was a separate branch for experimental & stable release and one has to pull them via curl utility from test.docker.com & get.docker.com repository respectively. For enabling experimental features, you need to start the Docker daemon with --experimental flag. You can also enable the daemon flag via /etc/docker/daemon.json. e.g.

Then make sure the experimental flag is enabled:

Docker 1.13 provides you an ability to set DOCKER_HIDE_LEGACY_COMMANDS environment variable to show only the management commands. You can enable it using the below command:

DOCKER_HIDE_LEGACY_COMMANDS=true docker --help

New CLI for System Resource Management:

Docker 1.13 addresses one of the biggest issue faced by Docker users & community – how to reclaim disk space used by docker? Issues like <none>:<none> images, dangling images, getting disk full even if container consumes less spaces etc. are few of pain points which has been reported number of times.Docker 1.13 introduces new system resource management commands to help users understand how much disk space Docker is using, and help remove unused data.

Docker team has introduced a new system management CLI – docker system to help users to get information like disk usage, system-wide information and real time events from the server.

Below tables depicts the various system-level commands & its usage:

A New Centralized Logging Under Swarm Mode

Under Docker 1.12, one feature which we really missed was “Centralized Logging”. There was no docker service logs due to which Docker users has to depend upon 3rd party tool like rsyslog or ELK based solution. With Docker 1.13, a new command docker service log has been introduced.  It is a powerful new experimental command that simplifies the debugging services. Now there is NO NEED to track down hosts and containers powering a particular service and pulling logs from those containers, docker service logs utility pulls logs from all containers running a service and streams them to your console.

Want to try out this feature? As shown in the screenshot, one can easily retrieve logs from scaled-out services from various worker and master nodes using this simple API:

Let us scale-out the redis service to 5 and check if the logs collects data from all of those worker nodes and push it to master node:

Auto Locking Feature under Swarm Mode:

Docker 1.13 brings an interesting security feature called Auto Locking for Swarm Mode. To understand this concept, let us go back to raft consensus which forms the backbone of Swarm Mode implementation. By default, all of the raft consensus data is stored encrypted at rest in all the managers.A key is generated and stored on disk, so that managers can restart without operator intervention.However, if a disk gets corrupted, or one accidentally backup both the data and the key, all of the cluster data will get leaked(which starting with 1.13 might include secrets). Because of this, the newer release allow you to take ownership of this key, and you are able to enable autolock, which effectively means that the key used for encryption of your data never gets persisted to disk. Taking ownerhip of the key also means that a manager can no longer restart without human intervention, since providing the key is now of the responsibility of the operator (or external application). At any point in time you may rotate the key that is used to unlock the cluster, or give the responsibility of managing the key back to the managers, so that they can go back to restarting without external intervention.You can change which mode the cluster is operating in using docker swarm update --autolock=true/false.You can inspect what the current key is, or rotate it, by using docker swarm unlock-key

There are two ways to implement AutoLocking feature under 1.13 Swarm Mode.Either using:

docker swarm init --autolock

or

docker swarm update --autolock( to turn on manager locking).

When you run the above command, it prints out a key.On restart docker swarm unlock is necessary to start the manager. This takes the key on stdin.You can retrieve the current key with docker swarm unlock-key, or rotate it with docker swarm unlock-key --rotate. The next question could be – What happens while the swarm in locked? The swarm components do not operate until you run docker swarm unlock.

Let us look at how it actually works:-

Step-1: Let assume that we have 6-node cluster and Master1 is our Leader node.

Step-2: Let us update the “Node1” as Leader node as shown below:

Step-3: Run docker swarm update command to enable autolock. This command displays the key to unlock a swarm manager.

Step-4: One can provide the key to unlock the manager node.

Step-5: Let us test if it really works by restarting the manager node.

Step-6: After the system come back, you need to join it back as manager only when you have unlock key.

Step-7: As shown below, your Manager1 join back as the manager node:

Deploy Stack directly from docker-compose

One of the compelling feature under Docker 1.12 was introduction to “Distributed Application Bundle” rightly called as DAB. DAB helped developers to build and test containerized apps and then wrap all the app artifacts into stable .dab files. Operation teams, in turn, can take those .dab  to deploy apps by creating stacks of services from DABs.

Under Docker 1.13, the two stage process has been simplified and turned into one single command to build microservices under Swarm Mode. You don’t need to run docker-compose bundle to build .DAB file, instead Docker 1.13 adds Compose-file support to the `docker stack deploy` command so that services can be deployed using a `docker-compose.yml` file directly

Below picture depicts one single-liner command to achieve this:

To test this out, let us write a docker-compose file as shown below:

1. Create a directory called “collab” and place the below docker-compose.yml file under the same directory:

ajeetsraina81@ucp-1:~/collab$ cat docker-compose.yml
version: ‘3’
services:
db:
image: mysql:5.7
volumes:
– db_data:/var/lib/mysql
restart: always
environment:
MYSQL_ROOT_PASSWORD: wordpress
MYSQL_DATABASE: wordpress
MYSQL_USER: wordpress
MYSQL_PASSWORD: wordpress
networks:
– collabnet
wordpress:
depends_on:
– db
image: wordpress:latest
ports:
– “8000:80”
restart: always
environment:
WORDPRESS_DB_HOST: db:3306
WORDPRESS_DB_PASSWORD: wordpress
networks:
– collabnet
volumes:
db_data:
networks:
collabnet:
driver: overlay

[Please note that the above file might incur spaces and blank lines while you copy-paste directly, instead use http://lorry.io/ to validate it]

Before we run docker-compose, let us verify the listed networks:

ajeetsraina81@ucp-1:~/collab$ sudo docker network ls
NETWORK ID          NAME                 DRIVER              SCOPE
7f8a98db7b71        bridge                          bridge              local
fb456b7434a0        docker_gwbridge     bridge              local
420d78209766       host                            host                local
kpa5bmhehoe5       ingress                     overlay             swarm
40b2c7537075        none                             null                local

Building microservices using docker-compose is now one-liner command.

ajeetraina@master001:~/collab$ sudo docker stack deploy –compose-file ./docker-compose.yml myapp
Ignoring unsupported options: restart
Creating network myapp_default
Creating network myapp_collabnet
Creating service myapp_wordpress
Creating service myapp_db

Done. Let us verify if this creates list of networks:

ajeetraina@master001:~/collab$ sudo docker network ls
NETWORK ID          NAME                    DRIVER              SCOPE
7f8a98db7b71           bridge                          bridge              local
fb456b7434a0         docker_gwbridge       bridge              local
420d78209766         host                              host                  local
kpa5bmhehoe5         ingress                        overlay             swarm
zxsd88aanqa9           myapp_collabnet    overlay             swarm
wx0nbvatrvax           myapp_default        overlay             swarm
40b2c7537075           none                            null                 local

ajeetsraina81@ucp-1:~/collab$ sudo docker-compose config –services
db
wordpress

In the future blog, we will explore further with Secret Management, Plugins Management and other bugfixes/improvement which comes with 1.13 release.