Top 5 Features of Docker Engine v18.09.1 That You Shouldn’t Miss Out

Estimated Reading Time: 11 minutes

Docker Engine v18.09.1 went GA last month. It was made available for both the Community and Enterprise users. Docker Enterprise is a superset of all the features in Docker Community Edition. It incorporates defect fixes that you can use in environments where new features cannot be adopted as quickly for consistency and compatibility reasons.

New in 18.09 is an aligned release model for Docker Engine – Community and Docker Engine – Enterprise. The new versioning scheme is YY.MM.x where x is an incrementing patch version. They will ship concurrently with the same x patch version based on the same code base.

[Updated – 2/15/2019]:Security fixes for Docker Engine – Enterprise and Docker Engine – Community

  • Update runc to address a critical vulnerability that allows specially-crafted containers to gain administrative privileges on the host. CVE-2019-5736
  • Ubuntu 14.04 customers using a 3.13 kernel will need to upgrade to a supported Ubuntu 4.x kernel

Docker Engine v18.09.1 comes with dozens of new features, improvements and bug fixes. Let us go through the list of Top features which you should be aware of while you upgrade to Docker 18.09.1 release.

  • Docker Engine v18.09.1 comes with containerd v1.2.2
  • BuildKit 0.3.3 is Available & is out of experimental mode
  • Support for Compose on Kubernetes
  • Exposing Product Info under docker info command
  • Process isolation on Windows 10 for the first time
  • Support for remote connections using SSH
  • Support for SSH agent socket forwarder
  • Support for “registry-mirrors” and “insecure-registries” when using BuildKit
  • Support for build-time secrets using a --secret flag when using BuildKit
  • Support for docker build –pull … when using BuildKit

Docker Engine v18.09.1 comes with containerd 1.2.2 version

Docker Engine v18.09.2 release for CE and EE both is shipped with containerd 1.2.2 version. In Docker versions prior to 18.09, containerd was managed by the Docker engine daemon. In Docker Engine 18.09, containerd is managed by systemd.

Let us try installing the latest Docker 18.09.2 on a fresh Ubuntu 18.10 VM and try verifying the containerd version.

sudo curl -sSL https://get.docker.com/ | sh
# Executing docker install script, commit: 26dda3d
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sudo -E sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sudo -E sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu cosmic edge" > /etc/apt/sources.list.d/docker.list
+ sudo -E sh -c apt-get update -qq >/dev/null
+ sudo -E sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
+ sudo -E sh -c docker version
Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:46 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:42:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false
If you would like to use Docker as a non-root user, you should now consider
adding your user to the "docker" group with something like:

  sudo usermod -aG docker robertsingh181

Remember that you will have to log out and back in for this to take effect!

WARNING: Adding a user to the "docker" group will grant the ability to run
         containers which can be used to obtain root privileges on the
         docker host.
         Refer to https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
         for more information.
@collabnix:~$ sudo usermod -aG docker robertsingh181

Verifying Docker Version

$sudo docker version
Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:46 2019
 OS/Arch:           linux/amd64
 Experimental:      false
Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:42:13 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Please Note: The client and container runtime are now in separate packages from the daemon in Docker Engine 18.09. Users should install and update all three packages at the same time to get the latest patch releases.

For example, on Ubuntu: sudo apt install docker-ce docker-ce-cli containerd.io.

$ sudo dpkg --list | grep container
ii  containerd.io                  1.2.2-3                             amd64        An open and reliable container runtime
ii  docker-ce                      5:18.09.2~3-0~ubuntu-cosmic         amd64        Docker: the open-source application container engine
ii  docker-ce-cli                  5:18.09.2~3-0~ubuntu-cosmic         amd64        Docker CLI: the open-source application container engine

Verifying Containerd Version

$ sudo containerd --help
NAME:
 containerd - 
                  __        _                     __
_________  ____  / /_____ _(_)___  ___  _________/ /
/ ___/ __ \/ __ \/ __/ __ `/ / __ \/ _ \/ ___/ __  /
/ /__/ /_/ / / / / /_/ /_/ / / / / /  __/ /  / /_/ /
\___/\____/_/ /_/\__/\__,_/_/_/ /_/\___/_/   \__,_/
high performance container runtime
USAGE:
 containerd [global options] command [command options] [arguments...]
VERSION:
 1.2.2
COMMANDS:
   config    information on the containerd config
   publish   binary to publish events to containerd
   oci-hook  provides a base for OCI runtime hooks to allow arguments to be injected.
   help, h   Shows a list of commands or help for one command
GLOBAL OPTIONS:
 --config value, -c value     path to the configuration file (default: "/etc/containerd/config.toml")
 --log-level value, -l value  set the logging level [trace, debug, info, warn, error, fatal, panic]
 --address value, -a value    address for containerd's GRPC server
 --root value                 containerd root directory
 --state value                containerd state directory
 --help, -h                   show help
 --version, -v                print the version
$ sudo ctr version
Client:
Version:  1.2.2
Revision: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
Server:
Version:  1.2.2
Revision: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce

Since containerd is managed by systemd under v18.09.1 release, any custom configuration to the docker.service systemd configuration which changes mount settings (for example, MountFlags=slave) breaks interactions between the Docker Engine daemon and containerd, and you will not be able to start containers. Run the following command to get the current value of the MountFlags property for the docker.service:

sudo systemctl show --property=MountFlags docker.service
MountFlags=

Update your configuration if this command prints a non-empty value for MountFlags, and restart the docker service.

BuildKit 0.3.3 is Available & is out of experimental mode

BuildKit is a new project under the Moby umbrella for building and packaging software using containers. It is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. Docker 18.09.0 is the first release with buildkit support. With this latest release, you can now run Buildkit without experimental mode enabled. Buildkit can now be configured with an option in daemon.json too.

The docker build is a Docker integrated tool for building images using Dockerfile. It requires Docker daemon to be running. It’s similar to docker run command but some features are intentionally removed for security reasons like no volumes(docker run -v, docker run --mount) and no privileged mode(`docker run –privileged). Buildkit is meant to become the next generation backend implementation for docker build command and github.com/docker/docker/builder package. This doesn’t mean any changes to Dockerfile format as buildkit draws a boundary between build backends and frontends. Dockerfile would be one of the frontend implementations. When invoked from the Docker CLI buildkit would be capable of exposing clients context directory as a source and use Docker containers as a worker. The snapshots would be backed by Docker’s layer store(containerD snapshot drivers). End results from the builder would be exported to docker images.

What problems does BuildKit solve for us?

If you look at Dockerfile, we are reading all commands in Dockerfile from start to the end one by one until we reach to the end.Modifying a single line always invalidates the caches of the subsequent lines. For example: N-th line is assumed to be always dependent on (N-1)th line

FROM debian
EXPOSE 80
RUN apt update && apt install git

As shown above, modifying the 2nd line(EXPOSE 80) always invalidates the apt cache due to false dependency. A user need to arrange the instructions carefully for efficient caching. This brings inefficient caching problem.

Not only this, inaccessible to private assets is another major problem with traditional Dockerfile. There is no safe way to access private assets(eg. Git Repos, S3) from build containers. Hence, copying credentials using COPY can leak the credentials accidently.Buildkit solves the above problems by using DAG-style low level language called LLB.

The main areas BuildKit improves on the current experience are performance, storage management, and extensibility. From the performance side, a significant update is a new fully concurrent build graph solver. It can run build steps in parallel when possible and optimize out commands that don’t have an impact on the final result. We have also optimized the access to the local source files. By tracking only the updates made to these files between repeated build invocations, there is no need to wait for local files to be read or uploaded before the work can begin.

Let us compare docker build Vs Buildkit and see how Buildkit builds the Docker image a more faster than traditional approach.

$ git clone https://github.com/ajeetraina/hellowhale
Cloning into 'hellowhale'...
remote: Enumerating objects: 28, done.
remote: Total 28 (delta 0), reused 0 (delta 0), pack-reused 28
Unpacking objects: 100% (28/28), done.
~$ cd hellowhale/
~$ ls
Dockerfile  README.md  html  wrapper.sh
:~/hellowhale$ time docker build -t ajeetraina/hellowhale .
Sending build context to Docker daemon  153.1kB
Step 1/4 : FROM nginx:latest
latest: Pulling from library/nginx
6ae821421a7d: Pull complete 
da4474e5966c: Pull complete 
eb2aec2b9c9f: Pull complete 
Digest: sha256:dd2d0ac3fff2f007d99e033b64854be0941e19a2ad51f174d9240dda20d9f534
Status: Downloaded newer image for nginx:latest
 ---> f09fe80eb0e7
Step 2/4 : COPY wrapper.sh /
 ---> 10d671c6cf08
Step 3/4 : COPY html /usr/share/nginx/html
 ---> 3e8a09f56168
Step 4/4 : CMD ["./wrapper.sh"]
 ---> Running in b1f24992f9e5
Removing intermediate container b1f24992f9e5
 ---> 9dae85ca0867
Successfully built 9dae85ca0867
Successfully tagged ajeetraina/hellowhale:latest
real    0m6.359s
user    0m0.035s
sys     0m0.022s

Let’s build it with buildkit:

 time docker build -t ajeetraina/hellowhale .
[+] Building 1.7s (9/9) FINISHED                                                                                                                                                 
 => [internal] load build definition from Dockerfile                                                                                                                        0.1s
 => => transferring dockerfile: 135B                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                           0.0s
 => => transferring context: 2B                                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/nginx:latest                                                                                                             0.0s
 => [internal] helper image for file operations                                                                                                                             0.4s
 => => resolve docker.io/docker/dockerfile-copy:v0.1.9@sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061                                              0.7s
 => => sha256:e8f159d3f00786604b93c675ee2783f8dc194bb565e61ca5788f6a6e9d304061 2.03kB / 2.03kB                                                                              0.0s
 => => sha256:a546a4352bcaa6512f885d24fef3d9819e70551b98535ed1995e4b567ac6d05b 736B / 736B                                                                                  0.0s
 => => sha256:494e63343c3f0d392e7af8d718979262baec9496a23e97ad110d62b9c90d6182 766B / 766B                                                                                  0.0s
 => => sha256:df3b4bed1f63b36992540a09e0d10bd3f9d0b082d50810313841d745d7cce368 898.21kB / 898.21kB                                                                          0.2s
 => => sha256:f7b6696c3fee7264ec4486cebe146a6a98aa8d1e46747843107ff473aada8d56 861.00kB / 861.00kB                                                                          0.2s
 => => extracting sha256:df3b4bed1f63b36992540a09e0d10bd3f9d0b082d50810313841d745d7cce368                                                                                   0.1s
 => => extracting sha256:f7b6696c3fee7264ec4486cebe146a6a98aa8d1e46747843107ff473aada8d56                                                                                   0.1s
 => [1/3] FROM docker.io/library/nginx:latest                                                                                                                               0.0s
 => => resolve docker.io/library/nginx:latest                                                                                                                               0.0s
 => [internal] load build context                                                                                                                                           0.0s
 => => transferring context: 34.39kB                                                                                                                                        0.0s
 => [2/3] COPY wrapper.sh /                                                                                                                                                 0.2s
 => [3/3] COPY html /usr/share/nginx/html                                                                                                                                   0.2s
 => exporting to image                                                                                                                                                      0.1s
 => => exporting layers                                                                                                                                                     0.0s
 => => writing image sha256:db60ac4c90d7412b8c9f9382711f0d97a9ad9d4a33c05200aa36dc4c935c8cb3                                                                                0.0s
 => => naming to docker.io/ajeetraina/hellowhale                                                                                                                            0.0s
real    0m1.732s
user    0m0.042s
sys     0m0.019s
~/hellowhale$

Buildkit took just 0m1.732sec compared to 0m6.359sec taken by traditional docker build method.

Below are the list of dozens of enhancements which comes around Buildkit and I shall be discussing these in details in my upcoming blog posts:

Support for Compose on Kubernetes

Compose on Kubernetes allows you to deploy Docker Compose files onto a Kubernetes cluster. Compose on Kubernetes comes installed on Docker Desktop and Docker Enterprise. On Docker Desktop you will need to activate Kubernetes in the settings to use Compose on Kubernetes.

Check out my last 2 blogs which talks about how Compose on Kubernetes work for Play with Kubernetes & Minikube.

Docker v18.09.1 now exposes Product Info under docker info command

Under Docker v18.09.1 it is now possible to verify if it is an enterprise or community product as shown below:

Process isolation on Windows 10 for the first time

Process-isolation containers were already possible on Windows Server, but for the first time they are now also available on the regular Windows 10 of your laptop. Windows 10–1809 (“October 2018 Update”) + Docker engine 18.09.1 + the Windows 1809 base images from Dockerhub are the first combination that allows you to run “real” Windows containers on Windows 10, without the need of Hyper-v virtualization.

One need to install the Docker Desktop   Edge version 2.0.1.0 or newer. Please remember that Docker Engine should be at version 18.09.1 or higher.You must select a Windows base image from Dockerhub that matches the kernel of your host’s Windows version. For Windows 10–1809, that’s the :1809version/tag of nanoserverservercore and windows (or any higher-level image, that builds up on one of these).

Let us run a test container in process-isolation mode by adding the parameter --isolation=process :

docker run --isolation=process mcr.microsoft.com/windows/nanoserver:1809 cmd.exe /c ping 127.0.0.

In case you’re on Windows 10 Build 1706 and try to run the below command, it won’t work as expected. One need to be running atleast 1809 build for smooth operation.

Support for remote connections using SSH

Docker Engine v18.09 offers the possibility for a Docker client to communicate with a remote daemon via ssh. The Docker client communicates usually with the daemon either locally, via the unix socket /var/run/docker.sock, or over a network via a tcp socket. With Docker 18.09.1 you can now SSH to remote Docker host and execute docker CLI flawlessly.

This new connection method between client and engine allows for simple and shared SSH configuration which is far more common and more easily administered that the prior custom CA/certs solution via the docker cli, it is easily done by setting the env var DOCKER_HOST=ssh://hostname or directly on the docker command using the -H parameter as in

docker -H ssh://hostname info
$ docker -H ssh://ajeetraina@10.94.26.28 run -ti ubuntu echo “hello”
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
6cf436f81810: Pull complete 
987088a85b96: Pull complete 
b4624b3efe06: Pull complete 
d42beb8ded59: Pull complete 
Digest: sha256:7a47ccc3bbe8a451b500d2b53104868b46d60ee8f5

Please note that you might need to configure SSH key based login and run an SSH agent so you only need to enter a passphrase once. Especially if you are on cloud platform you will need SSH passphrase to allow this command to work as it directly doesn’t allow SSH from one cloud instance to another.

Below are further details of features which I am planning to discuss in my upcoming blog posts:

  • Added --chown flag support for ADD and COPY commands on Windows moby/moby#35521
  • Added docker engine subcommand to manage the lifecycle of a Docker Engine running as a privileged container on top of containerd, and to allow upgrades to Docker Engine Enterprise docker/cli#1260

A Bonus..

Please note that the client and container runtime are now in separate packages from the daemon in Docker Engine 18.09. Users should install and update all three packages at the same time to get the latest patch releases. For example, on Ubuntu:sudo apt install docker-ce docker-ce-cli containerd.io.

Hope you found this blog informative. In case of any query, feel free to drop me a message or join me at DockerLabs

Clap

(15)