Docker CE 19.03.0 Beta 1 went public 2 week back. It was the first release which arrived with sysctl support for Docker Swarm Mode for the first time. This is definitely a great news for popular communities like Elastic Stack, Redis etc. as they rely on tuning the kernel parameter to get rid of memory exceptions. For example, Elasticsearch uses a
mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions, hence one need to increase the limits everytime using sysctl tool. Great to see that Docker Inc. acknowledged the fact that kernel tuning is required sometimes and provides explicit support under Docker 19.03.0 Pre-Release. Great Job !
Wait..Do I really need sysctl?
Say, you have deployed your application on Docker Swarm. it’s pretty simple and it’s working great. Your application is growing day by day and now you just need to scale it. How are you going to do it? The simple answer is:
Surely, it is possible today but your containers can quickly hit kernel limits. One of the most popular kernel parameter is
docker service scale app=<number of tasks>.
. This parameter represents the maximum number of connections that can be queued for acceptance. The default value on Linux is 128, which is rather low.
The Linux kernel is flexible, and you can even modify the way it works on the fly by dynamically changing some of its parameters, thanks to the sysctl command. The sysctl programs allow to limit system-wide resource use. This can help a lot in system administration, e.g. when a user starts too many processes and therefore makes the system unresponsive for other users. Sysctl basically provides an interface that allows you to examine and change several hundred kernel parameters in Linux or BSD. Changes take effect immediately, and there’s even a way to make them persist after a reboot. By using sysctl judiciously, you can optimize your box without having to recompile the kernel and get the results immediately.
Please note that Not all sysctls are namespaced as of Docker 19.03.0 CE Pre-Release. Docker does not support changing sysctls inside of a container that also modify the host system.
Docker does support setting namespaced kernel parameters at runtime & runc honors this. Have a look:
$ docker run --runtime=runc --sysctl net.ipv4.ip_forward=1 -it alpine sh Unable to find image 'alpine:latest' locally latest: Pulling from library/alpine bdf0201b3a05: Pull complete Digest: sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913 Status: Downloaded newer image for alpine:latest / # sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 / #
It is important to note that sysctl support is not new to Docker. The support for sysctl in Docker Compose all started during compose file format v2.1.
For example, to set Kernel parameters in the container, you can use either an array or a dictionary.
sysctls: net.core.somaxconn: 1024 net.ipv4.tcp_syncookies: 0 sysctls: - net.core.somaxconn=1024 - net.ipv4.tcp_syncookies=0
Under this blog post, I will showcase how to use sysctl under 2-Node Docker Swarm Cluster. Let us get started –
Downloading the static binary archive. Go to https://download.docker.com/linux/static/stable/ (or change stable to nightly or test), choose your hardware platform, and download the .tgz file relating to the version of Docker CE you want to install.
Captain'sBay==>wget https://download.docker.com/linux/static/test/x86_64/docker-19.03.0-beta1.tgz --2019-04-10 09:20:01-- https://download.docker.com/linux/static/test/x86_64/docker-19.03.0-beta1.tgz Resolving download.docker.com (download.docker.com)... 126.96.36.199, 188.8.131.52, 184.108.40.206, ... Connecting to download.docker.com (download.docker.com)|220.127.116.11|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 62701372 (60M) [application/x-tar] Saving to: ‘docker-19.03.0-beta1.tgz’ docker-19.03.0-beta1.tgz 100%[=====================================>] 59.80M 10.7MB/s in 7.1s 2019-04-10 09:20:09 (8.38 MB/s) - ‘docker-19.03.0-beta1.tgz’ saved [62701372/62701372] Extract the archive You can use the tar utility. The dockerd and docker binaries are extracted.
Extract the tar file
Captain'sBay==>tar xzvf docker-19.03.0-beta1.tgz docker/ docker/ctr docker/containerd-shim docker/dockerd docker/docker-proxy docker/runc docker/containerd docker/docker-init docker/docker Captain'sBay==>
Move the binaries to executable path
Move the binaries to a directory on your executable path It could be such as /usr/bin/. If you skip this step, you must provide the path to the executable when you invoke docker or dockerd commands.
Captain'sBay==>sudo cp -rf docker/* /usr/local/bin/
$ sudo dockerd & Client: Docker Engine - Community Version: 19.03.0-beta1 API version: 1.40 Go version: go1.12.1 Git commit: 62240a9 Built: Thu Apr 4 19:15:07 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.0-beta1 API version: 1.40 (minimum version 1.12) Go version: go1.12.1 Git commit: 62240a9 Built: Thu Apr 4 19:22:34 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.2.5 GitCommit: bb71b10fd8f58240ca47fbb579b9d1028eea7c84 runc: Version: 1.0.0-rc6+dev GitCommit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30 docker-init: Version: 0.18.0 GitCommit: fec3683 Captain'sBay==>
Captain'sBay==>sudo docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 1b930d010525: Pull complete Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535 Status: Downloaded newer image for hello-world:latest INFO[2019-04-10T09:26:23.338596029Z] shim containerd-shim started address="/containerd-shim/m oby/5b23a7045ca683d888c9d1026451af743b7bf4005c6b8dd92b9e95e125e68134/shim.sock" debug=false pid=2953 Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ ## Verifying the new `docker context` command
root@DebianBuster:~# docker version Client: Version: 19.03.0-beta1 API version: 1.40 Go version: go1.11.5 Git commit: 62240a9 Built: Thu Apr 4 19:18:53 2019 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.0-beta1 API version: 1.40 (minimum version 1.12) Go version: go1.11.5 Git commit: 62240a9 Built: Thu Apr 4 19:17:35 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.5 GitCommit: bb71b10fd8f58240ca47fbb579b9d1028eea7c84 runc: Version: 1.0.0-rc6+dev GitCommit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30 docker-init: Version: 0.18.0 GitCommit: fec3683 root@DebianBuster:~#
Creating a 2-Node Docker Swarm Mode Cluster
swarm-node-1:~$ sudo docker swarm init --advertise-addr 10.140.0.6 --listen-addr 10.140.0 .6:2377 Swarm initialized: current node (c78wm1g99q1a1g2sxiuawqyps) is now a manager. To add a worker to this swarm, run the following command: docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxfztwgj2r443337mja-cmhuu258lu0327 e32l0g4pl47 10.140.0.6:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
swarm-node2:~$ sudo docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxf ztwgj2r443337mja-cmhuu258lu0327e32l0g4pl47 10.140.0.6:2377 This node joined a swarm as a worker.
$ sudo docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION rf3xns913p4tlprmu98z2o8hi swarm-node2 Ready Active 19.03.0-beta1 isbcijzlrft3ahpbzhgipwr9a * swarm-node-1 Ready Active Leader 19.03.0-beta1
Running Multi-service Docker Compose for Redis
Redis is an open source, in-memory data structure store, used as a database, cache and message broker. Redis Commander is an application that allows users to explore a Redis instance through a browser. Let us look at the below Docker compose file for Redis as well as Redis Commander shown below:
version: '3' services: redis: hostname: redis image: redis redis-commander: hostname: redis-commander image: rediscommander/redis-commander:latest restart: always environment: - REDIS_HOSTS=local:redis:6379 ports: - "8081:8081"
Ensure that Docker Compose is installed on your system using the below commands:
curl -L https://github.com/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose
Run the below command to bring up Redis application running on Docker Swarm Mode:
sudo docker stack deploy -c docker-compose.yml myapp
$ sudo docker stack deploy -c docker-compose.yml myapp Ignoring unsupported options: restart Creating network myapp_default Creating service myapp_redis-commander Creating service myapp_redis
Verifying if the services are up and running:
~$ sudo docker stack ls NAME SERVICES ORCHESTRATOR myapp 2 Swarm ~$ sudo docker service ls ID NAME MODE REPLICAS IMAGE PORTS ucakpqi7ozg1 myapp_redis replicated 1/1 redis:latest fxor8v90a4m0 myapp_redis-commander replicated 0/1 rediscommander/redis -commander:latest *:8081->8081/tcp
Checking the service logs:
$ docker service logs -f myapp3_redis myapp3_redis.1.7jpnbigi8kek@manager1 | 1:C 17 Apr 2019 06:26:08.006 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo myapp3_redis.1.7jpnbigi8kek@manager1 | 1:C 17 Apr 2019 06:26:08.006 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started myapp3_redis.1.7jpnbigi8kek@manager1 | 1:C 17 Apr 2019 06:26:08.006 # Warning: no config file specified, using the default config. In order to specify a configfile use redis-server /path/to/redis.conf myapp3_redis.1.7jpnbigi8kek@manager1 | 1:M 17 Apr 2019 06:26:08.009 * Running mode=standalone, port=6379. myapp3_redis.1.7jpnbigi8kek@manager1 | 1:M 17 Apr 2019 06:26:08.009 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
As you see above, there is a warning around /proc/sys/net/core/somaxconn lower value set to 128.
Building Docker Compose File using Sysctl parameter
Let us try to build a new Docker compose file with sysctl parameter specified:
Copy the below content and save it as a docker-compose.yml file.
version: '3' services: redis: hostname: redis image: redis sysctls: net.core.somaxconn: 1024 redis-commander: image: rediscommander/redis-commander:latest restart: always environment: - REDIS_HOSTS=local:redis:6379 ports: - "8081:8081"
Running Your Redis application
$ sudo docker stack deploy -c docker-compose.yml myapp Ignoring unsupported options: restart Creating network myapp_default Creating service myapp_redis Creating service myapp_redis-commander $ sudo docker service ls ID NAME MODE REPLICAS IMAGE PORTS 2oxhaychob7s myapp_redis replicated 1/1 redis:latest pjdwti7hkg1q myapp_redis-commander replicated 1/1 rediscommander/redis -commander:latest *:80->8081/tcp
Verifying the Redis service logs
$ sudo docker service logs -f myapp_redis myapp_redis.1.mp57syo3okka@swarm-node-1 | 1:C 17 Apr 2019 06:59:44.510 # oO0OoO0OoO0Oo Redis is start ing oO0OoO0OoO0Oo myapp_redis.1.mp57syo3okka@swarm-node-1 | 1:C 17 Apr 2019 06:59:44.510 # Redis version=5.0.4, bits=64 , commit=00000000, modified=0, pid=1, just started myapp_redis.1.mp57syo3okka@swarm-node-1 | 1:M 17 Apr 2019 06:59:44.511 * Running mode=standalone, port=6379.
You can see that the warning around /proc/sys/net/core/somaxconn is no longer being displayed which shows that the sysctls parameter has really worked.
In my next blog post, I will talk around rootless Docker and how to get it tested. Stay tuned !