Docker CE 19.03.0 Beta 1 went public 2 week back. It was the first release which arrived with sysctl support for Docker Swarm Mode for the first time. This is definitely a great news for popular communities like Elastic Stack, Redis etc. as they rely on tuning the kernel parameter to get rid of memory exceptions. For example, Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions, hence one need to increase the limits everytime using sysctl tool. Great to see that Docker Inc. acknowledged the fact that kernel tuning is required sometimes and provides explicit support under Docker 19.03.0 Pre-Release. Great Job !
Wait..Do I really need sysctl?
Say, you have deployed your application on Docker Swarm. it’s pretty simple and it’s working great. Your application is growing day by day and now you just need to scale it. How are you going to do it? The simple answer is: docker service scale app=<number of tasks>.net.core.somaxconn
The Linux kernel is flexible, and you can even modify the way it works on the fly by dynamically changing some of its parameters, thanks to the sysctl command. The sysctl programs allow to limit system-wide resource use. This can help a lot in system administration, e.g. when a user starts too many processes and therefore makes the system unresponsive for other users. Sysctl basically provides an interface that allows you to examine and change several hundred kernel parameters in Linux or BSD. Changes take effect immediately, and there’s even a way to make them persist after a reboot. By using sysctl judiciously, you can optimize your box without having to recompile the kernel and get the results immediately.
Please note that Not all sysctls are namespaced as of Docker 19.03.0 CE Pre-Release. Docker does not support changing sysctls inside of a container that also modify the host system.
Docker does support setting namespaced kernel parameters at runtime & runc honors this. Have a look:
$ docker run --runtime=runc --sysctl net.ipv4.ip_forward=1 -it alpine sh
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
bdf0201b3a05: Pull complete
Digest: sha256:28ef97b8686a0b5399129e9b763d5b7e5ff03576aa5580d6f4182a49c5fe1913
Status: Downloaded newer image for alpine:latest
/ # sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
/ # It is important to note that sysctl support is not new to Docker. The support for sysctl in Docker Compose all started during compose file format v2.1.
For example, to set Kernel parameters in the container, you can use either an array or a dictionary.
sysctls:
net.core.somaxconn: 1024
net.ipv4.tcp_syncookies: 0
sysctls:
- net.core.somaxconn=1024
- net.ipv4.tcp_syncookies=0Under this blog post, I will showcase how to use sysctl under 2-Node Docker Swarm Cluster. Let us get started –
Installing a Node with Docker 19.03.0 Beta 1 Test Build on Ubuntu 18.10
Method:I
Downloading the static binary archive. Go to https://download.docker.com/linux/static/stable/ (or change stable to nightly or test), choose your hardware platform, and download the .tgz file relating to the version of Docker CE you want to install.
Captain'sBay==>wget https://download.docker.com/linux/static/test/x86_64/docker-19.03.0-beta1.tgz
--2019-04-10 09:20:01-- https://download.docker.com/linux/static/test/x86_64/docker-19.03.0-beta1.tgz
Resolving download.docker.com (download.docker.com)... 54.230.75.15, 54.230.75.117, 54.230.75.202, ...
Connecting to download.docker.com (download.docker.com)|54.230.75.15|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 62701372 (60M) [application/x-tar]
Saving to: ‘docker-19.03.0-beta1.tgz’
docker-19.03.0-beta1.tgz 100%[=====================================>] 59.80M 10.7MB/s in 7.1s
2019-04-10 09:20:09 (8.38 MB/s) - ‘docker-19.03.0-beta1.tgz’ saved [62701372/62701372]
Extract the archive
You can use the tar utility. The dockerd and docker binaries are extracted.
Extract the tar file
Captain'sBay==>tar xzvf docker-19.03.0-beta1.tgz
docker/
docker/ctr
docker/containerd-shim
docker/dockerd
docker/docker-proxy
docker/runc
docker/containerd
docker/docker-init
docker/docker
Captain'sBay==>
Move the binaries to executable path
Move the binaries to a directory on your executable path It could be such as /usr/bin/. If you skip this step, you must provide the path to the executable when you invoke docker or dockerd commands.
Captain'sBay==>sudo cp -rf docker/* /usr/local/bin/
Start the Docker daemon:
$ sudo dockerd &
Client: Docker Engine - Community
Version: 19.03.0-beta1
API version: 1.40
Go version: go1.12.1
Git commit: 62240a9
Built: Thu Apr 4 19:15:07 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.0-beta1
API version: 1.40 (minimum version 1.12)
Go version: go1.12.1
Git commit: 62240a9
Built: Thu Apr 4 19:22:34 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.5
GitCommit: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc:
Version: 1.0.0-rc6+dev
GitCommit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
docker-init:
Version: 0.18.0
GitCommit: fec3683
Captain'sBay==>
Testing with hello-world
Captain'sBay==>sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest
INFO[2019-04-10T09:26:23.338596029Z] shim containerd-shim started address="/containerd-shim/m
oby/5b23a7045ca683d888c9d1026451af743b7bf4005c6b8dd92b9e95e125e68134/shim.sock" debug=false pid=2953
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
## Verifying the new `docker context` command
Verifying Docker version
root@DebianBuster:~# docker version
Client:
Version: 19.03.0-beta1
API version: 1.40
Go version: go1.11.5
Git commit: 62240a9
Built: Thu Apr 4 19:18:53 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.0-beta1
API version: 1.40 (minimum version 1.12)
Go version: go1.11.5
Git commit: 62240a9
Built: Thu Apr 4 19:17:35 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.5
GitCommit: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc:
Version: 1.0.0-rc6+dev
GitCommit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
docker-init:
Version: 0.18.0
GitCommit: fec3683
root@DebianBuster:~#
Creating a 2-Node Docker Swarm Mode Cluster
swarm-node-1:~$ sudo docker swarm init --advertise-addr 10.140.0.6 --listen-addr 10.140.0
.6:2377
Swarm initialized: current node (c78wm1g99q1a1g2sxiuawqyps) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxfztwgj2r443337mja-cmhuu258lu0327
e32l0g4pl47 10.140.0.6:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
Run the below command on worker node:
swarm-node2:~$ sudo docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxf
ztwgj2r443337mja-cmhuu258lu0327e32l0g4pl47 10.140.0.6:2377
This node joined a swarm as a worker.
Listing the Swarm Mode CLuster
$ sudo docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
ENGINE VERSION
rf3xns913p4tlprmu98z2o8hi swarm-node2 Ready Active
19.03.0-beta1
isbcijzlrft3ahpbzhgipwr9a * swarm-node-1 Ready Active Leader
19.03.0-beta1Running Multi-service Docker Compose for Redis
Redis is an open source, in-memory data structure store, used as a database, cache and message broker. Redis Commander is an application that allows users to explore a Redis instance through a browser. Let us look at the below Docker compose file for Redis as well as Redis Commander shown below:
version: '3'
services:
redis:
hostname: redis
image: redis
redis-commander:
hostname: redis-commander
image: rediscommander/redis-commander:latest
restart: always
environment:
- REDIS_HOSTS=local:redis:6379
ports:
- "8081:8081"Ensure that Docker Compose is installed on your system using the below commands:
curl -L https://github.com/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-composeRun the below command to bring up Redis application running on Docker Swarm Mode:
sudo docker stack deploy -c docker-compose.yml myapp$ sudo docker stack deploy -c docker-compose.yml myapp
Ignoring unsupported options: restart
Creating network myapp_default
Creating service myapp_redis-commander
Creating service myapp_redis
Verifying if the services are up and running:
~$ sudo docker stack ls
NAME SERVICES ORCHESTRATOR
myapp 2 Swarm
~$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE
PORTS
ucakpqi7ozg1 myapp_redis replicated 1/1 redis:latest
fxor8v90a4m0 myapp_redis-commander replicated 0/1 rediscommander/redis
-commander:latest *:8081->8081/tcp
Checking the service logs:
$ docker service logs -f myapp3_redis
myapp3_redis.1.7jpnbigi8kek@manager1 | 1:C 17 Apr 2019 06:26:08.006 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
myapp3_redis.1.7jpnbigi8kek@manager1 | 1:C 17 Apr 2019 06:26:08.006 # Redis version=5.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
myapp3_redis.1.7jpnbigi8kek@manager1 | 1:C 17 Apr 2019 06:26:08.006 # Warning: no config file specified, using the default config. In order to specify a configfile use redis-server /path/to/redis.conf
myapp3_redis.1.7jpnbigi8kek@manager1 | 1:M 17 Apr 2019 06:26:08.009 * Running mode=standalone, port=6379.
myapp3_redis.1.7jpnbigi8kek@manager1 | 1:M 17 Apr 2019 06:26:08.009 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
As you see above, there is a warning around /proc/sys/net/core/somaxconn lower value set to 128.
Building Docker Compose File using Sysctl parameter
Let us try to build a new Docker compose file with sysctl parameter specified:
Copy the below content and save it as a docker-compose.yml file.
version: '3'
services:
redis:
hostname: redis
image: redis
sysctls:
net.core.somaxconn: 1024
redis-commander:
image: rediscommander/redis-commander:latest
restart: always
environment:
- REDIS_HOSTS=local:redis:6379
ports:
- "8081:8081"Running Your Redis application
$ sudo docker stack deploy -c docker-compose.yml myapp
Ignoring unsupported options: restart
Creating network myapp_default
Creating service myapp_redis
Creating service myapp_redis-commander
$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE
PORTS
2oxhaychob7s myapp_redis replicated 1/1 redis:latest
pjdwti7hkg1q myapp_redis-commander replicated 1/1 rediscommander/redis
-commander:latest *:80->8081/tcpVerifying the Redis service logs
$ sudo docker service logs -f myapp_redis
myapp_redis.1.mp57syo3okka@swarm-node-1 | 1:C 17 Apr 2019 06:59:44.510 # oO0OoO0OoO0Oo Redis is start
ing oO0OoO0OoO0Oo
myapp_redis.1.mp57syo3okka@swarm-node-1 | 1:C 17 Apr 2019 06:59:44.510 # Redis version=5.0.4, bits=64
, commit=00000000, modified=0, pid=1, just started
myapp_redis.1.mp57syo3okka@swarm-node-1 | 1:M 17 Apr 2019 06:59:44.511 * Running mode=standalone, port=6379.You can see that the warning around /proc/sys/net/core/somaxconn is no longer being displayed which shows that the sysctls parameter has really worked.
In my next blog post, I will talk around rootless Docker and how to get it tested. Stay tuned !
