When Kubernetes Meet Docker Swarm for the First time under Docker for Mac 17.12 Release

Estimated Reading Time: 7 minutes

Docker For Mac 17.12 GA is the first release which includes both the orchestrators – Docker Swarm & Kubernetes under the same Docker platform. As of 1/7/2018 – Experimental Kubernetes has been released under Edge Release(still not available under D4M Stable Release). Experimental Kubernetes is still not available for Docker for Windows & Linux platform. It is slated to be available for Docker for Windows next month(mid of February) and then for Linux by March or April.

Now you might ask why Docker Inc. is making this announcement? What is the fun of having 2 orchestrator under the same roof?  To answer this, let me go little back to the past and see how Docker platform looked like:

                                                                                                                             ~ Source – Docker Inc.

 

Docker platform is like a stack with various layers. The first base layer is called containerd. Containerd is an industry-standard core container runtime with an emphasis on simplicity, robustness and portability. Based on the Docker Engine’s core container runtime, it is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc. Containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users. It basically includes a daemon exposing gRPC API over a local UNIX socket. The API is a low-level one designed for higher layers to wrap and extend. It also includes a barebone CLI (ctr) designed specifically for development and debugging purpose. It uses runC to run containers according to the OCI specification. The code can be found on GitHub, and here are the contribution guidelines. Let us accept the fact that over the last few years, there has been lots of iteration around this layer but now Docker Inc. has finalised it to a robust, popular and widely accepted container runtime.

On top of containerd, there is an orchestration layer rightly called Docker Swarm. Docker Swarm ties all of your individual machines together which runs container runtime. It allows you to deploy application not on a single machine at a time but into a whole system, thereby making your application distributed.

To take advantage of these layers, as a developer you need tools & environment which can build & package your application that takes advantage of your environment, hence Docker Inc. provides Community Edition like  Docker for Mac, Docker for Windows etc. If you are considering to move your application to the production, Docker Enterprise Edition is the right choice.

If the stack looks really impressive, why again the change in architecture?

The reason is – Not everybody uses Swarm.

~ Source – Docker Inc.

Before Swarm & Kubernetes Integration – If you are a developer and you are using Docker, the workflow look something like as shown below. A Developer typically uses Docker for Mac or Docker for Windows.Using a familiar docker build, docker-compose build tool you build your environment and ensure that it gets deployed across a single node cluster OR use docker stack deploy to deploy it across the multiple cluster nodes.

~ Source – Docker Inc.

 

If your production is in swarm, then you can test it locally on Swarm as it is already inbuilt in Docker platform. But if your production environment runs in Kubernetes, then surely there is lot of work to be done like translating files, compose etc. using 3rd party open source tools and negotiating with their offerings. Though it is possible today but it is not still smooth as Swarm Mode CLI.

With the newer Docker platform, you can seamlessly use both Swarm and Kubernetes flawlessly. Interestingly, you use the same familiar tools like docker stack ls, docker stack deploy, docker ps, `docker stack ps`to display Swarm and Kubernetes containers. Isn’t it cool? You don’t need to learn new tools to play around with Kubernetes cluster.

~ Source – Docker Inc.

 

The new Docker platform includes both Kubernetes and Docker Swarm side by side and at the same level as shown below. Please note that it is a real kubernetes sitting next to Docker Swarm and NOT A FORK OR WRAPPER.

                                                                                                 ~ Source – Docker Inc.

Still not convinced why this announcement?

 

                                                                                                  ~ Source – Docker Inc.

How does SWARM CLI builds Kubernetes cluster side-by-side?

The docker compose file analyses the input file format and convert it to pods along with creating replicas set as per the instruction set. With the newer Docker for Mac 17.12 release, a new stack command has been added as the first class citizen to Kubernetes CLI.

 

Ajeets-MacBook-Air:~ ajeetraina$ kubectl get stacks -o wide
NAME      AGE
webapp    1h

 

 

Important Points –

 

  • Future Release of Docker Platform will include both orchestration options available – Kubernetes and Swarm
  • Swarm CLI will be used for Cluster Management while for orchestration you have a choice of Kubernetes & Swarm
  • Full Kubernetes API is exposed in the stack, hence support for overall Kubernetes Ecosystem is possible.
  • Docker Stack Deploy will be able to target both of Swarm or Kubernetes.
  • Kubernetes is recommended for the production environment
  • Running both Swarm & Kubernetes is not recommended for the production environment.
  • AND by now, you must be convinced – “SWARM MODE CLI is NOT GOING ANYWHERE”

Let us test drive the latest Docker for Mac 17.12 beta release and see how Swarm CLI can be used to bring up both Swarm and Kubernetes cluster seamlessly.

  • Ensure that you have Docker for Mac 17.12 Edge Release running on your Mac system. If you still don’t see 17.12-kube_beta client version, I suggest you to go through my last blog post.

A First Look at Kubernetes Integrated Docker For Mac Platform

 

 

Please note that Kubernetes/kubectl comes by default with Docker for Mac 17.12 Beta release. YOU DON”T NEED TO INSTALL KUBERNETES. By default, a single node cluster is already setup for you by default.

As we have Kubernetes & Swarm Orchestration already present, let us head over to build NGINX services as piece of demonstration on this single node Cluster node.

Writing a Docker Compose File for NGINX containers

Let us write a Docker compose file for  nginx image and deploy 3 containers of that image. This is how my docker-compose.yml looks like:

 

Deploying Application Stack using docker stack deploy 

Ajeets-Air:mynginx ajeetraina$ DOCKER_ORCHESTRATOR=kubernetes docker stack deploy --compose-file docker-compose.yml webapp
Stack webapp was created
Waiting for the stack to be stable and running…
-- Service nginx has one container running
Stack webapp is stable and running

 

Verifying the NGINX replica sets through the below command:

 

As shown above, there are 3 replicas of the same NGINX image running as containers.

Verify the cluster using Kubectl CLI displaying the stack information:

Ajeets-MacBook-Air:mynginx ajeetraina$ kubectl get stack -o wide
NAME      AGE
webapp    8h

As you see, kubectl and stack deploy displays the same cluster information.

Verifying the cluster using kubectl CLI displaying YAML file:

You can verify that Docker analyses the docker-compose.yaml input file format and  convert it to pods along with creating replicas set as per the instruction set which can be verified using the below YAML output format.

 

 

We can use the same old stack deploy CLI to verify the cluster information

 

Managing Docker Stack

Ajeets-MacBook-Air:mynginx ajeetraina$ docker stack services webapp
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
20e31598-e4c        nginx               replicated          3/3                 nginx               *:82->80/tcp,*:444->443/tcp

 

It’s time to verify if the NGINX webpage comes up well:

 

Hence, we saw that NGINX service is running both on Kubernetes & Swarm Cluster side by side.

Cleaning up

Run the below Swarm CLI related command to clean up the NGINX service as shown below:

docker stack ls
docker stack rm webapp
kubectl get pods

Output:

Want to see this in action?

https://asciinema.org/a/8lBZqBI3PWenBj6mSPzUd6i9Y

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina.

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

 

Docker, Prometheus & Pushgateway for NVIDIA GPU Metrics & Monitoring

Estimated Reading Time: 5 minutes

In my last blog post, I talked about how to get started with NVIDIA docker & interaction with NVIDIA GPU system. I demonstrated NVIDIA Deep Learning GPU Training System, a.k.a DIGITS by running it inside Docker container. ICYMI –  DIGITS is essentially a webapp for training deep learning models and is used to rapidly train the highly accurate deep neural network (DNNs) for image classification, segmentation and object detection tasks.The currently supported frameworks are: Caffe, Torch, and Tensorflow. It simplifies common deep learning tasks such as managing data, designing and training neural networks on multi-GPU systems, monitoring performance in real time with advanced visualizations, and selecting the best performing model from the results browser for deployment. 

 

In a typical HPC environment where you run 100 and 100s of NVIDIA GPU equipped cluster of nodes, it becomes important to monitor those systems to gain insight of the performance metrics, memory usage, temperature and utilization. . Tools like Ganglia & Nagios etc. are very popular due to their scalable  & distributed monitoring architecture for high-performance computing systems such as clusters and Grids. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. But with the advent of container technology, there is a need of modern monitoring tools and solutions which works well with Docker & Microservices. 

It’s all modern world of Prometheus Stack…

Prometheus is 100% open-source service monitoring system and time series database written in Go.It is a full monitoring and trending system that includes built-in and active scraping, storing, querying, graphing, and alerting based on time series data. It has knowledge about what the world should look like (which endpoints should exist, what time series patterns mean trouble, etc.), and actively tries to find faults.

How is it different from Nagios?

Though both serves a purpose of monitoring, Prometheus wins this debate with the below major points – 

  • Nagios is host-based. Each host can have one or more services, which has one check.There is no notion of labels or a query language. But Prometheus comes with its robust query language called “PromQL”. Prometheus provides a functional expression language that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus’s expression browser, or consumed by external systems via the HTTP API.
  • Nagios is suitable for basic monitoring of small and/or static systems where blackbox probing is sufficient. But if you want to do whitebox monitoring, or have a dynamic or cloud based environment then Prometheus is a good choice.
  • Nagios is primarily just about alerting based on the exit codes of scripts. These are called “checks”. There is silencing of individual alerts, however no grouping, routing or deduplication.

Let’s talk about Prometheus Pushgateway..

Occasionally you will need to monitor components which cannot be scraped. They might live behind a firewall, or they might be too short-lived to expose data reliably via the pull model. The Prometheus Pushgateway allows you to push time series from these components to an intermediary job which Prometheus can scrape. Combined with Prometheus’s simple text-based exposition format, this makes it easy to instrument even shell scripts without a client library.

The Prometheus Pushgateway allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus. It is important to understand that the Pushgateway is explicitly not an aggregator or distributed counter but rather a metrics cache. It does not have statsd-like semantics. The metrics pushed are exactly the same as you would present for scraping in a permanently running program.For machine-level metrics, the textfile collector of the Node exporter is usually more appropriate. The Pushgateway is intended for service-level metrics. It is not an event store

Under this blog post, I will showcase how NVIDIA Docker, Prometheus & Pushgateway come together to  push NVIDIA GPU metrics to Prometheus Stack.

Infrastructure Setup:

  • Docker Version: 17.06
  • OS: Ubuntu 16.04 LTS
  • Environment : Managed Server Instance with GPU
  • GPU: GeForce GTX 1080 Graphics card

Cloning the GITHUB Repository

Run the below command to clone the below repository to your Ubuntu 16.04 system equipped with GPU card:

git clone https://github.com/ajeetraina/nvidia-prometheus-stats

Script to bring up Prometheus Stack(Includes Grafana)

Change to nvidia-prometheus-stats directory with proper execute permission & then execute the ‘start_containers.sh’ script as shown below:

cd nvidia-prometheus-stats
sudo chmod +x start_containers.sh
sudo sh start_containers.sh

This script will bring up 3 containers in sequence – Pushgateway, Prometheus & Grafana

Executing GPU Metrics Script:

NVIDIA provides a python module for monitoring NVIDIA GPUs using the newly released Python bindings for NVML (NVIDIA Management Library). These bindings are under BSD license and allow simplified access to GPU metrics like temperature, memory usage, and utilization.

Next, under the same directory, you will find a python script called “test.py”.

Execute the script (after IP under line number – 124 as per your host machine) as shown below:

sudo python test.py

That’s it. It is time to open up Prometheus & Grafana UI under http://<IP-address>:9090

Just type gpu under the Expression section and you will see the list of GPU metrics automatically turned up as shown below:

Accessing the targets

Go to Status > Targets to see what targets are accessible. The Status should show up UP.

Click on Push gateway Endpoint to access the GPU metrics in details as shown:

Accessing Grafana

 You can access Grafana through the below link:

                                          http://<IP-address>:3000

 

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

 

Walkthrough: Enabling IPv6 Functionality for Docker & Docker Compose

Estimated Reading Time: 5 minutes

By default, Docker assigns IPv4 addresses to containers. Does Docker support IPv6 protocol too? If yes, how complicated is to get it enabled? Can I use docker-compose to build micro services which uses IPv6 addresses? What if I work for a company where our services run natively under IPv6 only environment? How shall I build Multi-Node Cluster setup using IPv6? Does Docker 17.06 Swarm Mode support IPv6?

I have been reading numerous queries, GITHUB issues around breaking IPv6 configuration while upgrading Docker version, issues related to IPv6 changes with host configuration etc. and just thought to share few of the findings around IPv6 effort ongoing in Docker upcoming releases.

Does Docker support IPv6 Protocol?

Yes. Support for IPv6 address has been there since Docker Engine 1.5 release.As of Docker 17.06 version (which is the latest stable release as of August 2017) by default, the Docker server configures the container network for IPv4 only. You can enable IPv4/IPv6 dualstack support by adding the below entry under daemon.json file as shown below:

 

File: /etc/docker/daemon.json

{
“ipv6”: true,
“fixed-cidr-v6”: “2001:db8:1::/64”
}

This is very similar to old way of running the Docker daemon with the --ipv6 flag. Docker will set up the bridge docker0 with the IPv6 link-local address fe80::1.

Why did we add “fixed-cidr-v6”: “2001:db8:1::/64” entry?

By default, containers that are created will only get a link-local IPv6 address. To assign globally routable IPv6 addresses to your containers you have to specify an IPv6 subnet to pick the addresses from. Setting the IPv6 subnet via the --fixed-cidr-v6 parameter when starting Docker daemon will help us achieve globally routable IPv6 address.

The subnet for Docker containers should at least have a size of /80. This way an IPv6 address can end with the container’s MAC address and you prevent NDP neighbor cache invalidation issues in the Docker layer.

With the --fixed-cidr-v6 parameter set Docker will add a new route to the routing table. Further IPv6 routing will be enabled (you may prevent this by starting dockerd with --ip-forward=false).

Let us closely examine the changes which Docker Host undergoes before & after IPv6 Enablement:

A Typical Host Network Configuration – Before IPv6 Enablement 

 

As shown above, before IPv6 protocol is enabled, the docker0 bridge network shows IPv4 address only.

Let us enable IPv6 on the Host system. In case you find daemon.json already created under /etc/docker directory, don’t delete the old entries, rather just add these two below entries into the file as shown:

{
“ipv6”: true,
“fixed-cidr-v6”: “2001:db8:1::/64”
}

 

 

Restarting the docker daemon to reflect the changes:

sudo systemctl restart docker

 

A Typical Host Network Configuration – After IPv6 Enablement 

Did you see anything new? Yes, the docker0 now gets populated with IPV6 configuration.(shown below)

docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:06:62:82:4d brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 scope global docker0 valid_lft forever preferred_lft forever
inet6 2001:db8:1::1/64 scope global tentative
valid_lft forever preferred_lft forever
inet6 fe80::1/64 scope link tentative valid_lft forever preferred_lft forever

Not only this, the docker_gwbridge network interface too received IPV6 changes:

docker_gwbridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:bc:0b:2a:84 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 scope global docker_gwbridge
valid_lft forever preferred_lft forever
inet6 fe80::42:bcff:fe0b:2a84/64 scope link
valid_lft forever preferred_lft forever

 

PING TEST – Verifying IPv6 Functionalities For Docker Host containers

Let us try bringing up two containers on the same host and see if they can ping each using IPV6 address:

 

Setting up Ubuntu Container:

mymanager1==>sudo docker run -itd ajeetraina/ubuntu-iproute bash

Setting up CentOS Container:

mymanager1==>sudo docker run -itd ajeetraina/centos-iproute bash

[Please Note: If you are using default Ubuntu or CentOS Docker Image, you will be surprised to find that ip or ifconfig command doesn’t work. You might need to install iproute package for ip command to work & net-tools package for ifconfig to work. If you want to save time, use ajeetraina/ubuntu-iproute for Ubuntu OR ajeetraina/centos-iproute for CentOS directly.]

Now let us initiate the quick ping test:

 

In this example the Docker container is assigned a link-local address with the network suffix /64 (here: fe80::42:acff:fe11:3/64) and a globally routable IPv6 address (here: 2001:db8:1:0:0:242:ac11:3/64). The container will create connections to addresses outside of the 2001:db8:1::/64 network via the link-local gateway at fe80::1 on eth0.

mymanager1==>sudo docker exec -it 907 ping6 fe80::42:acff:fe11:2
PING fe80::42:acff:fe11:2(fe80::42:acff:fe11:2) 56 data bytes
64 bytes from fe80::42:acff:fe11:2%eth0: icmp_seq=1 ttl=64 time=0.153 ms
64 bytes from fe80::42:acff:fe11:2%eth0: icmp_seq=2 ttl=64 time=0.100 ms
^C
--- fe80::42:acff:fe11:2 ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 999
msrtt min/avg/max/mdev = 0.100/0.126/0.153/0.028 ms

So the two containers are able to reach out to each other using IPv6 address.

Does Docker Compose support IPv6 protocol?

The answer is Yes. Let us verify it using docker-compose version 1.15.0 and compose file format 2.1. I faced an issue while I use the latest 3.3 file format. As Docker Swarm Mode doesn’t support IPv6, hence it is not included under 3.3 file format. Till then, let us try to bring up container using IPv6 address using 2.1 file format:

docker-compose version
version 1.15.0, build e12f3b9
docker-py version: 2.4.2
CPython version: 2.7.13
OpenSSL version: OpenSSL 1.0.1t 3 May 2016

Let us first verify the network available in the host machine:

 

File: docker-compose.yml

version: ‘2.1’
services:
  app:
    image: busybox
    command: ping www.collabnix.com
    networks:
      app_net:
        ipv6_address: 2001:3200:3200::20
networks:
  app_net:
    enable_ipv6: true
    driver: bridge
    ipam:
      driver: default
      config:
        -- subnet: 2001:3200:3200::/64
          gateway: 2001:3200:3200::1

 

The above docker-compose file will create a new network called testping_app_net based on IPv6 network under the subnet 2001:3200:3200::/64 and container should get IPv6 address automatically assigned.

Let us bring up services using docker-compose up and see if the services communicates over IPv6 protocol:

 

Verifying the IPv6 address for each container:

As shown above, this new container gets IPv6 address – 2001:3200:3200::20 and hence they are able to reach other flawlessly.

What’s Next? Under the next blog post, I am going to showcase how does IPv6 works across the multiple host machine and will talk about ongoing effort to bring IPv6 support in terms of Swarm Mode. 

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.