How I built Elastic Stack for Docker Swarm using Docker Application Packages(docker-app)

Estimated Reading Time: 7 minutes


Let’s begin with Problem Statement !

DockerHub is a cloud-based registry service which allows you to link to code repositories, build your images, test them, store manually pushed images so you can deploy images to your hosts. It provides a centralized resource for container image discovery, distribution and change management as well as workflow automation throughout the development pipeline. We share Docker images all the time, but let’s agree to the fact that we don’t have a good way of sharing the multi-service applications that use them.

Let us take an example of Elastic Stack. Built on an open source foundation, the Elastic Stack lets you reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time with the help of Elasticsearch, Logstash, Kibana and multiple other tools and technique. In order to build these tools in the form of containers, one need to start building Docker Image for each of these tools. The recommended way is constructing a Dockerfile for each of these tools. In turn, Docker Compose uses these images to build required services. Whenever the docker stack deploy CLI is used to deploy the application stack, these Docker images are pulled from Dockerhub for the first time and then picked up locally once downloaded to your system. What if you could upload your whole application stack to DockerHub? Yes, it’s possible today and docker-app is the tool which can make Compose-based applications shareable on Docker Hub and DTR.

Docker-app v0.5.0 is now Available !

Docker Application Package v0.5.0 is the latest offering from  Docker, Inc. You can download it from this link. The binaries are available for Linux, Windows and MacOS Platform. If you are looking out for source code,  this is the direct link.


The docker-app v0.5.0 comes with notable features and improvements which are listed below:

  • The improved docker-app inspect command to shows a summary of services, networks, volumes and secrets.

  • The docker-app push CLI now works on Windows and bypasses the local docker daemon by talking directly to the registry.
  • The docker-app save and docker-app ls have been obsoleted.
  • All commands now accept an application package as a URL.
  • The docker-app push command now accepts a custom repository name.
  • The docker-app completion command can generate zsh completion in addition to bash.

In my last blog post, I talked about docker-app for the first time and showcased its usage soon after I returned back from Dockercon. Under this post, I will show how I built Elastic Stack using docker-app for 5-Node Docker Swarm cluster.


  • Click on Icon near to Instance to choose 3 Managers & 2 Worker Nodes

Deploy 5 Node Docker Swarm Cluster

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUSENGINE VERSION
iy9mbeduxd4mmjxoikbn5ulds *   manager1            Ready               Active              Reachable18.03.1-ce
mx916kgqg6gfgqdr2gn1eksxy     manager2            Ready               Active              Leader18.03.1-ce
xaeq943o84g9spy6mebj64tw3     manager3            Ready               Active              Reachable18.03.1-ce
8umdv6m82nrpevuris1e45wnq     worker1             Ready               Active18.03.1-ce
o3yobqgg7wjvjw2ec5ythszgw     worker2             Ready               Active18.03.1-ce

Cloning the Repository

$ git clone
Cloning into 'app'...remote: Enumerating objects: 134, done.
remote: Counting objects: 100% (134/134), done.remote: Compressing objects: 100% (134/134), done.
remote: Total 14511 (delta 95), reused 0 (delta 0), pack-reused 14377Receiving objects: 100% (14511/14511), 17.37 MiB | 13.35 MiB/s, done.
Resolving deltas: 100% (5391/5391), done.

Install Docker-app

$ cd app/examples/elk/
[manager1] (local) root@ ~/app/examples/elk$ ls          devel              elk.dockerapp      install-dockerapp  prod
[manager1] (local) root@ ~/app/examples/elk
$ chmod +x install-dockerapp[manager1] (local) root@ ~/app/examples/elk
$ sh install-dockerappConnecting to (
Connecting to ( 100% |*************************************************************|  8895k  0:00:00 ETA
[manager1] (local) root@ ~/app/examples/elk

Verifying Docker-app Version

$ docker-app version
Version:      v0.4.0
Git commit:   525d93bc
Built:        Tue Aug 21 13:02:46 2018
OS/Arch:      linux/amd64
Experimental: off
Renderers:    none

I assume you have a docker compose file for ELK stack application already available with you. If not, you can download a sample file from this link. Place this YAML file under the same directory(app/examples/elk/). Now with docker-app installed, let’s create an Application Package based on this Compose file:

$ docker-app init elk

Once you run the above command, it create a new directory elk.dockerapp/ that contains three different YAML files:

docker-compose.yml  elk.dockerapp
[manager1] (local) root@ ~/myelk
$ tree elk.dockerapp/
├── docker-compose.yml
├── metadata.yml
└── settings.yml

0 directories, 3 files

Edit each of these files as shown to look similar to what are placed under this link.

Rendering Docker Compose file

$ docker-app render elk
version: "3.4"
  elasticsearch:    command:
    - elasticsearch    -
      mode: replicated
      replicas: 2
      ES_JAVA_OPTS: -Xms2g -Xmx2g
    image: elasticsearch:5
      elk: null
    - type: volume
      target: /usr/share/elasticsearch/data
      mode: replicated
      replicas: 2
      ELASTICSEARCH_URL: http://elasticsearch:9200
      - CMD-SHELL
      - wget -qO- http://localhost:5601 > /dev/null
      interval: 30s
      retries: 3
    image: kibana:latest
      elk: null
    - mode: ingress
      target: 5601
      published: 5601
      protocol: tcp
    - sh
    - -c
    - logstash -e 'input { syslog  { type => syslog port => 10514   } gelf { } } output
      { stdout { codec => rubydebug } elasticsearch { hosts => [ "elasticsearch" ]
      } }'
      mode: replicated
      replicas: 2
    hostname: logstash
    image: logstash:alpine
      elk: null
    - mode: ingress
      target: 10514
      published: 10514
      protocol: tcp
    - mode: ingress
      target: 10514
      published: 10514
      protocol: udp
    - mode: ingress
      target: 12201
      published: 12201
      protocol: udp
  elk: {

Setting the kernel parameter for ELK stack

sysctl -w vm.max_map_count=262144

Deploying the Application Stack

[manager1] (local) root@ ~/app/examples/elk
$ docker-app deploy elk --settings-files elk.dockerapp/settings.yml
Creating network elk_elk
Creating service elk_kibana
Creating service elk_logstash
Creating service elk_elasticsearch
[manager1] (local) root@ ~/app/examples/elk

Inspecting ELK Stack

[manager1] (local) root@ ~/app/examples/elk
$ docker-app inspect elk
myelk 0.1.0
Maintained by: Ajeet_Raina <>

ELK using Dockerapp

Setting                       Default
-------                       -------
elasticsearch.deploy.mode     replicated
elasticsearch.deploy.replicas 2      elasticsearch:5
kibana.deploy.mode            replicated
kibana.deploy.replicas        2             kibana:latest
kibana.port                   5601
logstash.deploy.mode          replicated

Verifying Stack services are up & running

[manager1] (local) root@ ~/app/examples/elk/docker101/play-with-docker/visualizer
$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
uk2whax6f3jq        elk_elasticsearch   replicated          2/2                 elasticsearch:5
nm4p3yswvh5y        elk_kibana          replicated          2/2                 kibana:latest       *:5601->56
g5ubng6rhcyp        elk_logstash        replicated          2/2                 logstash:alpine     *:10514->1
0514/tcp, *:10514->10514/udp, *:12201->12201/udp
[manager1] (local) root@192

Pushing the App Package to Dockerhub

Password:[manager1] (local) root@ ~/app/examples/elk
$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to to create one.
Username: ajeetraina
Password:Login Succeeded

Pushing the App package to DockerHub

[manager1] (local) root@ ~/app/examples/elk$ docker-app push --namespace ajeetraina --tag 1.0.2
The push refers to repository []
15e73d68a400: Pushed
1.0.2: digest: sha256:c5a8e3b7e2c7a5566a3e4247f8171516033e7e9791dfdb6ebe622d3830884d9b size: 524
[manager1] (local) root@ ~/app/examples/elk

Important Note: If you are using Docker-app v0.5.0, you might face issue related to pulling the image from Dockerhub as it report unsupported OS error message. Here’s a link to this open issue.

Testing the Application Package

Open up a new PWD window. Install docker-app as shown above and try to run the below command:

docker-app deploy ajeetraina/elk.dockerapp:1.0.2

This should bring up your complete Elastic Stack Platform.

Did you find this blog helpful?  Feel free to share your experience. Get in touch with me at twitter @ajeetsraina.

If you want to keep track of latest Docker related information, follow me at

Getting Started with OpenUSM on Docker for Windows Platform

Estimated Reading Time: 6 minutes


OpenUSM is a modern approach to DellEMC Server Management, Insight Logs Analytics and Machine Learning solution integrated with monitoring & logging pipeline using Docker containers & Redfish. It is 100% container-based platform-agnostic solution which can be run from laptop, server or cloud and works seamlessly on any of Linux or Windows platform with Docker Engine running on top of it. It follows “Container-Per-Server(CPS)” model. For each server management tasks, there are Python-scripts which when executed builds and run Docker containers, uses Redfish API to communicate directly with Dell iDRAC, collects iDRAC/LC logs and pushes it to ELK(Elasticsearch, Logstash & Kibana) stack for further log analytics and Machine Learning. OpenUSM is currently hosted at

OpenUSM today support both Linux and Windows Platform. It has already been validated on Linux OS like Debian, Ubuntu and CentOS system. You can find extensive documents here.

Under this blog post, I will showcase how to get started with OpenUSM on Docker for Windows Platform.

Tested Platform:

  • Microsoft Windows 10 Enterprise
  • X64 based PC


  • Installing Python 2.7
  • Installing Winsyslog
  • Installing Docker for Windows
  • Configuring Docker for Windows

Installing Python 2.7

To install Python 2.7, the simplest way is to use Chocolatey. Chocolatey is software management automation. Chocolatey works with over 20+ installer technologies for Windows, but it can manage things you would normally xcopy deploy (like runtime binaries and zip files). You can also work with registry settings or managing files and configurations, or any combination.

Run the below command to install Chocolatey on your Windows 10 laptop:

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString(‘’))

If you face any issue, do refer this link.

Now install python using choco as shown below:

choco install python

You can verify if python is installed or not using the below command:

PS C:\> C:\Python27\python.exe -V
Python 2.7.15

Installing WinSyslog

For OpenUSM to work, syslog server is required. You can install any Syslog server available in the internet. For this demo, I will use WinSyslog which is too easy to setup. Once you install it, it will open up the window as shown below:

Click on Options section under File Menu and you shall see the below window:

As shown above, you will need to enter your local laptop IP address and port to fetch logs for OpenUSM. You can test it by clicking on “Send” and you shall see “Syslog Messages send successfully to” if it goes well.

Installing Docker for Windows:

Open and click on “Install from Docker Store” to open up the below page to download Docker for Windows CE Edition.

Docker CE for Windows is Docker designed to run on Windows 10. It is a native Windows application that provides an easy-to-use development environment for building, shipping, and running dockerized apps. Docker CE for Windows uses Windows-native Hyper-V virtualization and networking and is the fastest and most reliable way to develop Docker apps on Windows. Docker CE for Windows supports running both Linux and Windows Docker containers. You can install either of stable or Edge release from the below link.


Double-click Docker for Windows Installer to run the installer. When the installation finishes, Docker starts automatically. The whale  in the notification area indicates that Docker is running, and accessible from a terminal.

You can verify Docker version either by visiting “About Docker” in the top menu:

Or you can open a command-line terminal like PowerShell, and try out below Docker command to check the version –

Configuring Docker for Windows for OpenUSM

We need to perform few of configuration changes related to Docker for Windows before we proceed with setting up ELK stack. First we need to enable share drives for ELK stack to work. Docker for Windows provides you a simplified approach to enable this feature. Click on Whale Icon > Shared Drives > Select “C:” local drive which will be made available to your Docker containers which run ELK Stack.

Once you select and click on “Apply” it will restart Docker as well as Kubernetes(if enabled earlier). This should be good enough for OpenUSM to work smoothly.

Cloning the OpenUSM Repository

git clone

cd openusm/logging/

Setting up ELK Stack

Docker for Windows is a development platform and comes with docker-compose installed by default. All you need is to run the below command to bring up ELK stack… Awesome, Isn’t it?

docker-compose up -d

You can verify if ELK has come up or not by running the below command as shown:

Open up to access Kibana UI as shown below:

Sending DellEMC HW Sensor Logs to ELK Stack

It’s time for an action now. I assume you are connected to your Lab infrastructure using VPN in order to access DellEMC PowerEdge servers. It just takes few seconds to send sensor logs(Fan, Temperature etc.) of DellEMC PowerEdge server sitting in your datacenter to ELK stack using the below python script.


PS C:\Users\Ajeet_Raina\openusm\logging> C:\Python27\python.exe .\ -i <idrac-ip> -ei -eu elastic -ep <password>

This script uses Redfish to talk to remote DellEMC PE Server, fetches the logs and send it to syslog server which we configured earlier, pushes it to Logstash and elasticsearch and get it displayed via Kibana UI – all in just few seconds. Isn’t it cool?

Visualizing the logs under Kibana UI

When you open kibana UI for the first time, the index pattern mightn’t come up. Click on “Index Pattern” under Management tab on the left hand side. Next, click on “Create Index Pattern”. Search for Fan* and temp*. By now, you should be able to see temperature and Fan speed logs under Discover tab.


Click on “Discover” tab to see the overall logs fetched directly from iDRAC IPs.


Click on “Visualize” tab to add filter. In the below example, I have chosen iDRAC IP, Minimum and Maximum Reading as shown below:

Click on “Dashboard” to add specific filters for Fan speed, choose your type of visualization(I selected “Pie Chart” option) and select the metrics to display it as shown below:


In my next blog post, I will talk about Elastic’s Machine Learning “anomaly score” and how the various scores presented in the dashboards relate to the “unusualness” of individual occurrences within the data set of fan speed and temperature as a whole. Stay tuned !

Introducing OpenUSM – Simplifying Server Management & Insight Log Analytics using Docker containers

Estimated Reading Time: 8 minutes

Its 2018 ! Let Containers Manage Your Datacenter..

Containers are changing the dynamics of modern data center. It is a growing technology that is drawing widespread attention across the Enterprise IT. One of the primary reason of the rise & adoption of containers is that it allows developers to move faster. Compared to VMs which takes minutes to stand up, containers take milliseconds and even microseconds. As organizations prioritize shipping new products and features faster, to keep up in the software-eaten world, developers are favoring technology that allows them to scale applications and deploy resources much faster than what traditional VMs on public and private clouds can support.

Docker containers bring variety of cost and performance benefits. It brings ability to run multiple applications on the same server or OS without a hypervisor which eliminates the drag of the hypervisor on system resources, which means that your workloads have a lighter footprint – the container footprint is zero, because it’s simply a boundary of permissions and resources within Linux. They fire up and decommission very rapidly compared to virtual machines – a perfect fit for the ephemeral nature of today’s short-lived workloads, which are often tied to real world events.

Docker is an open platform which make application and workloads more portable and distributed in an effective and standardized way. Combining Docker containers, micro services along with software-defined infrastructure makes the datacenter more agile and quick resource reallocation. Hence, this architecture works well to improve the datacenter operations.

Most DellEMC Server Management software offerings, as well as the entire Software Defined Infrastructure, are built upon standard implementation using RESTful architecture called Redfish. Redfish is a next generation system management standard using a data model representation inside a hypermedia RESTful interface. The data model is defined in terms of a standard, machine-readable schema, with the payload of the messages expressed in JSON and the protocol using OData v4. Since it is a hypermedia API, Redfish is capable of representing a variety of implementations by using a consistent interface. It has mechanisms for discovering and managing data center resources, handling events, and managing long-lived tasks. It is easy to implement, easy to consume and offer scalability advantages over old technologies. Redfish is a RESTful interface over HTTPS in JSON format based on ODATA v4 usable by clients, scripts, and browser-based GUIs


What is OpenUSM?



OpenUSM stands for Open Universal Systems Manager. It is basically a multi-tool product like “a Swiss army knife”. It is a suite of open source tools & scripts which purely uses containers & related tools to perform server management tasks, monitoring & insight Log Analytics. It is 100% container based solution which heavily uses Docker for building microservices for system management tasks like BIOS token change, Firmware update. It is completely an out-of-band system management solution purely based on Redfish API Interface. It is a platform agnostic solution (can be run from laptop, server or cloud) and works on any of Linux or Windows platform with Docker Engine running on top of it.

OpenUSM is a project which was created by Ajeet Singh Raina & Avinash Bendigeri, a DellEMC employee 3-months back.


Value Proposition:

OpenUSM follow an easy deployment model. It uses developer’s tools like Docker Compose & CLI to bring up microservices which ensures that system management tasks can be achieved flawlessly. It uses modern tools & technologies and integrates well with near real-time search analytics tools like ELK stack. It enables Sensor Log Analytics & visualization for operations team using the popular Grafana tool. It can scale both vertically & horizontally. As it is completely open source, you are free to build and customize based on your needs and holds a plug-and-play components and functionalities.

Technology Overview of OpenUSM:

OpenUSM is an integrated container solution which brings 3 basic functionalities– system management, monitoring and insight log analytics. System Management stack sits at the middle of the architecture which uses Python, Redfish, Django, Docker & Docker compose to enable system management tasks. On the left hand side of the architecture, we have monitoring stack which uses Prometheus, Grafana, Alert-Manager and Pushgateway to retrieve GPU and Sensor metrics. On the right hand side, it consists of open source version of Elasticsearch, Logstash & Kibana for sensors, lifecycle controller & SEL logs.

Understanding OpenUSM System Management WorkFlow

OpenUSM uses “Container-Per-Server (CPS)” model. For each server management tasks, there are scripts which when executed builds and run Docker containers against each of server platform. It purely uses Redfish API to communicate directly with Dell iDRAC, collects iDRAC/LC logs and pushes it to syslog server. Logstash collects the syslog server and pushes to elasticsearch and finally it gets visualized through Kibana. OpenUSM uses Prometheus Stack for monitoring System components like GPU/CPU monitoring using nvidia-docker & node exporter.


Under this blog post, I am going to demonstrate how OpenUSM simplifies the overall system management tasks with the help of Docker containers & Redfish. For this demonstration, I will leverage Ubuntu 17.10 VM running on my ESXi 6.0 system. This code should work on any of Linux & Windows platform too.

The source code for this project is open to the public and is available under


Cloning the Repository

git clone

Bootstrapping Docker

If you have Docker already installed on your system, you can skip to next step. If not, run the below command to install Docker & Docker Compose on your system.

sh install_docker

Based on your network connectivity, this step will take 1-2 minutes to complete.

Bootstrapping ELK

OpenUSM is 100% containerized solution and hence we will be running ELK inside Docker containers. To keep it simple, we designed a docker-compose file which can get you started in a matter of seconds.

Execute the same bootstrap file to bring up ELK stack as shown below:

sh provision_elk

Just wait for 30-40 seconds to get ELK stack up and running.

Verifying the ELK services

Run the below command to check if ELK services are up and running:

sh logs

Pushing DellEMC iDRAC Logs to ELK Stack

Under this section, I am going to demonstrate how OpenUSM makes it so easy to push logs of DellEMC system management tasks to a centralized  ELK stack and get it visualized via Kibana . Let us pick up a simple “BIOS Token Change” functionality and apply it for multitude of DellEMC servers.

To keep it simple, we designed a script named “” which is placed under the root of OpenUSM GIT repository. Let us first look at various parameters which can be supplied with script –

python --help
usage: [-h] [--verbose] [-i IDRAC] [-n NFS] [-s SHARE]
[-c CONFIG] [-f IPS]
Welcome to Universal Systems Manager Bios Token Change
optional arguments:
-h, --help            show this help message and exit
--verbose             Turn on verbose logging
-i IDRAC, --idrac IDRAC
iDRAC IP of the Host
-n NFS, --nfs NFS     NFS server IP address
-s SHARE, --share SHARE
NFS Share folder
-c CONFIG, --config CONFIG
XML File to be imported
-f IPS, --ips IPS     IP files to be updated

As shown above, the script is targeted both at a single Dell server as well as multitude of DellEMC servers via a plain text file. We are currently looking at Autodiscovery feature to automate this functionality. One need to provide NFS server IP, share name and BIOS token configuration files as argument to execute it successfully. Once this script is invoked, it creates as many number of Docker containers per DellEMC servers, collects iDRAC logs from each of servers and pushes it to syslog server which runs inside Docker container. Logstash collects it from syslog and dumps it into Elasticsearch to get it visualized under Kibana UI. -f ips.txt -s /var/nfsshare -c biosconfig.xml -n <NFS server-IP>


/var/nfsshare => NFS share

ips.txt => list of DellEMC iDRAC IP

biosconfig.xml => XML definining the BIOS tokens entry

By now, you should be able to see iDRAC logs visualized under Kibana UI. We can perform ample amount of customization around Kibana UI to display the logs per server basis.


Insight Log Analytics (LC, SEL & Sensor Logs)

Unpacking OpenUSM secrets further, this marks as an interesting use case and robust capability around insight Log Analytics. DellEMC server generates varieties of logs like system event logs (SEL), RAID controller logs & Lifecycle controller (LC) logs. When a system event occurs on a managed system, it is recorded in the System Event Log (SEL). The SEL page displays a system health indicator, a time stamp, and a description for each event logged. The same SEL entry is also available in the Lifecycle Controller (LC) log.

Considering a certain use cases where datacenter administrator want to collect LC logs for the last 1 year, it definitely requires a robust and modern tool to collect such huge data and perform analysis on top of the specific software.

We recently designed a script which simplifies such log analytics capability using Docker, Redfish & ELK. You can find the “” script under the root of GITHUB repository:

python -f ips.txt -ei <ip-address of ELK> -eu elastic -ep changeme


This script requires elasticsearch IP address, credentials & list of iDRAC IPs to get all iDRAC LC logs pushed to ELK stack. Whenever you execute this script, it will build a Docker image called “openusm-analytics” first and run this container which automatically pushes all LC, SEL and Sensors logs to ELK stack.

Below is the Kibana UI visualizing Pie-Chart for Dell Lifecycle Controller logs collected for the last 1 year timeframe.


Insight Log Metrics (Sensor Logs) using Grafana


Did you find OpenUSM interesting? OpenUSM is just 3 months old project and we are looking out for contributors across the globe. If you think the project really looks cool, come & join us to make it more robust. We welcome your participation..

Test Drive Elastic stack on PWD platform running Docker 17.06 CE Swarm Mode in 5 minutes

Estimated Reading Time: 6 minutes

Let’s talk about Dockerized Elastic Stack…

Elastic Stack is an open source solution that reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time. It is a collection of open source products – Elasticsearch, Logstash, Kibana & recently added  fourth product, called Beats. Elastic Stack can be deployed on premises or made available as Software as a Service.

Brief about Elastic Stack Components:


Elasticsearch is a RESTful, distributed, highly scalable, JSON-based search and analytics engine built on top of Apache Lucene and released under Apache license. It is Java-based and designed for horizontal scalability, maximum reliability, and easy management. It is basically an open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.



Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Elasticsearch). Logstash is a dynamic data collection pipeline with an extensible plugin ecosystem and strong Elasticsearch synergy. The product was originally optimized for log data but has expanded the scope to take data from all sources.Data is often scattered or siloed across many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Easily ingest from your logs, metrics, web applications, data stores, and various AWS services, all in continuous, streaming fashion.As data travels from source to store, Logstash filters parse each event, identify named fields to build structure, and transform them to converge on a common format for easier, accelerated analysis and business value.

Logstash dynamically transforms and prepare your data regardless of format or complexity:

  • Derive structure from unstructured data with grok
  • Decipher geo coordinates from IP addresses
  • Anonymize PII data, exclude sensitive fields completely
  • Ease overall processing independent of the data source, format, or schema.

Logstash has a pluggable framework featuring over 200 plugins. Mix, match, and orchestrate different inputs, filters, and outputs to work in pipeline harmony.


Lastly, Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack. It gives you the freedom to select the way you give shape to your data. And you don’t always have to know what you’re looking for. With its interactive visualizations, start with one question and see where it leads you.Kibana developer tools offer powerful ways to help developers interact with the Elastic Stack. With Console, you can bypass using curl from the terminal and tinker with your Elasticsearch data directly. The Search Profiler lets you easily see where time is spent during search requests. And authoring complex grok patterns in your Logstash configuration becomes a breeze with the Grok Debugger.

In next 5 minutes, we are going to test drive ELK stack on PWD playground.

Let’s get started –

Open up


Click on icon next to Instances to open up ready-made templates for Docker Swarm Mode:


Choose the first template (as highlighted in the above figure) to select 3 Managers and 2 Workers. It will bring up Docker 17.06 Swarm Mode cluster in just 10 seconds.

Run the below command to show up the cluster nodes:

docker node ls

Run the necessary command on node which will run elasticsearch:

sysctl -w vm.max_map_count=262144
echo ‘vm.max_map_count=262144’ >> /etc/sysctl.conf

Clone the GitHub repository:

git clone
cd docker101/play-with-docker/visualizer

Run the below command to bring up visualiser tool as shown below:

Soon you will notice port 8080 displayed on the top of the page which when clicked will open up visualiser tool.

It’s time to clone ELK stack and execute the below command to bring up ELK stack across Docker 17.06 Swarm Mode cluster:

git clone
cd swarm-elk
docker stack deploy -c docker-compose.yml myself


[Credits to Andrew Hromis for building this docker-compose file. I leveraged his project repository to bring up the ELK stack in the first try]

You will soon see the below list of containers appearing on the nodes:

Run the below command to see the list of services running across the cluster:

docker service ls

Click on port 5601 displayed on the top of the PWD page:

Please Note:  Kibana need data in Elasticsearch to work with. The .kibana index holds Kibana related data, and if they is the only index you have there is no data available that Kibana can visualise.Before you can use Kibana you will therefore need to index some data into Elasticsearch. This can be done e.g. using Logstash or directly through the REST interface using curl.

Soon you will see the below Kibana page:


Enabling High Availability for Elastic Stack through scaling

Let us scale out more number of replicas for elasticsearch:

Pushing data into Logstash:

Example #1:

Let us push NGINX web server logs into logstash and see if Kibana is able to detect it:

docker run -d --name nginx-with-syslog --log-driver=syslog --log-opt syslog-address=udp:// -p 80:80 nginx:alpine

Now if you open up Kibana UI, you should be able to see logs being displayed for Nginx:

Example #2:

We can also push logs to logstash using the below command:

docker run --rm -it --log-driver=gelf --log-opt gelf-address=udp:// alpine ping

Open up Kibana and now you will see the below GREEN status:

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.


Test Drive 5 Cool Docker Application Stacks on play-with-docker (PWD) platform

Estimated Reading Time: 7 minutes

Do you want to learn Docker FOR FREE OF COST? Yes, you read it correct. Thanks to a playground called “play-with-docker” – PWD in short. PWD is a website which allows you to create 5 instances to play around with Docker & Docker Swarm Mode for 4 hours – all for $0 cost. It is a perfect tool for demos, Meetups, beginners & advanced level training. I tested it on number of web browsers like Chrome, Safari, Firefox and IE and it just works flawlessly. You don’t need to install Docker as it comes by default. It’s ready-to-use platform.




Currently, PWD is hosted on AWS instance type 2x r3.4xlarge  having 16 cores and 120GB of RAM. It comes with the latest Docker 17.05 release, docker-compose 1.11.1 & docker-machine 0.9.0-rc1. You can setup your own PWD environment in your lab using this repository. Credits to Docker Captain – Marcos Nils & Jonathan Leibuisky for building this amazing tool for Docker Community. But one of the most interesting fact about PWD is its based on DIND (Docker-in-a-Docker) concept. When you are playing around with PWD instances & building application stack, you are actually inside Docker container itself. Interesting, isn’t it? PWD gives you an amazing experience of having a free Alpine Linux  3.5 Virtual Machine in the cloud where you can build and run Docker containers and even create Multi-Node Swarm Mode Cluster.

Said that, PWD is NOT just a platform for beginners. Today, it has matured enough to run sophisticated application stack on top of it. Within seconds of time, you can setup Swarm Mode cluster running application stack.Please remember that PWD is just for trying out new stuffs with Docker and its application and NOT to be used for production environment. The instances will vanish after 4:00 hours automatically.





Under this blog post, I am going to showcase 3 out of 5 cool Dockerized application stack which you can demonstrate to the advance level audience all running on PWD playground (listed below):

Docker UI Management & Local Registry Service

  • Bringing Portainer & Portus together for PWD under Swarm Mode

Building Monitoring Stack with Prometheus & Grafana:

  • Prometheus, Grafana & cAdvisor Stack on Swarm Mode

Demonstrating Voting Application under Swarm Mode

  • Voting App on Swarm Mode

Highly Available Web Application

  • LAMP Stack under Swarm Mode

Demonstrating RSVP

To make it quite simple, I have collected the list of sample application under

You can use git clone to pull the repository on the manager node:

$git clone

First, we need to setup Swarm Mode cluster on PWD. As Docker 17.05 already comes installed by default, we are all set to initialize the swarm mode cluster. Click on “New Instance” button on the left hand side of PWD and this will open up an instance on the right side as shown.Run the below command on the 1st instance as shown below:

$docker swarm init –advertise-addr <manager-ip>:2377 –listen-addr <manager-ip>:2377

This command will initialize the Swarm and suggest a command to join the worker node as shown:

$docker swarm join  –token <token-id> <manager-ip>:2377

Once you run the command on all the 4 worker nodes. You can go back to manager node and check the list of Swarm nodes:




1. Docker UI Management & Local Registry Service:

Setting up Portainer for Swarm Mode Cluster

It’s time to get started with our first application – Portainer. Portainer is an open-source lightweight management UI which allows you to easily manage your Docker host or Swarm cluster. Run the below command to setup Portainer for Swarm Mode cluster:



Ensure that portainer is up and running:



The moment Portainer service gets started, PWD displays 9000 port which Portainer works on as shown. You can click on this port number to directly open the Management UI.




This opens up a fancy UI which displays lots of management features related to image, container, swarm, network, volumes etc.




Portainer clearly displays the swarm cluster as shown:



Setting up Portus for Local Docker Registry:

Portus is an authorization service for your Docker registry. It provide a useful and powerful UI on top of your registry. To test driver Portus, one need to clone the repository:

$git clone && cd Portus

Run the below script with your manager IP to get Portus up and running:










Once this looks good, you should be able to browse its UI as shown:









I faced the issue related to Webpack::Rails::Manifest::ManifestLoadError which I got it fixed within few minutes. You now have control over Docker registry using this fancy UI. You can create Users, Organization for your development team and push/pull Docker images privately.

Hence, you can now open up Portainer and provide Portus as a local Docker registry instead of standard Docker registry. This makes Portainer & Portus work together flawlessly.

2. Building Monitoring Stack with Prometheus & Grafana

Prometheus is an open-source systems monitoring and alerting toolkit. Most Prometheus components are written in Go while some  written in Java, Python, and Ruby. It is designed for capturing high dimensional data. It is designed to be used for monitoring. On the other hand, ELK is a general-purpose NOSQL stack that is also very popular for monitoring and Logging.

To test-drive Prometheus & ELK, you can change the directory to /play-with-docker/docker-prometheus-swarm directory and run the stack deploy command as shown below:

$git clone

#cd docker101/play-with-docker/docker-prometheus-swarm/

$docker stack deploy -c docker-compose.yml myprom1

[ A Special Credits to Basi for building repository & tremendous effort for enabling this solution]




That’s it. Your Prometheus, Grafana & cadvisor for ELK stack is ready to be used.





Demonstrating Voting Application under Swarm Mode

Voting app is a perfect piece of example where it showcase dependencies among the services, and a potential division of services between the manager and worker nodes in a swarm.  You can learn more about voting app and how it actually works under this link. To quickly demonstrate it, let us pick up the right directory under the pulled repository:

$cd play-with-docker/example-voting-app

Run the below command:

$docker stack deploy -c docker-stack.yml myvotingapp









In the next series of this blog post, I am going to cover on other 2 application stacks – CloudYuga RSVP & WordPress Web Application.

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution, join me at Docker Community Slack Channel.