Test Drive Elastic stack on PWD platform running Docker 17.06 CE Swarm Mode in 5 minutes

Let’s talk about Dockerized Elastic Stack…

Elastic Stack is an open source solution that reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time. It is a collection of open source products – Elasticsearch, Logstash, Kibana & recently added  fourth product, called Beats. Elastic Stack can be deployed on premises or made available as Software as a Service.

Brief about Elastic Stack Components:

Elasticsearch:

Elasticsearch is a RESTful, distributed, highly scalable, JSON-based search and analytics engine built on top of Apache Lucene and released under Apache license. It is Java-based and designed for horizontal scalability, maximum reliability, and easy management. It is basically an open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

                                                                                                                                              ~Source: https://www.elastic.co

Logstash:

Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Elasticsearch). Logstash is a dynamic data collection pipeline with an extensible plugin ecosystem and strong Elasticsearch synergy. The product was originally optimized for log data but has expanded the scope to take data from all sources.Data is often scattered or siloed across many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Easily ingest from your logs, metrics, web applications, data stores, and various AWS services, all in continuous, streaming fashion.As data travels from source to store, Logstash filters parse each event, identify named fields to build structure, and transform them to converge on a common format for easier, accelerated analysis and business value.

Logstash dynamically transforms and prepare your data regardless of format or complexity:

  • Derive structure from unstructured data with grok
  • Decipher geo coordinates from IP addresses
  • Anonymize PII data, exclude sensitive fields completely
  • Ease overall processing independent of the data source, format, or schema.

Logstash has a pluggable framework featuring over 200 plugins. Mix, match, and orchestrate different inputs, filters, and outputs to work in pipeline harmony.

Kibana:

Lastly, Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack. It gives you the freedom to select the way you give shape to your data. And you don’t always have to know what you’re looking for. With its interactive visualizations, start with one question and see where it leads you.Kibana developer tools offer powerful ways to help developers interact with the Elastic Stack. With Console, you can bypass using curl from the terminal and tinker with your Elasticsearch data directly. The Search Profiler lets you easily see where time is spent during search requests. And authoring complex grok patterns in your Logstash configuration becomes a breeze with the Grok Debugger.

In next 5 minutes, we are going to test drive ELK stack on PWD playground.

Let’s get started –

Open up https://play-with-docker.com

 

Click on icon next to Instances to open up ready-made templates for Docker Swarm Mode:

 

Choose the first template (as highlighted in the above figure) to select 3 Managers and 2 Workers. It will bring up Docker 17.06 Swarm Mode cluster in just 10 seconds.

Run the below command to show up the cluster nodes:

docker node ls

Run the necessary command on node which will run elasticsearch:

sysctl -w vm.max_map_count=262144
echo ‘vm.max_map_count=262144’ >> /etc/sysctl.conf

Clone the GitHub repository:

git clone https://github.com/ajeetraina/docker101
cd docker101/play-with-docker/visualizer

Run the below command to bring up visualiser tool as shown below:

Soon you will notice port 8080 displayed on the top of the page which when clicked will open up visualiser tool.

It’s time to clone ELK stack and execute the below command to bring up ELK stack across Docker 17.06 Swarm Mode cluster:

git clone https://github.com/ajeetraina/swarm-elk
cd swarm-elk
docker stack deploy -c docker-compose.yml myself

 

[Credits to Andrew Hromis for building this docker-compose file. I leveraged his project repository to bring up the ELK stack in the first try]

You will soon see the below list of containers appearing on the nodes:

Run the below command to see the list of services running across the cluster:

docker service ls

Click on port 5601 displayed on the top of the PWD page:

Please Note:  Kibana need data in Elasticsearch to work with. The .kibana index holds Kibana related data, and if they is the only index you have there is no data available that Kibana can visualise.Before you can use Kibana you will therefore need to index some data into Elasticsearch. This can be done e.g. using Logstash or directly through the REST interface using curl.

Soon you will see the below Kibana page:

 

Enabling High Availability for Elastic Stack through scaling

Let us scale out more number of replicas for elasticsearch:

Pushing data into Logstash:

Example #1:

Let us push NGINX web server logs into logstash and see if Kibana is able to detect it:

docker run -d --name nginx-with-syslog --log-driver=syslog --log-opt syslog-address=udp://10.0.173.7:12201 -p 80:80 nginx:alpine

Now if you open up Kibana UI, you should be able to see logs being displayed for Nginx:

Example #2:

We can also push logs to logstash using the below command:

docker run --rm -it --log-driver=gelf --log-opt gelf-address=udp://10.0.173.7:12201 alpine ping 8.8.8.8

Open up Kibana and now you will see the below GREEN status:

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

 

0
0

Test Drive 5 Cool Docker Application Stacks on play-with-docker (PWD) platform

Do you want to learn Docker FOR FREE OF COST? Yes, you read it correct. Thanks to a playground called “play-with-docker” – PWD in short. PWD is a website which allows you to create 5 instances to play around with Docker & Docker Swarm Mode for 4 hours – all for $0 cost. It is a perfect tool for demos, Meetups, beginners & advanced level training. I tested it on number of web browsers like Chrome, Safari, Firefox and IE and it just works flawlessly. You don’t need to install Docker as it comes by default. It’s ready-to-use platform.

learn

 

 

Currently, PWD is hosted on AWS instance type 2x r3.4xlarge  having 16 cores and 120GB of RAM. It comes with the latest Docker 17.05 release, docker-compose 1.11.1 & docker-machine 0.9.0-rc1. You can setup your own PWD environment in your lab using this repository. Credits to Docker Captain – Marcos Nils & Jonathan Leibuisky for building this amazing tool for Docker Community. But one of the most interesting fact about PWD is its based on DIND (Docker-in-a-Docker) concept. When you are playing around with PWD instances & building application stack, you are actually inside Docker container itself. Interesting, isn’t it? PWD gives you an amazing experience of having a free Alpine Linux  3.5 Virtual Machine in the cloud where you can build and run Docker containers and even create Multi-Node Swarm Mode Cluster.

Said that, PWD is NOT just a platform for beginners. Today, it has matured enough to run sophisticated application stack on top of it. Within seconds of time, you can setup Swarm Mode cluster running application stack.Please remember that PWD is just for trying out new stuffs with Docker and its application and NOT to be used for production environment. The instances will vanish after 4:00 hours automatically.

PWDD

 

 

 

Under this blog post, I am going to showcase 3 out of 5 cool Dockerized application stack which you can demonstrate to the advance level audience all running on PWD playground (listed below):

Docker UI Management & Local Registry Service

  • Bringing Portainer & Portus together for PWD under Swarm Mode

Building Monitoring Stack with Prometheus & Grafana:

  • Prometheus, Grafana & cAdvisor Stack on Swarm Mode

Demonstrating Voting Application under Swarm Mode

  • Voting App on Swarm Mode

Highly Available Web Application

  • LAMP Stack under Swarm Mode

Demonstrating RSVP

To make it quite simple, I have collected the list of sample application under https://github.com/ajeetraina/docker101

You can use git clone to pull the repository on the manager node:

$git clone https://github.com/ajeetraina/docker101

First, we need to setup Swarm Mode cluster on PWD. As Docker 17.05 already comes installed by default, we are all set to initialize the swarm mode cluster. Click on “New Instance” button on the left hand side of PWD and this will open up an instance on the right side as shown.Run the below command on the 1st instance as shown below:

$docker swarm init –advertise-addr <manager-ip>:2377 –listen-addr <manager-ip>:2377

This command will initialize the Swarm and suggest a command to join the worker node as shown:

$docker swarm join  –token <token-id> <manager-ip>:2377

Once you run the command on all the 4 worker nodes. You can go back to manager node and check the list of Swarm nodes:

33

 

 

1. Docker UI Management & Local Registry Service:

Setting up Portainer for Swarm Mode Cluster

It’s time to get started with our first application – Portainer. Portainer is an open-source lightweight management UI which allows you to easily manage your Docker host or Swarm cluster. Run the below command to setup Portainer for Swarm Mode cluster:

4

 

Ensure that portainer is up and running:

5

 

The moment Portainer service gets started, PWD displays 9000 port which Portainer works on as shown. You can click on this port number to directly open the Management UI.

6

 

 

This opens up a fancy UI which displays lots of management features related to image, container, swarm, network, volumes etc.

7

 

 

Portainer clearly displays the swarm cluster as shown:

8

 

Setting up Portus for Local Docker Registry:

Portus is an authorization service for your Docker registry. It provide a useful and powerful UI on top of your registry. To test driver Portus, one need to clone the repository:

$git clone https://github.com/SUSE/Portus && cd Portus

Run the below script with your manager IP to get Portus up and running:

9

 

 

10

 

 

15

 

 

Once this looks good, you should be able to browse its UI as shown:

12

 

 

13

 

 

 

 

I faced the issue related to Webpack::Rails::Manifest::ManifestLoadError which I got it fixed within few minutes. You now have control over Docker registry using this fancy UI. You can create Users, Organization for your development team and push/pull Docker images privately.

Hence, you can now open up Portainer and provide Portus as a local Docker registry instead of standard Docker registry. This makes Portainer & Portus work together flawlessly.

2. Building Monitoring Stack with Prometheus & Grafana

Prometheus is an open-source systems monitoring and alerting toolkit. Most Prometheus components are written in Go while some  written in Java, Python, and Ruby. It is designed for capturing high dimensional data. It is designed to be used for monitoring. On the other hand, ELK is a general-purpose NOSQL stack that is also very popular for monitoring and Logging.

To test-drive Prometheus & ELK, you can change the directory to /play-with-docker/docker-prometheus-swarm directory and run the stack deploy command as shown below:

$git clone https://github.com/ajeetraina/docker101

#cd docker101/play-with-docker/docker-prometheus-swarm/

$docker stack deploy -c docker-compose.yml myprom1

[ A Special Credits to Basi for building https://github.com/bvis/docker-prometheus-swarm repository & tremendous effort for enabling this solution]

16

 

 

That’s it. Your Prometheus, Grafana & cadvisor for ELK stack is ready to be used.

22

18

 

 

Demonstrating Voting Application under Swarm Mode

Voting app is a perfect piece of example where it showcase dependencies among the services, and a potential division of services between the manager and worker nodes in a swarm.  You can learn more about voting app and how it actually works under this link. To quickly demonstrate it, let us pick up the right directory under the pulled repository:

$cd play-with-docker/example-voting-app

Run the below command:

$docker stack deploy -c docker-stack.yml myvotingapp

vote1

 

 

vote2

 

 

vote4

 

In the next series of this blog post, I am going to cover on other 2 application stacks – CloudYuga RSVP & WordPress Web Application.

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution, join me at Docker Community Slack Channel.

0
0