Running Prometheus Docker container for monitoring Microservices on Raspberry Pi

Estimated Reading Time: 5 minutes

In the previous post, we talked about running Nagios container on Raspberry Pi for the first time.Nagios is a free and open source monitoring and alerting service which basically collects the statistics of your server using agents like nrpe, check_mk or using SNMP and sends the alert if the metric value is above the predefined threshold. It is a great tool for monitoring the nodes or monolithic application but with the advent of the ‘microservices’ architecture and Docker, monitoring the small piece of services running across the cluster becomes very important.With that also grows the need to monitor these services around the clock, to maintain the healthy functioning of the application.

pi33

One of the most talked-about monitoring tools are Prometheus & Grafana. Prometheus is an open-source systems monitoring and alerting toolkit written in Go. It is a next-generation monitoring system and well appreciated by Docker Core Engineering Team. It’s a service especially well designed for containers and provides perspective about the data intensiveness of this new age and how even Internet scale companies have had to adapt.

On the other hand, Grafana can be rightly called the face of Prometheus. It is also an open-source, general purpose dashboard and graph composer, which runs as a web application. It supports graphite, influxdb or opentsdb as backends. While Prometheus exposes some of its internals like settings and the stats it gathers via basic web front-ends, it delegates the heavy lifting of proper graphical displays and dashboards to Grafana. For anyone who want to collect metrics from the development environment to be graphed and measured, Prometheus as time series store and Grafana as a visualization tool is the perfect solution.

What’s unique about Prometheus?

Prometheus is a self-hosted set of tools which collectively provide metrics storage, aggregation, visualization and alerting. Usually most of the available monitoring tools are push based, i.e. agents on the monitored servers talk to a central server (or set of servers) and send out their metrics. Prometheus is little different on this aspect – It is a pull based server which expects monitored servers to provide a web interface from which it can scrape data. This is a unique characteristic of Prometheus.

The main features of Prometheus can be summed below:

  • Decentralized architecture
  • Scalable data collection
  • Powerful Query language (leverages the data model for meaningful alerting (including easy silencing) and graphing (for dashboards and for ad-hoc exploration).
  • A multi-dimensional data model.( data can be sliced and diced at will, along dimensions like instance, service, endpoint, and method)
  • No reliance on distributed storage; single server nodes are autonomous.
  • Time series collection happens via a pull model over HTTP.
  • Pushing time series is supported via an intermediary gateway.
  • Targets are discovered via service discovery or static configuration.
  • Multiple modes of graphing and dashboard support.

You can deep dive more into its architecture at https://prometheus.io/docs/introduction/overview/#architecture.

I was in need of lightweight monitoring tool like Prometheus and Fancy UI like Grafana for my Raspberry Pi which I can spin up as Docker container in few seconds and then grow /spread it across the cluster primarily to test it for Docker 1.12 Swarm Mode. Unfortunately, I didn’t find any Docker image for Prometheus. I decided to build one for myself to explore it on Raspberry Pi 3 box.

After spending considerable amount of time on Pi box, I was able to build the first Prometheus Docker image and pushed it to Dockerhub. In case you are interested, check out my github page : https://github.com/ajeetraina/prometheus-armv7 to build and run your own Prometheus container.

[Thanks to Alex Ellis, #Docker Captain for reviewing this container and cleaning the Dockerfile.]

 

pi3

Running Prometheus & Grafana Docker Image on Raspberry Pi

Follow the below command to spin up Prometheus container on the fly:

pi11

Luckily, there was already a Grafana ARM image available in Dockerhub which I spent no minute to spin up easily as shown below:

 

pi22

You can also use docker-compose to bring up both the container in a single shot as shown:

pi@raspberrypi:~/testingPi $ cat docker-compose.yml

prometheus:
image: ajeetraina/prometheus-armv7
volumes:
– ./prometheus.yml:/etc/prometheus/prometheus.yml
command:
– ‘-config.file=/etc/prometheus/prometheus.yml’
ports:
– ‘9090:9090’

grafana:
image: fg2it/grafana-armhf:v3.1.1
environment:
– GF_SECURITY_ADMIN_PASSWORD=collabnix

ports:
– “3000:3000”

Ensure that you have either sample or your own customized prometheus.yml file under / directory to run this container.

Just one command and you are ready to spin up both the containers as shown below:

pi432pi90

That’s it. Browse to http://<ipaddress>:9090 to check out the Prometheus UI.

pi323

Also, Browse to http://<ipaddress>:3000 to open up fancy Grafana UI as shown below. Supply admin/collabnix as credentials to enter into Grafana UI.

graf2

Importing Prometheus Data into Grafana

Once you login into Grafana UI, click on Data Sources followed by “Add Data Source” button. You will find the below screen:

pi44

Next, Let’s import the prometheus into Grafana dashboard:

import

pi2345

I firmly believe that the newly built Prometheus container is sure to help developers to monitor their microservice application on top of Raspberry Pi box.In the next post, I will talk about the other available data sources and how it integrates with Prometheus and Grafana as a monitoring stack in detail.

Turn Your Raspberry Pi into Out-of-band Monitoring Device using Docker

Estimated Reading Time: 5 minutes

Why should I deploy Docker to IoT devices? What are the use cases of running containers inside the IoT device like Raspberry Pi? Is Docker on IoT devices just for hobbyists? – To answer all these burning queries, I just thought of sharing one of the most interesting use case inside the datacentre lab which will surely attract infinite number of possibilities and implementation with Docker containers.

pi90

Imagine a small credit card size server sitting just below your ToR (Top of the Rack) switch, consuming very low power ( similar to your mobile device), not bound by wired Ethernet( Wi-Fi Direct, ZigBee) for mobile access and control, connected to ToR switch to discover the overall hardware, software managed and deployed through air, no monitor, keyboard and mouse intervention required – just boots up, discovers all your servers(rack/blades), storage(FC/iSCSI), Converged Infrastructure and network devices automatically and ready to monitor all of these datacentre equipment on the mobile & Web UI . Sounds interesting?? What if I say I have a single DOCKER container running inside this credit card size device handling all these functionality? Sounds crazy? Let me share the recent experience I had prototyping and designing this solution.

Why should I deploy Docker to IoT devices?

Docker is an open platform to build, ship and run your application anywhere: in your laptop, workstation, server, in a private or public cloud, within a local VM, or on physical hardware including IoT devices. Docker is very lightweight and this is what attracts developers who can apply to rapidly develop, test, deploy, and update IoT applications at scale. With very less disk, low memory resources and almost zero run-time overhead, deploying Docker on IoT devices like Raspberry Pi is considered a feasible solution.

Example: Let’s consider NagiosPi – a lightweight, feature rich & fully configured Server Monitoring Distro for the Raspberry Pi. To setup NagiosPi, one need to download the image and flash to SD card. Then, you need to boot through this SD card to use this tool.  Why not just pull Nagios Docker image from Dockerhub and run it? I am sure you will save a lot rather than flashing your entire SD card and loosing the underlying OS.In short,  if you run containers directly on your device, you can avoid having to constantly flash your device or overwrite the entire firmware. Instead you can build new images, pull them and run them within the host OS on the device.

I built the first Nagios container for ARM device few days back and pushed it to Dockerhub. You can refer https://github.com/ajeetraina/nagios-centos-arm

pi1

pi4

Just run the single Docker run command and Nagios is ready which you can easily see on the WebUI through http://<IP>/nagios.

pi3

Building Out-of-Band Monitoring Tool using Docker

Nagios is a free and open source monitoring and alerting service for servers, switches,applications and services. It basically collects the statistics of your server using agents like nrpe, check_mk or using SNMP and sends the alert if the metric value is above the predefined threshold. One of the implementation  of such tool is most of the time it’s configured as in-band connectivity which means software that is installed on the remote system being managed and only works after the operating system has been booted. If you want to monitor hundreds and thousands of such servers, you need to ensure that the agent plugin should be running on each nodes.

What if I want to monitor the physical health of servers, storage and network switches? Say, you are running a huge data center for High Performance Cluster Computing (HPCC) kind of workloads and you want to make sure that the health of your servers is all fine. In case, any of the servers goes bad in terms of memory DIMMs, the in-band monitoring system needs lots of complex configuration to be carried out to meet the right implementation. Out-of-band(OOB) monitoring system finally comes for a rescue. OOB uses a dedicated channel for managing the hardware devices like network devices. This allows the network operator to establish trust boundaries in accessing the management function to apply it to network resources.

How about using the tiny Raspberry Pi which can do OOB monitoring for the entire infrastructure?  Why do I need to setup a big system if this small credit card size can do what I really need? I spent considerable amount of time to come up with a viable solution and finally initiated building Out of Band monitoring tool . This was purely based on one of my previous work https://hub.docker.com/r/ajeetraina/dell-oob-monitoring/  which I built for X86 infrastructure.

https://www.youtube.com/watch?v=lpaeTepVETg&feature=youtu.be

pi6

How does it work?

All you need is list out all the IP addresses in one simple text file and run the below command:

Step-1: List out the IP addresses of Chassis, Storage devices, iDRAC IPs etc. in a plain text file using notepad or vim editor. Example: I saved the file called “ips” under /IP folder in the Docker Host machine as:

File: /IP/ips

100.94.214.181
100.94.214.178
100.94.214.190
100.94.214.210
……
……

Step:2: Execute the below command to start the container:

$sudo docker run -dit  --net=host -v /IP:/IP ajeetraina/dell-oob-monitoring:v1.0

Step3: Execute the following command to add the list of Dell servers, storage and chassis in a single shot:

$sudo docker exec -it <container-id> sh discover

Open up your browser http://<dockerhost>/nagios to see all the listed servers, storage and chassis added automatically to Nagios console and ready to be monitored.

Sounds Interesting?? Though I have tested this container internally and it just works out of box, I am still working on making it more lightweight. Stay tuned !

In the future post, I am going to talk more use cases of Docker on Raspberry Pi box. Keep Reading !

Building the first CentOS 7.2 ARM docker image on Raspberry Pi 3

Estimated Reading Time: 5 minutes

In the previous post, I showed how to get Docker 1.12.1 up and running on Raspberry Pi3 in just 5 minutes. Under 1.12.1 release, Docker Inc. provided a FIRST ARM debian package which can be installed using a single curl command. It works great and there has been numerous Ubuntu, Alpine and Debian based ARM Docker images been built and pushed out to Dockerhub. If you are CentOS/Red Hat user, you might be interested in building CentOS based ARM images on top of Raspberry Pi. As of today, there is NO CentOS based Docker image available for ARM architecture under Dockerhub. I had few of X86 based legacy application which only runs on CentOS based system and hence decided to build one of my own for ARM. It is important to note that X86 based Docker image is not going to work on Raspberry Pi box.

This step-by-step guide will help you to build the CentOS 7.2 ARM Docker Image.

raspbi

Step-1: Download the CentOS 7 specific ARM image from the official CentOS website.

root@raspberry5:/# wget http://mirror.centos.org/altarch/7/isos/armhfp/CentOS-Userland-7-armv7hl-Minimal-1603-RaspberryPi3.img.xz

 

ic1

Luckily, I found that CentOS community has ARM images availability for all popular vendors such as  BananaPi, CubieTruck, Pi2 and Pi3. I picked up the right one which will work for my Pi 3 box.

Step-2:  Format your microSD card in case you have old Raspbian OS using SD Formatter(Refer here)

Step-3: Use Win32 disk imager to burn it directly into microSD card.

Step-4: Insert microSD card into microSD slot of your Pi 3 box and power it on.

The default username & password is root/centos. This is a minimal CentOS 7 ARM image and hence don’t expect GUI out of this image. By default, WiFi is not enabled. Refer README file under the root directory to get WiFi working. I had a small cable to connect to my router and the network just worked out of the box. I installed vncserver using YUM and could connect to the Pi 3 box without any issue.

 

ic4

Step-5:  Ensure that you run “yum update” to get the latest ARM packages for CentOS 7

Step-6: By default, 2GB is the size allocated for / root partition. Follow the below step for expanding it to utilize the full 12GB of the space.

$sudo git clone https://github.com/ctyler/rootfs-resize

$sudo touch /.rootfs-repartition

$sudo systemctl reboot

Step-7:Now configure the YUM repo to pick up the required packages for Docker. I did a little tweaking here by including RPMs available from foreign unofficial repository.

ic3

Ensure that you run ‘yum update’ command again to get the latest packages for Docker packages.

Step-8: Run the below command to get the required number of packages to be installed:

$sudo yum install lxc bridge-utils device-mapper device-mapper-libs \

libsqlite3x docker-registry docker-storage-setup docker-io

Ensure that you run the below command to start the docker service:

$sudo systemctl start docker.service

Step-9: Verify if Docker is installed or not on Raspberry Pi 3 through ‘docker version’ utility:

ic6

You can get the detailed information about Docker  through ‘docker info’ command:

ic8

Yipee ! You got Docker installed on your CentOS base image on Raspberry Pi 3 box for the first time.

Let’s quickly try to pull few of the container from Dockerhub.

ic10

One can quickly start a container as shown below:

ic11

Let’s build up the first Docker ARM image for Dockerhub so that everyone can use it to build application on RPM based ARM container.

Building a container is a simplified process. All you need is a script which you can pull from this github link.Just run the below command to get CentOS 7.2 up and tested automatically:

[root@centos-rpi3 contrib]# sh mkimage-yum.sh -y /etc/yum.repos.d/CentOS-Base.repo centos7

As you see the above command, I have used CentOS-Base.repo as a source repository to create Docker image successfully. Hence, this image is just 210 MB in size. I am planning to shrink it to lower size soon so that it comes up better than before.

Feel free to pull and try out the first CentOS 7.2 container in your Pi box:

$sudo docker run -i -t –rm centos7:7.2 /bin/bash -c ‘echo success’
success

ic91

This is just awesome ! Now I have a platform to build monitoring tool on top of very tiny Raspberry Pi and this is going to be real fun. Will keep you all posted on the latest findings. Till then, keep Reading !