Test-Drive Continuous Integration Pipeline using Docker, Jenkins & GitHub under $0

Are you new to CI/CD?

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. CI doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.

CI/CD merges development with testing, allowing developers to build code collaboratively, submit it the master branch, and checked for issues. This allows developers to not only build their code, but also test their code in any environment type and as often as possible to catch bugs early in the applications development lifecycle. Since Docker can integrate with tools like Jenkins and GitHub, developers can submit code in GitHub, test the code and automatically trigger a build using Jenkins, and once the image is complete, images can be added to Docker registries. This streamlines the process, saves time on build and set up processes, all while allowing developers to run tests in parallel and automate them so that they can continue to work on other projects while tests are being run.

A Typical CI workflow look like:

1. Developer pushes a commit to GitHub
2. GitHub uses a webhook to notify Jenkins of the update
3. Jenkins pulls the GitHub repository, including the Dockerfile describing the image, as well as the application and test code.
4. Jenkins builds a Docker image on the Jenkins slave node
5. Jenkins instantiates the Docker container on the slave node, and executes the appropriate tests
6. If the tests are successful the image is then pushed up to Dockerhub or Docker Trusted Registry.

Under these series of blog post, I will help you get started with Continuous Integration with Jenkins, Docker & GitHub under $0. I will be using Play with Docker platform which comes for free for the general public.

Let’s get started..

Go to your browser and type –  https://labs.play-with-docker.com/


Open up a “New Instance” on the left hand side of the screen.



Cloning the Repository

Let us start with a simple HTML webpage application. I have a sample GITHUB repository already created for you which contains Dockerfile and Jenkinsfile under the root of the repository as shown below:



$ git clone https://github.com/ajeetraina/webpage


What is Jenkinsfile & Jenkins Pipeline?

A Jenkinsfile is a text file that contains the definition of a Jenkins Pipeline and is checked into source control.

Jenkins Pipeline (or simply “Pipeline”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. It provides an extensible set of tools for modeling simple-to-complex delivery pipelines “as code”. The definition of a Jenkins Pipeline is typically written into a text file (called a Jenkinsfile) which in turn is checked into a project’s source control repository.
I have created a sample Jenkinsfile which clones the repository, builds the Docker Image, test it and then push it to Dockerhub automatically using Jenkins.
node {
    def app

    stage('Clone repository') {
        /* Let's make sure we have the repository cloned to our workspace */

        checkout scm

    stage('Build image') {
        /* This builds the actual image; synonymous to
         * docker build on the command line */

        app = docker.build("ajeetraina/webpage")

    stage('Test image') {
        /* Ideally, we would run a test framework against our image.
         * Just an example */

        app.inside {
            sh 'echo "Tests passed"'

    stage('Push image') {
        /* Finally, we'll push the image with two tags:
         * First, the incremental build number from Jenkins
         * Second, the 'latest' tag.
         * Pushing multiple tags is cheap, as all the layers are reused. */
        docker.withRegistry('https://registry.hub.docker.com', 'dockerhub') {

In the above Jenkinsfile, there are various stages like cloning the SCM, build, test and pushing the Docker image. You will need to change it to your respective GITHUB repo if you want to try to build your own Docker image.

Let us quickly setup Jenkins on PWD platform and believe me it’s just a matter of a single docker-compose CLI. Before we proceed, ensure that the below permission is granted so as to get Docker plugin to work flawlessly –

$chmod 666 /var/run/docker.sock

It’s time to clone the repository:

$ git clone https://github.com/ajeetraina/jenkins
Cloning into ‘jenkins’…
remote: Counting objects: 18, done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 18 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (18/18), done.
[node1] (local) root@ ~/webpage
$ cd jenkins
[node1] (local) root@ ~/webpage/jenkins
$docker-compose up

Copy the password of the jenkins starting with “5ee571..” as shown in the above screenshot.

Click on 8080 port which appear on the top of the screen and this will redirect you to Jenkins page.

Once you add the Administrator password which we copied earlier, then it will ask to install plugins. Choose “Install Suggested Plugins” on the left hand side as shown below:

Go back to Jenkins dashboard and click “New Item”. Supply any name(i.e demo-webpage) and select “Pipeline” as type of the project.

This will show up a page with Build Triggers as one of the option. Select “Pipeline script from SCM” and choose GIT under SCM. This will allow you to enter your GITHUB URL which in my case is https://github.com/ajeetraina/webpage. This will automatically add Jenkinsfile under Script Path. Click on “Apply” and then Save.

Click on “Build Now” option on the left side of the Jenkins screen. This traverse through the build pipeline i.e cloning the repository, building Docker Image , testing it and pushing it to your Dockerhub account as shown below:





Once you click on “Build Now” you will see that it initiates the build pipeline.


Jenkins Pipeline Stages View:


Click on “Console Output” to show detailed build process as shown below:


A Quick View of Stage View:



Till now, we have a new Docker Image pushed to Dockerhub automatically. Let’s go ahead and verify it by bringing up webpage container in the separate instance window as shown below:



Here you see…a static HTML page appears running on port 80. This was built as part of Jenkins pipeline.

In order to trigger this pipeline every 15 minutes, you can go back to Jenkins page for specific job and then choose “Build Periodically” under Configure page as shown below:


Under my future blog post, I will show you how CI-CD works in terms of Docker Swarm Mode and Kubernetes.

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina.

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.


Test Drive Elastic stack on PWD platform running Docker 17.06 CE Swarm Mode in 5 minutes

Let’s talk about Dockerized Elastic Stack…

Elastic Stack is an open source solution that reliably and securely take data from any source, in any format, and search, analyze, and visualize it in real time. It is a collection of open source products – Elasticsearch, Logstash, Kibana & recently added  fourth product, called Beats. Elastic Stack can be deployed on premises or made available as Software as a Service.

Brief about Elastic Stack Components:


Elasticsearch is a RESTful, distributed, highly scalable, JSON-based search and analytics engine built on top of Apache Lucene and released under Apache license. It is Java-based and designed for horizontal scalability, maximum reliability, and easy management. It is basically an open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

                                                                                                                                              ~Source: https://www.elastic.co


Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.” (Elasticsearch). Logstash is a dynamic data collection pipeline with an extensible plugin ecosystem and strong Elasticsearch synergy. The product was originally optimized for log data but has expanded the scope to take data from all sources.Data is often scattered or siloed across many systems in many formats. Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Easily ingest from your logs, metrics, web applications, data stores, and various AWS services, all in continuous, streaming fashion.As data travels from source to store, Logstash filters parse each event, identify named fields to build structure, and transform them to converge on a common format for easier, accelerated analysis and business value.

Logstash dynamically transforms and prepare your data regardless of format or complexity:

  • Derive structure from unstructured data with grok
  • Decipher geo coordinates from IP addresses
  • Anonymize PII data, exclude sensitive fields completely
  • Ease overall processing independent of the data source, format, or schema.

Logstash has a pluggable framework featuring over 200 plugins. Mix, match, and orchestrate different inputs, filters, and outputs to work in pipeline harmony.


Lastly, Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack. It gives you the freedom to select the way you give shape to your data. And you don’t always have to know what you’re looking for. With its interactive visualizations, start with one question and see where it leads you.Kibana developer tools offer powerful ways to help developers interact with the Elastic Stack. With Console, you can bypass using curl from the terminal and tinker with your Elasticsearch data directly. The Search Profiler lets you easily see where time is spent during search requests. And authoring complex grok patterns in your Logstash configuration becomes a breeze with the Grok Debugger.

In next 5 minutes, we are going to test drive ELK stack on PWD playground.

Let’s get started –

Open up https://play-with-docker.com


Click on icon next to Instances to open up ready-made templates for Docker Swarm Mode:


Choose the first template (as highlighted in the above figure) to select 3 Managers and 2 Workers. It will bring up Docker 17.06 Swarm Mode cluster in just 10 seconds.

Run the below command to show up the cluster nodes:


$docker node ls


Run the necessary command on node which will run elasticsearch:


$sysctl -w vm.max_map_count=262144

$echo ‘vm.max_map_count=262144’ >> /etc/sysctl.conf


Clone the GitHub repository:


$git clone https://github.com/ajeetraina/docker101

$cd docker101/play-with-docker/visualizer


Run the below command to bring up visualiser tool as shown below:

Soon you will notice port 8080 displayed on the top of the page which when clicked will open up visualiser tool.

It’s time to clone ELK stack and execute the below command to bring up ELK stack across Docker 17.06 Swarm Mode cluster:


$git clone https://github.com/ajeetraina/swarm-elk

$cd swarm-elk

$docker stack deploy -c docker-compose.yml myself



[Credits to Andrew Hromis for building this docker-compose file. I leveraged his project repository to bring up the ELK stack in the first try]

You will soon see the below list of containers appearing on the nodes:

Run the below command to see the list of services running across the cluster:


$docker service ls


Click on port 5601 displayed on the top of the PWD page:

Please Note:  Kibana need data in Elasticsearch to work with. The .kibana index holds Kibana related data, and if they is the only index you have there is no data available that Kibana can visualise.Before you can use Kibana you will therefore need to index some data into Elasticsearch. This can be done e.g. using Logstash or directly through the REST interface using curl.

Soon you will see the below Kibana page:


Enabling High Availability for Elastic Stack through scaling

Let us scale out more number of replicas for elasticsearch:

Pushing data into Logstash:

Example #1:

Let us push NGINX web server logs into logstash and see if Kibana is able to detect it:


$docker run -d –name nginx-with-syslog –log-driver=syslog –log-opt syslog-address=udp:// -p 80:80 nginx:alpine


Now if you open up Kibana UI, you should be able to see logs being displayed for Nginx:

Example #2:

We can also push logs to logstash using the below command:


$docker run –rm -it –log-driver=gelf –log-opt gelf-address=udp:// alpine ping


Open up Kibana and now you will see the below GREEN status:

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.


Topology Aware Scheduling under Docker v17.05.0 Swarm Mode Cluster


Docker 17.05.0 Final release went public exactly 2 week back.This community release was the first release as part of new Moby project.  With this release, numerous interesting features  like Multi-Stage Build support to the builder, using build-time ARG in FROM, DEB packaging for Ubuntu 17.04(Zesty Zapus)  has been included. With this latest release, Docker Team brought numerous new features and improvements in terms of Swarm Mode. Example – synchronous service commands, automatic service rollback on failure, improvement over raft transport package, service logs formatting etc. 



Placement Preference under Swarm Mode:

One of the prominent new feature introduced is placement preference  under 17.05.0-CE Swarm Mode . Placement preference feature allows you to divide tasks evenly over different categories of nodes. It allows you to balance tasks between multiple datacenters or availability zones. One can use a placement preference to spread out tasks to multiple datacenters and make the service more resilient in the face of a localized outage. You can use additional placement preferences to further divide tasks over groups of nodes.  Under this blog, we will setup 5-node Swarm Mode cluster on play-with-docker platform and see how to balance them over multiple racks within each datacenter. (Note – This is not real time scenario but based on assumption that nodes are being placed in 3 different racks).

Assumption:  There are 3 datacenter Racks holding respective nodes as shown:

{Rack-1=> Node1, Node2 and Node3},

{Rack-2=> Node4}  &

{Rack-3=> Node5}


Creating Swarm Master Node:

Open up Docker Playground to build up Swarm Cluster.


$ docker swarm init –advertise-addr


Screen Shot 2017-05-20 at 7.04.26 AM


Adding Worker Nodes to Swarm Cluster


$ docker swarm join –token <token-id>


Screen Shot 2017-05-20 at 7.07.53 AM

Create 3 more instances and add those nodes as worker nodes. This should build up 5 node Swarm Mode cluster.

Screen Shot 2017-05-20 at 7.08.51 AM


Setting up Visualizer Tool 

To showcase this demo, I will leverage a fancy popular Visualizer tool.


$git clone https://github.com/ajeetraina/docker101

$cd docker101/play-with-docker/visualizer



All you need is to execute docker-compose command to bring up visualizer container:



$docker-compose up -d



Screen Shot 2017-05-20 at 7.09.57 AM


Click on port “8080” which gets displayed on top centre of this page and it should display a fancy visualiser depicting Swarm Mode cluster nodes.

Screen Shot 2017-05-20 at 7.13.50 AM

Creating an Overlay Network:


 $docker network create -d overlay collabnet


Screen Shot 2017-05-20 at 7.15.15 AM


Let us try to create service with no preference placement or no node labels.

Setting up WordPress DB service:


$docker service create –replicas 10 –name wordpressdb1 –network collabnet –env MYSQL_ROOT_PASSWORD=collab123 –env MYSQL_DATABASE=wordpress mysql:latest


Screen Shot 2017-05-20 at 7.19.02 AM

When you run the above command, the swarm will spread the containers evenly node-by-node. Hence, you will see 2-containers per node as shown below:

 Screen Shot 2017-05-20 at 7.22.58 AM 

Setting up WordPress Web Application:


$docker service create –env WORDPRESS_DB_HOST=wordpressdb1 –env WORDPRESS_DB_PASSWORD=collab123 –network collabnet –replicas 3 –name wordpressapp –publish 80:80/tcp wordpress:latest


Screen Shot 2017-05-20 at 7.30.55 AM


Screen Shot 2017-05-20 at 7.34.51 AM 

As per the visualizer, you might end up with uneven distribution of services. Example., Rack-1 holding node-1, node-2 and node-3 looks to have almost equal distribution of services, Rack-2 which holds node3 lack WordPress fronted application.

Here Comes Placement Preference for a rescue…

Under the latest release, Docker team has introduced a new feature called “Placement Preference Scheduling”. Let us spend some time to understand what it actually means. You can set up the service to divide tasks evenly over different categories of nodes. One example of where this can be useful is to balance tasks over a set of datacenters or availability zones. 

This uses --placement-pref with a spread strategy (currently the only supported strategy) to spread tasks evenly over the values of the datacenter node label. In this example, we assume that every node has a datacenter node label attached to it. If there are three different values of this label among nodes in the swarm, one third of the tasks will be placed on the nodes associated with each value. This is true even if there are more nodes with one value than another. For example, consider the following set of nodes:

  • Three nodes with node.labels.datacenter=india
  • One node with node.labels.datacenter=uk
  • One node with node.labels.datacenter=us

Considering the last example, since we are spreading over the values of the datacenter label and the service has 5 replicas, at least 1 replica should be available  in each datacenter. There are three nodes associated with the value “india”, so each one will get one of the three replicas reserved for this value. There is 1 node with the value “uk”, and hence 1 replica for this value will be receiving it. Finally, “us” has a single node that will again get atleast 1 replica of each service reserved.

To understand more clearly, let us assign node labels to Rack nodes as shown below:

Rack-1 : 



$ docker node update –label-add datacenter=india node1


$ docker node update –label-add datacenter=india node2


$ docker node update –label-add datacenter=india node3



$ docker node update –label-add datacenter=uk node4



$ docker node update –label-add datacenter=us node5


Screen Shot 2017-05-20 at 7.46.33 AM


Removing both the services:


$ docker service rm wordpressdb1 wordpressapp


Let us now pass placement preference parameter to the docker service command:


$docker service create –replicas 10 –name wordpressdb1 –network collabnet –placement-pref “spread=node.labels.datacenter” –env MYSQL_ROOT_PASSWORD=collab123 –env MYSQL_DATABASE=wordpress mysql:latest


Screen Shot 2017-05-20 at 8.05.52 AM



Screen Shot 2017-05-20 at 8.05.14 AM

Rack-1(node1+node2+node3) has 4 copies, Rack-2(node4) has 3 copies and Rack-3(node5) has 3 copies.

Let us run WordPress Application service likewise:


$docker service create –env WORDPRESS_DB_HOST=wordpressdb1 –env WORDPRESS_DB_PASSWORD=collab123 –placement-pref “spread=node.labels.datacenter” –network collabnet –replicas 3 –name wordpressapp –publish 80:80/tcp wordpress:latest


Screen Shot 2017-05-20 at 8.09.29 AM

Visualizer: As shown below, we have used placement preference feature to ensure that the service containers get distributed across the swarm cluster on both the racks.

Screen Shot 2017-05-20 at 8.10.41 AM


As shown above, –placement-pref ensures that the task is spread evenly over the values of the datacenter node label. Currently spread strategy is only supported.Both engine labels and node labels are supported by placement preferences.

Please Note: If you want to try this feature with Docker compose , you will need Compose v3.3 which is slated to arrive under 17.06 release.

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

Know more about the latest Docker releases clicking on this link.