A First Look at Docker Application Package “docker-app”

Estimated Reading Time: 10 minutes

 

Did you know? There are more than 300,000 Docker Compose files on GitHub.

Docker Compose is a tool for defining and running multi-container Docker applications. It  is an amazing developer tool to create the development environment for your application stack. It allows you to define each component of your application following a clear and simple syntax in YAML files. It works in all environments: production, staging, development, testing, as well as CI workflows. Though Compose files are easy to describe a set of related services but there are couple of problems which has emerged in the past. One of the major concern has been around multiple environments to deploy the application with small configuration differences.

Consider a scenario where you have separate development, test, and production environments for your Web application. Under development environment, your team might be spending time in building up Web application(say, WordPress), developing WP Plugins and templates, debugging the issue etc.  When you are in development you’ll probably want to check your code changes in real-time. The usual way to do this is mounting a volume with your source code in the container that has the runtime of your application. But for production this works differently. Before you host your web application in production environment, you might want to turn-off the debug mode and host it under the right port so as to test your application usability and accessibility. In production you have a cluster with multiple nodes, and in most of the case volume is local to the node where your container (or service) is running, then you cannot mount the source code without complex stuff that involve code synchronization, signals, etc. In nutshell, this might require multiple Docker compose files for each environment and as your number of service applications increases, it becomes more cumbersome to manage those pieces of Compose files. Hence, we need a tool which can ease the way Compose files can be shareable across  different environment seamlessly.

To solve this problem, Docker, Inc recently announced a new tool called “docker-app”(Application Packages) which makes “Compose files more reusable and shareable”. This tool not only makes Compose file shareable but provide us with simplified approach to share multi-service application (not just Docker Image) directly on Dockerhub.

 

 

Under this blog post, I will showcase how docker-app tool makes it easier to use Docker Compose for sharing and collaboration and then pushing it directly to DockerHub. Let us get started-

Prerequisite:

  • Click on Icon near to Instance to choose 3 Managers & 2 Worker Nodes

Deploy 5 Node Swarm Mode Cluster

$ docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
juld0kwbajyn11gx3bon9bsct *   manager1            Ready               Active              Leader              18.03.1-ce
uu675q2209xotom4vys0el5jw     manager2            Ready               Active              Reachable           18.03.1-ce
05jewa2brfkvgzklpvlze01rr     manager3            Ready               Active              Reachable           18.03.1-ce
n3frm1rv4gn93his3511llm6r     worker1             Ready               Active                                  18.03.1-ce
50vsx5nvwx5rbkxob2ua1c6dr     worker2             Ready               Active                                  18.03.1-ce

Cloning the Repository

$ git clone https://github.com/ajeetraina/app
Cloning into 'app'...remote: Counting objects: 14147, done.
remote: Total 14147 (delta 0), reused 0 (delta 0), pack-reused 14147Receiving objects: 100% (14147/14147), 17.32 MiB | 18.43 MiB/s, done.
Resolving deltas: 100% (5152/5152), done.

Installing docker-app

wget https://github.com/docker/app/releases/download/v0.3.0/docker-app-linux.tar.gz
tar xf docker-app-linux.tar.gz
cp docker-app-linux /usr/local/bin/docker-app

OR

$ ./install.sh
Connecting to github.com (192.30.253.112:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.227.152:443)
docker-app-linux.tar 100% |**************************************************************|  8780k  0:00:00 ETA
[manager1] (local) root@192.168.0.13 ~/app
$ 

Verify docker-app version

$ docker-app version
Version:      v0.3.0
Git commit:   fba6a09
Built:        Fri Jun 29 13:09:30 2018
OS/Arch:      linux/amd64
Experimental: off
Renderers:    none

The docker-app tool comes with various options as shown below:

$ docker-app
Build and deploy Docker applications.

Usage:
  docker-app [command]

Available Commands:
  deploy      Deploy or update an application
  helm        Generate a Helm chart
  help        Help about any command
  init        Start building a Docker application
  inspect     Shows metadata and settings for a given application
  ls          List applications.
  merge       Merge the application as a single file multi-document YAML
  push        Push the application to a registry
  render      Render the Compose file for the application
  save        Save the application as an image to the docker daemon(in preparation for push)
  split       Split a single-file application into multiple files
  version     Print version information

Flags:
      --debug   Enable debug mode
  -h, --help    help for docker-app

Use "docker-app [command] --help" for more information about a command.
[manager1] (local) root@192.168.0.48 ~/app

WordPress Application under dev & Prod environment

If you browse to app/examples/wordpress directory under GitHub Repo, you will see that there is a folder called wordpress.dockerapp that contains three YAML documents:

  • metadatas
  • the Compose file
  • settings for your application

Okay, Fine ! But how you created those files?

The docker-app tool comes with an option “init” which initialize any application with the above  3 YAML files and the directory structure can be initialized with the below command:

docker-app init --single-file wordpress

I have already created a directory structure for my environment and you can find few examples under this directory.

Listing the WordPress Application package related files/directories

$ ls
README.md            install-wp           with-secrets.yml
devel                prod                 wordpress.dockerapp

As you see above, I have created each folder for my environment – dev and prod. Under these directories, I have created prod-settings.yml and dev-settings.yml. You can view the content via this link.

WordPress Application Package for Dev Environ

I can pass “-f” <YAML> parameter to docker-app tool to render the respective environment settings seamlessly as shown below:

$ docker-app render wordpress -f devel/dev-settings.yml
version: "3.6"
services:
  mysql:
    deploy:
      mode: replicated
      replicas: 1
      endpoint_mode: dnsrr
    environment:
      MYSQL_DATABASE: wordpressdata
      MYSQL_PASSWORD: wordpress
      MYSQL_ROOT_PASSWORD: wordpress101
      MYSQL_USER: wordpress
    image: mysql:5.6
    networks:
      overlay: null
    volumes:
    - type: volume
      source: db_data
      target: /var/lib/mysql
  wordpress:
    depends_on:
    - mysql
    deploy:
      mode: replicated
      replicas: 1
      endpoint_mode: vip
    environment:
      WORDPRESS_DB_HOST: mysql
      WORDPRESS_DB_NAME: wordpressdata
      WORDPRESS_DB_PASSWORD: wordpress
      WORDPRESS_DB_USER: wordpress
      WORDPRESS_DEBUG: "true"
    image: wordpress
    networks:
      overlay: null
    ports:
    - mode: ingress
      target: 80
      published: 8082
      protocol: tcp
networks:
  overlay: {}
volumes:
  db_data:
    name: db_data

WordPress Application Package for Prod

Under Prod environment, I have the following content under prod/prod-settings.yml as shown :

debug: false
wordpress:
  port: 80

For production environment, it is obvious that I want my application to be exposed under the standard port:80. Post rendering, you should be able to see port:80 exposed as shown below in the snippet:

 image: wordpress
    networks:
      overlay: null
    ports:
    - mode: ingress
      target: 80
      published: 80
      protocol: tcp
networks:
  overlay: {}
volumes:
  db_data:
    name: db_data

 

Inspect the WordPress App

$ docker-app inspect wordpress
wordpress 1.0.0
Maintained by: ajeetraina <ajeetraina@gmail.com>

Welcome to Collabnix

Setting                       Default
-------                       -------
debug                         true
mysql.database                wordpressdata
mysql.image.version           5.6
mysql.rootpass                wordpress101
mysql.scale.endpoint_mode     dnsrr
mysql.scale.mode              replicated
mysql.scale.replicas          1
mysql.user.name               wordpress
mysql.user.password           wordpress
volumes.db_data.name          db_data
wordpress.port                8081
wordpress.scale.endpoint_mode vip
wordpress.scale.mode          replicated
wordpress.scale.replicas      1
[manager1] (local) root@192.168.0.13 ~/app/examples/wordpress
$

Deploying the WordPress App

$ docker-app deploy wordpress
Creating network wordpress_overlay
Creating service wordpress_mysql
Creating service wordpress_wordpress

Switching to Dev Environ

If I want to switch back to Dev environment, all I need is to pass the dev specific YAML file using “-f” parameter  as shown below:

$docker-app deploy wordpress -f devel/dev-settings.yml

docker-app

Switching to Prod Environ

$docker-app deploy wordpress -f prod/prod-settings.yml

docker-app

[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app deploy -f devel/dev-settings.yml
Updating service wordpress_wordpress (id: l95b4s6xi7q5mg7vj26lhzslb)
Updating service wordpress_mysql (id: lhr4h2uaer861zz1b04pst5sh)
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app deploy -f prod/prod-settings.yml
Updating service wordpress_wordpress (id: l95b4s6xi7q5mg7vj26lhzslb)
Updating service wordpress_mysql (id: lhr4h2uaer861zz1b04pst5sh)
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$

Pushing Application Package to Dockerhub

So I have my application ready to be pushed to Dockerhub. Yes, you heard it right. I said, Application packages and NOT Docker Image.

Let me first authenticate myself before I push it to Dockerhub registry:

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to
 https://hub.docker.com to create one.
Username: ajeetraina
Password:
Login Succeeded
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$

Saving this Application Package as Docker Image

The docker-app CLI is feature-rich and allows to save the entire application as a Docker Image. Let’s try it out –

[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app save wordpress
Saved application as image: wordpress.dockerapp:1.0.0
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$

Listing out the images

$ docker images
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
wordpress.dockerapp   1.0.0               c1ec4d18c16c        47 seconds ago      1.62kB
mysql                 5.6                 97fdbdd65c6a        3 days ago          256MB
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$

Listing out the services

$ docker stack services wordpress
ID                  NAME                  MODE                REPLICAS            IMAGE               PORTS
l95b4s6xi7q5        wordpress_wordpress   replicated          1/1                 wordpress:latest    *:80->80
/tcp
lhr4h2uaer86        wordpress_mysql       replicated          1/1                 mysql:5.6
[manager1] (local) root@192.168.0.48 ~/docker101/play-with-docker/visualizer

Using docker-app ls command to list out the application packages

The ‘ls’ command has been recently introduced under v0.3.0. Let us try it once –

$ docker-app ls
REPOSITORY            TAG                 IMAGE ID            CREATED              SIZE
wordpress.dockerapp   1.0.1               299fb78857cb        About a minute ago   1.62kB
wordpress.dockerapp   1.0.0               c1ec4d18c16c        16 minutes ago       1.62kB

Pusing it to Dockerhub

$ docker-app push --namespace ajeetraina --tag 1.0.1
The push refers to repository [docker.io/ajeetraina/wordpress.dockerapp]
51cfe2cfc2a8: Pushed
1.0.1: digest: sha256:14145fc6e743f09f92177a372b4a4851796ab6b8dc8fe49a0882fc5b5c1be4f9 size: 524

Say, you built WordPress application package and pushed it to Dockerhub. Now one of your colleague want to pull it on his development system and deploy in his environment.

Pulling it from Dockerhub

$ docker pull ajeetraina/wordpress.dockerapp:1.0.1
1.0.1: Pulling from ajeetraina/wordpress.dockerapp
a59931d48895: Pull complete
Digest: sha256:14145fc6e743f09f92177a372b4a4851796ab6b8dc8fe49a0882fc5b5c1be4f9
Status: Downloaded newer image for ajeetraina/wordpress.dockerapp:1.0.1
[manager3] (local) root@192.168.0.24 ~/app
$ docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             SIZE
ajeetraina/wordpress.dockerapp   1.0.1               299fb78857cb        8 minutes ago       1.62kB
[manager3] (local) root@192.168.0.24 ~/app
$

Deploying the Application

$ docker images
REPOSITORY                       TAG                 IMAGE ID            CREATED             SIZE
ajeetraina/wordpress.dockerapp   1.0.1               299fb78857cb        9 minutes ago       1.62kB
[manager3] (local) root@192.168.0.24 ~/app
$ docker-app deploy ajeetraina/wordpress
Creating network wordpress_overlay
Creating service wordpress_mysql
Creating service wordpress_wordpress
[manager3] (local) root@192.168.0.24 ~/app
$

Using docker-app merge option

Docker Team has introduced docker-app merge option under the new 0.3.0 release.

[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ docker-app merge -o mywordpress
[manager1] (local) root@192.168.0.48 ~/app/examples/wordpress
$ ls
README.md            install-wp           prod                 wordpress.dockerapp
devel                mywordpress          with-secrets.yml
$ cat mywordpress
version: 1.0.1
name: wordpress
description: "Welcome to Collabnix"
maintainers:
  - name: ajeetraina
    email: ajeetraina@gmail.com
targets:
  swarm: true
  kubernetes: true

--
version: "3.6"

services:

  mysql:
    image: mysql:${mysql.image.version}
    environment:
      MYSQL_ROOT_PASSWORD: ${mysql.rootpass}
      MYSQL_DATABASE: ${mysql.database}
      MYSQL_USER: ${mysql.user.name}
      MYSQL_PASSWORD: ${mysql.user.password}
    volumes:
       - source: db_data
         target: /var/lib/mysql
         type: volume
    networks:
       - overlay
    deploy:
      mode: ${mysql.scale.mode}
      replicas: ${mysql.scale.replicas}
      endpoint_mode: ${mysql.scale.endpoint_mode}

  wordpress:
    image: wordpress
    environment:
      WORDPRESS_DB_USER: ${mysql.user.name}
      WORDPRESS_DB_PASSWORD: ${mysql.user.password}
      WORDPRESS_DB_NAME: ${mysql.database}
      WORDPRESS_DB_HOST: mysql
      WORDPRESS_DEBUG: ${debug}
    ports:
      - "${wordpress.port}:80"
    networks:
      - overlay
    deploy:
      mode: ${wordpress.scale.mode}
      replicas: ${wordpress.scale.replicas}
      endpoint_mode: ${wordpress.scale.endpoint_mode}
    depends_on:
      - mysql

volumes:
  db_data:
     name: ${volumes.db_data.name}

networks:
  overlay:

--
debug: true
mysql:
  image:
    version: 5.6
  rootpass: wordpress101
  database: wordpressdata
  user:
    name: wordpress
    password: wordpress
  scale:
    endpoint_mode: dnsrr
    mode: replicated
    replicas: 1
wordpress:
  scale:
    mode: replicated
    replicas: 1
    endpoint_mode: vip
  port: 8081
volumes:
  db_data:
    name: db_data

docker-app comes with a few other helpful commands as well, in particular the ability to create Helm Charts from your Docker Applications. This can be useful if you’re adopting Kubernetes, and standardising on Helm to manage the lifecycle of your application components, but want to maintain the simplicity of Compose when writing you applications. This also makes it easy to run the same applications locally just using Docker, if you don’t want to be running a full Kubernetes cluster.

Did you find this blog helpful?  Feel free to share your experience. Get in touch with me at twitter @ajeetsraina.

If you want to keep track of latest Docker related information, follow me at https://www.linkedin.com/in/ajeetsraina/.

Top 5 Most Exciting Dockercon 2018 Announcements

Estimated Reading Time: 8 minutes

 

 

Yet another amazzing Dockercon !

I attended Dockercon 2018 last week which happened in the most geographically blessed city of Northern California – San Francisco & in the largest convention and exhibition complex- Moscone Center. With over 5000+ attendees from around the globe, 100+ sponsors, Hallway tracks, workshops & Hands-on labs, Dockercon allowed developers, sysadmins, Product Managers & industry evangelists come closer to share their wealth of experience around the container technology.This time I was lucky enough to get chance to visit Docker HQ, Townsend Street for the first time. It was an emotional as well as proud feeling to be part of such vibrant community home.

 

 

This Dockercon, there has been couple of exciting announcements.Three of the new features were targeted at Docker EE, while the two were for Docker Desktop. Here’s a rundown of what I think are the most 5 exciting announcements made last week:

 

Under this blog post, I will go through each one of the announcements in details.

1. Federated Application Management in Docker Enterprise Edition

 

With an estimated 85% of today’s enterprise IT organizations employing a multi-cloud strategy, it has become more critical that customers have a ‘single pane of glass’ for managing their entire application portfolio. Most enterprise organisations have a hybrid and multi-cloud strategy. Containers has helped to make applications portable but let us accept the fact that even though containers are portable today but the management of containers is still a nightmare. The reason being –

  • Each Cloud is managed under a separate operational model, duplicating efforts
  • Different security and access policies across each platform
  • Content is hard to distribute and track
  • Poor Infrastructure utilisation still remains
  • Emergence of Cloud-hosted K8s is exacerbating the challenges with managing containerised applications across multiple Clouds

This time Docker introduced new application management capabilities for Docker Enterprise Edition that will allow organisations to federate applications across Docker Enterprise Edition environments deployed on-premises and in the cloud as well as across cloud-hosted Kubernetes. This includes Azure Kubernetes Service (AKS), AWS Elastic Container Service for Kubernetes (EKS), and Google Kubernetes Engine (GKE). The federated application management feature will automate the management and security of container applications on premises and across Kubernetes-based cloud services.It will provide a single management platform to enterprises so that they can centrally control and secure the software supply chain for all the containerized applications.

With this announcement, undoubtedly Docker Enterprise Edition is the only enterprise-ready container platform that can deliver federated application management with a secure supply chain. Not only does Docker give you your choice of Linux distribution or Windows Server, the choice of running in a virtual machine or on bare metal, running traditional or microservices applications with either Swarm or Kubernetes orchestration, it also gives you the flexibility to choose the right cloud for your needs.

 

 

Below are the list of use cases that are driving the need for federated management of Containerised applications  –

 

If you want to read more about it, please refer this official blog.

 

2. Kubernetes Support for Windows Server Container in Docker Enterprise Edition

The partnership between Docker and Microsoft is not new. They have been working together since 2014 to bring containers to Windows and .NET applications. This DockerCon, Docker & Microsoft both shared the next step in this partnership with the preview and demonstration of Kubernetes support on Windows Server with Docker Enterprise Edition.

With this announcement, Docker is the only platform to support production-grade Linux and Windows containers, as well as dual orchestration options with Swarm and Kubernetes.

There has been a rapid rise of Windows containers as organizations recognize the benefits of containerisation and want to apply them across their entire application portfolio and not just their Linux-based applications.

Docker and Microsoft brought container technology into Windows Server 2016, ensuring consistency for the same Docker Compose file and CLI commands across both Linux and Windows. Windows Server ships with a Docker Enterprise Edition engine, meaning all Windows containers today are based on Docker. Recognizing that most enterprise organizations have both Windows and Linux applications in their environment, we followed that up in 2017 with the ability to manage mixed Windows and Linux clusters in the same Docker Enterprise Edition environment, enabling support for hybrid applications and driving higher efficiencies and lower overhead for organizations. Using Swarm orchestration, operations teams could support different application teams with secure isolation between them, while also allowing Windows and Linux containers to communicate over a common overlay network.

If you want to know further details, refer this official blog.

3. Docker Desktop Template-Based Workflows for Enterprise Developers

Dockercon 2018 was NOT just for Enterprise customers, but  also for Developers. Talking about the new capabilities for Docker Desktop, it is getting new template-based workflows which will enable developers to build new containerized applications without having to learn Docker commands or write Docker files. This template-based workflows will also help development teams to share their own practices within the organisation.

On the 1st day of Dockercon, Docker team previewed an upcoming Docker Desktop feature that will make it easier than ever to design your own container-based applications. For a certain set of developers, the current iteration of Docker Desktop has everything one might need to containerize an applications, but it does require an understanding of the Dockerfile and Compose file specifications in order to get started and the Docker CLI to build and run your applications.

In the upcoming Docker Desktop release, you can expect the below features –

  • You will see new option – “Design New application”  as shown below Preference Pane UI.
  • It will be 100% graphical tool/feature.
  • This tool is a gift for anyone who doesn’t want to write Dockerfiles or Docker compose file.
  • Once a user click the button to start the “Custom application” workflow , he will be presented with a list  services which he can add to the application.
  • Each service which is selected will eventually become a container in the final application, but Docker Desktop will take care of creating the Dockerfiles and Compose files  as complete later steps.
  • Under this beta release, currently one can do some basic customization to the service like changing versions, port numbers, and a few other options depending on the service selected.
  • When all the services are selected, one should be ready to proceed, supply the application a name and specify where to store the files that will be generated and then hit the “Assemble” button.
  • The assemble step creates the Dockerfiles for each service, the Compose file used to start the entire application, and for most services some basic code stubs, giving one enough to start the application.

 

 

If you’re interested in getting early access to the new app design feature in Docker Desktop then please sign up at beta.docker.com.

4. Making Compose Easier with to use with the Application Packages

Soon after Dockercon, one of the most promising tool announced for Developers was Docker Application Packages (docker-app). The “docker-app” is an experimental utility to help make Compose files more reusable and sharable.

What problem does application packages solve?

Compose files do a great job of describing a set of related services. Not only are Compose files easy to write, they are generally easy to read as well. However, a couple of problems often emerge:

  1. You have several environments where you want to deploy the application, with small configuration differences
  2. You have lots of similar applications

Fundamentally, Compose files are not easy to share between concerns. Docker Application Packages aim to solve these problems and make Compose more useful for development and production.

In my next blog post, I will talk more about this tool. If you want to try your hands, head over to https://github.com/docker/app

5. Upcoming Support for Serverless Platform under Docker EE

Recently, Function as a Service (FaaS) programming paradigm has gained a lot of traction in the cloud community. At first, only large cloud providers such as AWS Lambda, Google Cloud Functions or Azure Functions provided such services with a pay-per-invocation model, but since then interest has increased for developers and entreprises to build their own solutions on an open source model.

This Dockercon, Docker identified at least 9 different frameworks out of which the following six: OpenFaaS, nuclio, Gestalt, Riff, Fn and OpenWhisk were already confirmed to be supported under the upcoming Docker Enterprise Edition. Docker, Inc started an open source repository to document how to install all these frameworks on Kubernetes on Docker EE, with the goal of providing a benchmark of these frameworks: docker serverless benchmark Github Repository. Pull Requests are welcome to document how to install other serverless frameworks on Docker EE.

 

Did you find this blog helpful? I am really excited about the upcoming Docker days and feel that these upcoming features will really excite the community. If you have any questions, join me this July 7th at Docker Bangalore Meetup Group, Nutanix Office where I can going to go deeper into Dockercon 2018 Announcements. See you there..

 

Kubernetes Application Deployment Made Easy using Helm on Docker for Mac 18.05.0 CE

Estimated Reading Time: 10 minutes

 

Docker for Mac 18.05.0 CE Release went GA last month. With this release, you can now select your orchestrator directly from the UI in the “Kubernetes” pane which allows “docker stack” commands to deploy to swarm clusters, even if Kubernetes is enabled in Docker for Mac. This feature was introduced for the first time under any Desktop Edition. To try out this feature, ensure that you are using Edge Release of Docker for Mac 18.05.0 CE. Once you update your Docker for Mac, you can find this new feature by clicking on Preference Pane UI and then selecting Kubernetes as shown below:

 

 

 

Whenever you select your choice of orchestrator, it updates ~/.docker/config.json file in the backend as shown below:

 

Docker for Mac is used everyday by hundreds of thousands of developers to build, test and debug containerized apps in their local and dev environment. Developers building both docker-compose and Swarm-based apps, and apps destined for deployment on Kubernetes now get a simple-to-use development system that takes optimal advantage of their laptop or workstation. All container tasks – build, run and push – will run on the same Docker instance with a shared set of images, volumes and containers. With the current release, it is way more simple to install, so you can have Docker containers running on your Mac in just a few minutes.

Check out the curated list of blogs around Docker for Mac

 

Docker for Mac is built with LinuxKit. How to access the LinuxKit VM
Top 5 Exclusive Features of Docker for Mac That you can’t afford to ignore
5 Minutes to Bootstrap Kubernetes Cluster on GKE using Docker for Mac 18.03.0
Context Switching Made Easy under Kubernetes powered Docker for Mac 18.02.0
2-minutes to Kubernetes Cluster on Docker for Mac 18.01 using Swarm CLI
Docker For Mac 1.13.0 brings support for macOS Sierra, now runs ARM & AARCH64 based Docker containers
Docker for Mac 18.03.0 now comes with NFS Volume Sharing Support for Kubernetes

Docker for Mac provides docker stack command to deploy your application for both Swarm and Kubernetes. This become very useful for Docker Swarm users as they can use the same Swarm CLI to bring up Kubernetes users. But here is an extra bonus – Docker for Mac now works flawlessly for Helm Package Manager.

Why Yet another Package Manager?

Let’s accept the fact that Kubernetes can become very complex with all the objects you need to handle ― such as ConfigMaps, services, pods, Persistent Volumes ― in addition to the number of releases you need to manage. These can be managed with Kubernetes Helm, which offers a simple way to package everything into one simple application and advertises what you can configure.

In case you are completely new – Helm is an open source project that enables developers to create packages of containerized apps to make installation much simpler. Helm is the package manager for Kubernetes and it’s the best way to find, share, and deploy software to k8s.The project was initially created by Deis and has since been donated to the Cloud Native Computing Foundation (CNCF).

Users can install Helm with one click or configure it to suit their organization’s needs. For example, if you want to package and release version 1.0, making only certain parts configurable, this can be done with Helm. Then with version 2.0, additional parts can be made configurable.

Up until now, it was a sub-project of Kubernetes, the popular container orchestration tool, but as of today it is a stand-alone project.

 

Helm is built on three important concepts:

  • Charts – a bundle of information necessary to create an instance of a Kubernetes application
  • Config – contains configuration information that can be merged into a packaged chart to create a releasable object
  • Release – a running instance of a chart, combined with a specific config

Architecture of Helm:

Architecturally it’s built on two major components:

 

Helm Client, a command line tool with the following responsibilities

  • Interacting with the Tiller server
  • Sending charts to be installed
  • Upgrading or uninstalling of existing releases
  • Managing repositories

Tiller Server, an in-cluster server with the following responsibilities:

  • Interacts with the Helm client
  • Interfaces the Kubernetes API server
  • Combining a chart and configuration to build a release
  • Installing charts and tracking the release
  • Upgrading and uninstalling charts

Both the Helm client and Tiller are written in Go and uses gRPC to interact with each other. Tiller (as the server part running inside Kubernetes) provides a gRPC server to connect with the client and it uses the k8s client library to communicate with Kubernetes. It does not require it’s own database as the information is stored within Kubernetes as ConfigMaps.

 

Installing Helm

Pre-requisites:

  • Docker for Mac 18.05.0 CE – Edge Release
  • Enable Kubernetes under Preference Pane UI

To install Helm, you just need a single-liner command on your macOS:

[Captains-Bay]🚩 >  brew install kubernetes-helm
[Captains-Bay]🚩 >  helm
The Kubernetes package manager

To begin working with Helm, run the 'helm init' command:

	$ helm init

This will install Tiller to your running Kubernetes cluster.
It will also set up any necessary local configuration.

Common actions from this point include:

- helm search:    search for charts
- helm fetch:     download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts

Environment:
  $HELM_HOME          set an alternative location for Helm files. By default, these are stored in ~/.helm
  $HELM_HOST          set an alternative Tiller host. The format is host:port
  $HELM_NO_PLUGINS    disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.
  $TILLER_NAMESPACE   set an alternative Tiller namespace (default "kube-system")
  $KUBECONFIG         set an alternative Kubernetes configuration file (default "~/.kube/config")

Usage:
  helm [command]

Available Commands:
  completion  Generate autocompletions script for the specified shell (bash or zsh)
  create      create a new chart with the given name
  delete      given a release name, delete the release from Kubernetes
  dependency  manage a chart's dependencies
  fetch       download a chart from a repository and (optionally) unpack it in local directory
  get         download a named release
  history     fetch release history
  home        displays the location of HELM_HOME
  init        initialize Helm on both client and server
  inspect     inspect a chart
  install     install a chart archive
  lint        examines a chart for possible issues
  list        list releases
  package     package a chart directory into a chart archive
  plugin      add, list, or remove Helm plugins
  repo        add, list, remove, update, and index chart repositories
  reset       uninstalls Tiller from a cluster
  rollback    roll back a release to a previous revision
  search      search for a keyword in charts
  serve       start a local http web server
  status      displays the status of the named release
  template    locally render templates
  test        test a release
  upgrade     upgrade a release
  verify      verify that a chart at the given path has been signed and is valid
  version     print the client/server version information

Flags:
      --debug                           enable verbose output
  -h, --help                            help for helm
      --home string                     location of your Helm config. Overrides $HELM_HOME (default "/Users/ajeetraina/.helm")
      --host string                     address of Tiller. Overrides $HELM_HOST
      --kube-context string             name of the kubeconfig context to use
      --tiller-connection-timeout int   the duration (in seconds) Helm will wait to establish a connection to tiller (default 300)
      --tiller-namespace string         namespace of Tiller (default "kube-system")

Use "helm [command] --help" for more information about a command.

Verify the Helm version. If Server and client version doesn’t match, you need to upgrade to deploy application seamlessly(as shown below):

[Captains-Bay]🚩 >  helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
[Captains-Bay]🚩 >  helm init --upgrade
$HELM_HOME has been configured at /Users/ajeetraina/.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!
[Captains-Bay]🚩 >  helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Installing WordPress Application using Helm

Say, you want to install WordPress application using Helm. First you need to update the repository and then you can search the application using helm search command as shown below:

[Captains-Bay]🚩 > helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈

[Captains-Bay]🚩 >  helm search wordpress
NAME            	CHART VERSION	APP VERSION	DESCRIPTION
stable/wordpress	1.0.2        	4.9.4      	Web publishing platform for building blogs and ...

Just a single-liner command and your WordPress application is up and running:

[Captains-Bay]🚩 >  helm install stable/wordpress --name mywp
NAME:   mywp
LAST DEPLOYED: Sat Jun  2 07:19:25 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Deployment
NAME            DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
mywp-mariadb    1        1        1           0          0s
mywp-wordpress  1        1        1           0          0s

==> v1/Pod(related)
NAME                             READY  STATUS             RESTARTS  AGE
mywp-mariadb-b689ddf74-mlprh     0/1    Init:0/1           0         0s
mywp-wordpress-774555bd4b-hcdc2  0/1    ContainerCreating  0         0s

==> v1/Secret
NAME            TYPE    DATA  AGE
mywp-mariadb    Opaque  2     1s
mywp-wordpress  Opaque  2     1s

==> v1/ConfigMap
NAME                DATA  AGE
mywp-mariadb        1     1s
mywp-mariadb-tests  1     1s

==> v1/PersistentVolumeClaim
NAME            STATUS   VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
mywp-mariadb    Bound    pvc-2e1f1122-6607-11e8-8d79-025000000001  8Gi       RWO           hostpath      1s
mywp-wordpress  Pending  hostpath                                  1s

==> v1/Service
NAME            TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                     AGE
mywp-mariadb    ClusterIP     10.98.65.236    <none>       3306/TCP                    0s
mywp-wordpress  LoadBalancer  10.109.204.199  localhost    80:31016/TCP,443:30638/TCP  0s


NOTES:
1. Get the WordPress URL:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace default -w mywp-wordpress'

  export SERVICE_IP=$(kubectl get svc --namespace default mywp-wordpress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
  echo http://$SERVICE_IP/admin

2. Login with the following credentials to see your blog

  echo Username: user
  echo Password: $(kubectl get secret --namespace default mywp-wordpress -o jsonpath="{.data.wordpress-password}" | base64 --decode)

[Captains-Bay]🚩 >

Checking the Status of WordPress Application

[Captains-Bay]🚩 >  kubectl get svc --namespace default -w mywp-wordpress
NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
mywp-wordpress   LoadBalancer   10.109.204.199   localhost     80:31016/TCP,443:30638/TCP   47s

That’s it. Just browse the WordPress using your localhost IP:

 

Cleaning up WordPress

[Captains-Bay] > helm delete mywp
release "mywp" deleted

Installing Prometheus Stack using Helm

[Captains-Bay]🚩 >  helm install stable/prometheus
NAME:   hasty-ladybug
LAST DEPLOYED: Sun Jun  3 09:00:30 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolumeClaim
NAME                                   STATUS  VOLUME                                    CAPACITY  ACCESS MODES  STORAGECLASS  AGE
hasty-ladybug-prometheus-alertmanager  Bound   pvc-7732a80b-66de-11e8-b7d4-025000000001  2Gi       RWO           hostpath      2s
hasty-ladybug-prometheus-server        Bound   pvc-773541f3-66de-11e8-b7d4-025000000001  8Gi       RWO           hostpath      2s

==> v1/ServiceAccount
NAME                                         SECRETS  AGE
hasty-ladybug-prometheus-alertmanager        1        2s
hasty-ladybug-prometheus-kube-state-metrics  1        2s
hasty-ladybug-prometheus-node-exporter       1        2s
hasty-ladybug-prometheus-pushgateway         1        2s
hasty-ladybug-prometheus-server              1        2s

==> v1beta1/ClusterRoleBinding
NAME                                         AGE
hasty-ladybug-prometheus-kube-state-metrics  2s
hasty-ladybug-prometheus-server              2s

==> v1beta1/DaemonSet
NAME                                    DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR  AGE
hasty-ladybug-prometheus-node-exporter  1        1        0      1           0          <none>         2s

==> v1/Pod(related)
NAME                                                          READY  STATUS             RESTARTS  AGE
hasty-ladybug-prometheus-node-exporter-9ggqj                  0/1    ContainerCreating  0         2s
hasty-ladybug-prometheus-alertmanager-5c67b8b874-4xxtj        0/2    ContainerCreating  0         2s
hasty-ladybug-prometheus-kube-state-metrics-5cbcd4d86c-788p4  0/1    ContainerCreating  0         2s
hasty-ladybug-prometheus-pushgateway-c45b7fd6f-2wwzm          0/1    Pending            0         2s
hasty-ladybug-prometheus-server-799d6c7c75-jps8k              0/2    Init:0/1           0         2s

==> v1/ConfigMap
NAME                                   DATA  AGE
hasty-ladybug-prometheus-alertmanager  1     2s
hasty-ladybug-prometheus-server        3     2s

==> v1beta1/ClusterRole
NAME                                         AGE
hasty-ladybug-prometheus-kube-state-metrics  2s
hasty-ladybug-prometheus-server              2s

==> v1/Service
NAME                                         TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
hasty-ladybug-prometheus-alertmanager        ClusterIP  10.96.193.91   <none>       80/TCP    2s
hasty-ladybug-prometheus-kube-state-metrics  ClusterIP  None           <none>       80/TCP    2s
hasty-ladybug-prometheus-node-exporter       ClusterIP  None           <none>       9100/TCP  2s
hasty-ladybug-prometheus-pushgateway         ClusterIP  10.97.92.108   <none>       9091/TCP  2s
hasty-ladybug-prometheus-server              ClusterIP  10.96.118.138  <none>       80/TCP    2s

==> v1beta1/Deployment
NAME                                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
hasty-ladybug-prometheus-alertmanager        1        1        1           0          2s
hasty-ladybug-prometheus-kube-state-metrics  1        1        1           0          2s
hasty-ladybug-prometheus-pushgateway         1        1        1           0          2s
hasty-ladybug-prometheus-server              1        1        1           0          2s


NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
hasty-ladybug-prometheus-server.default.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
hasty-ladybug-prometheus-alertmanager.default.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9093


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
hasty-ladybug-prometheus-pushgateway.default.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/
[Captains-Bay]🚩 >  helm ls
NAME         	REVISION	UPDATED                 	STATUS  	CHART           	NAMESPACE
hasty-ladybug	1       	Sun Jun  3 09:00:30 2018	DEPLOYED	prometheus-6.7.0	default
mywp         	1       	Sat Jun  2 07:19:25 2018	DEPLOYED	wordpress-1.0.2 	default

Accessing Prometheus UI under Web Browser

export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 9090

Now you can access Prometheus:

open http://localhost:9090

 

Cleaning Up

[Captains-Bay]🚩 >  helm delete hasty-ladybug
release "hasty-ladybug" deleted

Did you find this blog helpful?  Feel free to share your experience. Get in touch with me at twitter @ajeetsraina.

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

How to Get the Most Out of Dockercon 2018 – (June 12 – 15th, SF)

Estimated Reading Time: 8 minutes

 

In a recent survey of 191 Top Fortune 1000 executives, 69% of them believe that conference presents a wealth of networking opportunities which is clearly a driving force behind their decision making. What it means is that the conference is NOT only about attending sessions but all about –

  • Gaining visibility,
  • Networking ,
  • Building strong relationship,
  • Connecting with Speakers
  • Connection with your customer
  • Meeting Like-Minded People

Hence, attending conference might be one of the best things you can do for your career. It is the time where you learn about new industry trends, gain some new skills and make all kind of new connections.

Dockercon 2018 is just a month away. It is happening this year at Moscone Center ~ one of the largest convention & exhibition complex in San Francisco, California starting June 13-15, 2018.  The event is expected to welcome 6,000+ developers, sysadmins, architects, VP of Apps & other IT leaders to get hands-on with the latest innovations in the container ecosystem. This is where all general DockerCon activities such as keynotes, breakouts, networking, meals, etc. will take place.

 

Use code “CaptainAjeet” to get 10% off on 2018.

New to Dockercon?

DockerCon is an event where the Docker community comes to learn, belong, and collaborate. Attendees are a mix of beginner, intermediate and advanced users who are all looking to level up their skills and go home inspired. With a 2 full days of training, more than 100 sessions, 100+ sponsors, free workshops & hands-on labs, and the wealth of experience brought by each attendee, DockerCon is the place to be if you’re looking to learn Docker in 2018.

I assume that you are able to convince your boss for Dockercon. Awesome! If you are still planning for it, here is a bonus – Docker Team has prepared this booklet for you.

Starting with booking your conference – the cost of many conferences increases substantially in the run up to the event, make sure you book as soon as possible to take advantage of early bird registration rates. Booking in advance always keep your travel and accommodation costs lower too, so whether you’re funding yourself or spending from a grant, it’s always good to save a little cash along the way (which you can hopefully put towards your next conference!). But in case you missed it out, here is the latest price list for various training sessions you might be interested to invest on –

Please Note: Your Full conference pass includes talks, workshops, keynotes, breakfast/lunch, unlimited coffee & access to party. The training & certification packages are additional purchases & not included in the full conference pass. In case you’re new, please note that as  a DockerCon attendee, you’ll have the opportunity to appear and earn ‘Docker Certified Associate’ designation with the digital certificate and verification to prove it! You should be able to schedule your exam during the below timeframe:

 

Under this blog post, I have come up with the list of important suggestions on how to make the most out of this amazing event.

#1 – Let’s begin with Housekeeping stuffs…

Based on my past experience, I suggest to keep your phone and laptop chargers with you.You’re going to spend a huge part of your day on your devices — don’t get caught with dead batteries. If you are travelling from India, be aware a normal laptop charger plugin point mightn’t work for you. You might need to carry US complaint laptop battery charger plug point. During my first visit to Docker Distribution System Conference, Berlin(Germany), I had a hard time looking out the right plug point for my laptop charger.

Secondly, ensure that you pack enough business cards. Make sure you have some on hand and a stash in your luggage. You never know how many people you’re going to meet. Thirdly, if you are presenter or conducting a workshop in the conference, ensure that you  bring the materials you need for demos. By no means should you spend the conference pitching to people who don’t want to be pitched. However, if one of those pre-set prospect meetings turns into a real sales opportunity, it’ll be more efficient — and impressive — if you can provide a walkthrough on the spot.

 

 

 

#2 – Prioritize Your Time

At any conference your time will compete with many activities. How can you maximize your time while you’re there? Here are some things to consider before you arrive.Are there many tracks going on at once? Will you have access to video recordings after the conference?Will there be a lounge or hack area outside the sessions?

The one session you should not miss is the keynote. All conferences have a keynote. It dictates the rest of the conference with high profile speakers or major announcements. Yet, the keynote is the single common session for the entire conference. You can use the keynote to break the ice while networking. Below is the snapshot of Dockercon Schedule you can go through and plan for prioritizing your time.

 

 

Conferences are broken down into “tracks” – these are different sets of simultaneous panels, talks or workshops that can be set up using a variety of criteria. Some of those criteria can be based on subject matter (e.g. Technology track, Business track, Legal track, etc.). Other times, they can set up based on format (e.g. Talks, Panels, Workshops, etc.). Here are the list

  • Kubernetes extensibility by Tim Hockin and Eric Tune (Google)
  • Accelerating Development Velocity of Production ML Systems with Docker by Kinnary Jangla (Pinterest)
  • Digital Transformation with Docker, Cloud and DevOps: How JCPenney Handles Black Friday and 100K Deployments Per Year by Sanjoy Mukherjee, (JCPenney)
  • Don’t have a Meltdown! Practical Steps for Defending your Apps by Liz Rice (Aqua) and Justin Cormack (Docker)
  • Creating Effective Docker Images by Abby Fuller (AWS)
  • App Transformation with Docker: 5 Patterns for Success by Elton Stoneman (Docker)

#3 – Break Out of Your Comfort Zone

A Conference is the time to meet new people, but its also a time build on the relationships you already have. You might be interacting with your counterparts in other part of world, hence this is the time where you can strengthen your bonding. If you know of people you want to reconnect, you can reach out few weeks before the conference to setup a time to meet for coffee or a meal while you’re at the event.

 

 

DockerCon is all about learning new things and connecting with the right people. One of such innovative platform introduced during Dockercon 2017 was called Hallway Track. Docker Hallway Track is an innovative platform that helps you find like-minded people to meet one-on-one and share knowledge in a structured way, so you get tangible results from networking.It is a one-on-one or group conversations based on topics of interest that you schedule with other attendees during DockerCon. Hallway Track’s recommendation algorithm curates an individualized selection of Hallway Track topics for each participant, based on their behavior and interests. Don’t miss out this chance which allows you meet and share knowledge with community members and practitioners at the conference.

 

 

 

 

Dockercon is a great place to meet Docker Captains. Docker Captains are technology experts and leaders in their communities who are passionate about sharing their Docker knowledge with others. Don’t miss out this chance to say “Hi” to them and know about their contribution towards the community.

#4 – Don’t Miss out the Evenings

Conferences don’t stop after the final session of the day. There will be many opportunities to network, dinners and other events. Be sure to go to as much of these as possible and use to meet new people. These are the perfect networking events and can help you relax after a long conference day.

#5 – Don’t Forget to Follow-Up

The insights you gained at the conference are likely to be useful for your team, so make sure to set aside time to pass on what you learned. Whether it’s leading an in-person session or writing an email or post to document the most valuable information, proactively sharing information will help your colleagues do better work while establishing you as a leader on your team.There’s no better place than a conference to take stock of the state of your industry and your profession. Make the most of your time, and have fun!

References:

 

 

Under the Hood: Demystifying Docker For Mac CE Edition

Estimated Reading Time: 6 minutes

 

Docker is a full development platform for creating containerized apps, and Docker for Mac is the most efficient way to start and run Docker on your MacBook. It runs on a LinuxKit VM and NOT on VirtualBox or VMware Fusion. It embeds a hypervisor (based on xhyve), a Linux distribution which runs on LinuxKit and filesystem & network sharing that is much more Mac native. It is a Mac native application, that you install in /Applications. At installation time, it creates symlinks in /usr/local/bin for docker & docker-compose and others, to the commands in the application bundle, in /Applications/Docker.app/Contents/Resources/bin.

One of the most amazing feature about Docker for Mac is “drag & Drop” the Mac application to /Applications to run Docker CLI and it just works flawlessly. The way the filesystem sharing maps OSX volumes seamlessly into Linux containers and remapping macOS UIDs into Linux is one of the most anticipated feature.

Few Notables Features of Docker for Mac:

  • Docker for Mac runs in a LinuxKit VM.
  • Docker for Mac uses HyperKit instead of Virtual Box. Hyperkit is a lightweight macOS virtualization solution built on top of Hypervisor.framework in macOS 10.10 Yosemite and higher.
  • Docker for Mac does not use docker-machine to provision its VM. The Docker Engine API is exposed on a socket available to the Mac host at /var/run/docker.sock. This is the default location Docker and Docker Compose clients use to connect to the Docker daemon, so you to use docker and docker-compose CLI commands on your Mac.
  • When you install Docker for Mac, machines created with Docker Machine are not affected.
  • There is no docker0 bridge on macOS. Because of the way networking is implemented in Docker for Mac, you cannot see a docker0 interface on the host. This interface is actually within the virtual machine.
  • Docker for Mac has now Multi-Architectural support. It provides binfmt_misc multi architecture support, so you can run containers for different Linux architectures, such as arm, mips, ppc64le, and even s390x.

Top 5 Exclusive Features of Docker For Mac That You Can’t Afford to Ignore

Under this blog, I will deep dive into Docker for Mac architecture and show how to access service containers running on top of LinuxKit VM.

At the base of architecture, we have hypervisor called Hyperkit which is derived from xhyve. The xhyve hypervisor is a port of bhyve to OS X. It is built on top of Hypervisor.framework in OS X 10.10 Yosemite and higher, runs entirely in userspace, and has no other dependencies. HyperKit is basically a toolkit for embedding hypervisor capabilities in your application. It includes a complete hypervisor optimized for lightweight virtual machines and container deployment. It is designed to be interfaced with higher-level components such as the VPNKit and DataKit.

Just sitting next to HyperKit is Filesystem sharing solution. The osxfs is a new shared file system solution, exclusive to Docker for Mac. osxfs provides a close-to-native user experience for bind mounting macOS file system trees into Docker containers. To this end, osxfs features a number of unique capabilities as well as differences from a classical Linux file system.On macOS Sierra and lower, the default file system is HFS+. On macOS High Sierra, the default file system is APFS.With the recent release, NFS Volume sharing has been enabled both for Swarm & Kubernetes.

There is one more important component sitting next to Hyperkit, rightly called as VPNKit. VPNKit is a part of HyperKit attempts to work nicely with VPN software by intercepting the VM traffic at the Ethernet level, parsing and understanding protocols like NTP, DNS, UDP, TCP and doing the “right thing” with respect to the host’s VPN configuration. VPNKit operates by reconstructing Ethernet traffic from the VM and translating it into the relevant socket API calls on OSX. This allows the host application to generate traffic without requiring low-level Ethernet bridging support.

On top of these open source components, we have LinuxKit VM which runs containerd and service containers which includes Docker Engine to run service containers. LinuxKit VM is built based on YAML file. The docker-for-mac.yml contains an example use of the open source components of Docker for Mac. The example has support for controlling dockerd from the host via vsudd and port forwarding with VPNKit. It requires HyperKit, VPNKit and a Docker client on the host to run.

Sitting next to Docker CE service containers, we have kubelet binaries running inside LinuxKit VM. If you are new to K8s, kubelet is an agent that runs on each node in the cluster. It makes sure that containers are running in a pod. It basically takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.On top of Kubelet, we have kubernetes services running. We can either run Swarm Cluster or Kubernetes Cluster. We can use the same Compose YAML file to bring up both the clusters side by side.

Peeping into LinuxKit VM

Curious about VM and how Docker for Mac CE Edition actually look like?

Below are the list of commands which you can leverage to get into LinuxKit VM and see kubernetes services up and running. Here you go..

How to enter into LinuxKit VM?

Open MacOS terminal and run the below command to enter into LinuxKit VM:

$screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Listing out the service containers:

Earlier the ctr tasks ls used to list the service containers running inside LinuxKit VM but in the recent release, namespace concept has been introduced, hence you might need to run the below command to list out the service containers:

$ ctr -n services.linuxkit tasks ls
TASK                    PID     STATUS
acpid                   854     RUNNING
diagnose                898     RUNNING
docker-ce               936     RUNNING
host-timesync-daemon    984     RUNNING
ntpd                    1025    RUNNING
trim-after-delete       1106    RUNNING
vpnkit-forwarder        1157    RUNNING
vsudd                   1198    RUNNING

How to display containerd version?

Under Docker for Mac 18.05 RC1, containerd version 1.0.1 is available as shown below:

linuxkit-025000000001:~# ctr version
Client:
  Version:  v1.0.1
  Revision: 9b55aab90508bd389d7654c4baf173a981477d55

Server:
  Version:  v1.0.1
  Revision: 9b55aab90508bd389d7654c4baf173a981477d55
linuxkit-025000000001:~#

How shall I enter into docker-ce service container using containerd?

ctr -n services.linuxkit tasks exec -t --exec-id 936 docker-ce sh
/ # docker version
Client:
 Version:      18.05.0-ce-rc1
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   33f00ce
 Built:        Thu Apr 26 00:58:14 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.05.0-ce-rc1
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.10.1
  Git commit:   33f00ce
  Built:        Thu Apr 26 01:06:49 2018
  OS/Arch:      linux/amd64
  Experimental: true
/ #

How to verify Kubernetes Single Node Cluster?

/ # kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-23T09:38:59Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
/ # kubectl get nodes
NAME                 STATUS    ROLES     AGE       VERSION
docker-for-desktop   Ready     master    26d       v1.9.6
/ #

 

Interested to read further? Check out my curated list of blog posts –

Docker for Mac is built with LinuxKit. How to access the LinuxKit VM
Top 5 Exclusive Features of Docker for Mac That you can’t afford to ignore
5 Minutes to Bootstrap Kubernetes Cluster on GKE using Docker for Mac 18.03.0
Context Switching Made Easy under Kubernetes powered Docker for Mac 18.02.0
2-minutes to Kubernetes Cluster on Docker for Mac 18.01 using Swarm CLI
Docker For Mac 1.13.0 brings support for macOS Sierra, now runs ARM & AARCH64 based Docker containers
Docker for Mac 18.03.0 now comes with NFS Volume Sharing Support for Kubernetes

 

Did you find this blog helpful?  Feel free to share your experience. Get in touch with me at twitter @ajeetsraina.

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

Top 5 Exclusive Features of Docker For Mac That You Can’t Afford to Ignore

Estimated Reading Time: 7 minutes

 

Docker for Mac 18.04.0 CE Edge Release went GA early last month. This was the first time Kubernetes version 1.9.6 & Docker Compose 1.21.0 was introduced under any Docker Desktop edition. ICYMI – Docker for Mac VM is entirely built with LinuxKit, hence this was the first release which enabled the RBD and CephFS kernel modules under LinuxKit VM. In case you’re new to RBD, the linux kernel RBD (rados block device) driver allows striping a linux block device over multiple distributed object store data objects. Usually the libceph module takes care of that.This release brought a number of fixes around upgrades from Docker for Mac 17.12, synchronisation between CLI `docker login` & GUI login, support for AUFS and much more.

Under this blog post, I will talk about top 5 exclusive and very useful features of Docker of Mac that you can’t afford to miss out.

#1: Docker for Mac support Docker Swarm, Swarm Mode & Kubernetes

Starting from Docker for Mac 17.12 CE Edge Release, Docker Inc introduced a standalone Kubernetes server and client, as well as Docker CLI integration. The Kubernetes server runs locally within your Docker instance, is not configurable, and is a single-node cluster.

One of the most anticipated feature introduced with this release was the Kubernetes server running within a Docker container on your local system. When Kubernetes support is enabled, you can deploy your workloads, in parallel, on Kubernetes, Swarm, and as standalone containers. Enabling or disabling the Kubernetes server does not affect your other workloads.

You can use Docker for Mac to test single-node features of swarm mode introduced with Docker Engine 17.12, including initializing a swarm with a single node, creating services, and scaling services. Docker “Moby” on Hyperkit serves as the single swarm node. You can also use Docker Machine, which comes with Docker for Mac, to create and experiment a multi-node swarm.

While testing Kubernetes, you may want to deploy some workloads in swarm mode. You can use the DOCKER_ORCHESTRATOR variable to override the default orchestrator for a given terminal session or a single Docker command. This variable can be unset (the default, in which case Kubernetes is the orchestrator) or set to swarm or kubernetes.

 

Check out my blog post:

2-minutes to Kubernetes Cluster on Docker for Mac 18.01 using Swarm CLI

 

#2:  You can use the same Docker Compose to build Swarm & Kubernetes Cluster

Yes, you read it correct. Starting from Docker for Mac 17.12, Docker introduced a new type called “Stack” under compose.docker.com. This object, that you can create with kubectl or more easily with docker stack deploy, contains the compose file.Behind the scene, a controller watches for stacks and create/update corresponding kubernetes objets (deployments, services, etc). The job of the controller is to reconcile the stacks (stored in the api-server or crd) with k8s native object.

The docker stack deploy manages to deploy to K8s. It convert docker-compose files to k8s manifests (something like kompose) before deployment. Let me showcase an example which shows how one can use the same YAML file to build Swarm Mode as well as K8s cluster.

Clone the Repository

$git clone https://github.com/ajeetraina/docker101

Change to the right location

$cd docker101/play-with-kubernetes/examples/stack-deploy-on-mac/

Example-1 : Demonstrating a Simple Web Application

Building the Web Application Stack

$docker stack deploy -c docker-stack1.yml myapp1

Verifying the Stack

$docker stack ls

Verifying using Kubectl

$kubectl get pods

Verifying if the web application is accessible

$curl localhost:8083

Cleaning up the Stack

$docker stack rm myapp`

Example:2 – Demonstrating ReplicaSet

Building the Web Application Stack

$docker stack deploy -c docker-stack2.yml myapp2

Verifying the Stack

$docker stack ls

Verifying using Kubectl

$kubectl get pods
Ajeets-MacBook-Air:testenviron ajeetraina$ kubectl get stacks
NAME      AGE
myapp2    22m
Ajeets-MacBook-Air:testenviron ajeetraina$ kubectl get pods
NAME                    READY     STATUS    RESTARTS   AGE
db1-d977d5f48-l6v9d     1/1       Running   0          22m
db1-d977d5f48-mpd25     1/1       Running   0          22m
web1-6886bb478f-s7mvz   1/1       Running   0          22m
web1-6886bb478f-wh824   1/1       Running   0          22m
Ajeets-MacBook-Air:testenviron ajeetraina$ kubectl get stacks myapp2 -o yaml
apiVersion: compose.docker.com/v1beta2
kind: Stack
metadata:
  creationTimestamp: 2018-01-28T02:55:28Z
  name: myapp2
  namespace: default
  resourceVersion: "3186"
  selfLink: /apis/compose.docker.com/v1beta2/namespaces/default/stacks/myapp2
  uid: b25bf776-03d6-11e8-8d4c-025000000001
spec:
  stack:
    Configs: {}
    Networks: {}
    Secrets: {}
    Services:
    ..
      WorkingDir: ""
    Volumes: {}
status:
  message: Stack is started
  phase: Available

Verifying if the web application is accessible

$curl localhost:8083

Cleaning up the Stack

$docker stack rm myapp2

An Interesting Read:

5 Minutes to Bootstrap Kubernetes Cluster on GKE using Docker for Mac 18.03.0

#3:  Docker for Mac provides Multi-Architecture Support

Docker for Mac provides binfmt_misc multi architecture support. This means that now you can run containers for different Linux architectures, such as arm, mips, ppc64le, and even s390x.

#4:  Support for NFS Volume sharing under Swarm as well as Kubernetes

With Docker for Mac 18.03 release, NFS Volume sharing support for both Swarm & Kubernetes was introduced. To demonstrate this feature, follow the below steps:

Pre-Requisite:

  • Install Docker for Mac 18.03 and future version
  • Enable Kubernetes under Preference Pane UI

Cloning the Repository

git clone https://github.com/ajeetraina/docker101/
cd docker101/for-mac/nfs

Execute the below script on your macOS system

sh env_vars.sh
sh setup_native_nfs_docker_osx.sh

 +-----------------------------+
 | Setup native NFS for Docker |
 +-----------------------------+

WARNING: This script will shut down running containers.

-n Do you wish to proceed? [y]:
y

== Stopping running docker containers...
== Resetting folder permissions...
Password:
== Setting up nfs...
== Restarting nfsd...
The nfsd service does not appear to be running.
Starting the nfsd service
== Restarting docker...

SUCCESS! Now go run your containers 🐳

Bringing up Your Application

docker stack deploy -c docker-compose.yml myapp2
 docker stack ls
NAME                SERVICES
myapp2                1
[Captains-Bay]🚩 >  kubectl get po
NAME      READY     STATUS    RESTARTS   AGE
web-0     1/1       Running   0          3m
[Captains-Bay]🚩 >  kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP     1d
web          ClusterIP   None         <none>        55555/TCP   3m
[Captains-Bay]🚩 >  kubectl describe po web-0
Name:           web-0
Namespace:      default
Node:           docker-for-desktop/192.168.65.3
Start Time:     Wed, 11 Apr 2018 23:00:18 +0530
Labels:         com.docker.service.id=up2u-web
                com.docker.service.name=web
                com.docker.stack.namespace=up2u
                controller-revision-hash=web-7dbbf8689d
                statefulset.kubernetes.io/pod-name=web-0
Annotations:    <none>
Status:         Running
IP:             10.1.0.34
Controlled By:  StatefulSet/web
Containers:
  web:
    Container ID:  docker://ec9ad2a3192bdeb0cc5028453310f40fd0ac3595021b070465c4e7725f626d63
    Image:         alpine:3.6
    Image ID:      docker-pullable://alpine@sha256:3d44fa76c2c83ed9296e4508b436ff583397cac0f4bad85c2b4ecc193ddb5106
    Port:          <none>
    Args:
      ping
      127.0.0.1
    State:          Running
      Started:      Wed, 11 Apr 2018 23:00:19 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      #{CONTAINER_DIR} from nfsmount (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-n8trf (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          True
  PodScheduled   True
Volumes:
  nfsmount:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  nfsmount-web-0
    ReadOnly:   false
  default-token-n8trf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-n8trf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason                 Age   From                         Message
  ----    ------                 ----  ----                         -------
  Normal  Scheduled              5m    default-scheduler            Successfully assigned web-0 to docker-for-desktop
  Normal  SuccessfulMountVolume  5m    kubelet, docker-for-desktop  MountVolume.SetUp succeeded for volume "pvc-bbdc7903-3dad-11e8-a612-025000000001"
  Normal  SuccessfulMountVolume  5m    kubelet, docker-for-desktop  MountVolume.SetUp succeeded for volume "default-token-n8trf"
  Normal  Pulled                 5m    kubelet, docker-for-desktop  Container image "alpine:3.6" already present on machine
  Normal  Created                5m    kubelet, docker-for-desktop  Created container
  Normal  Started                5m    kubelet, docker-for-desktop  Started container

 

 

#5:  Docker for Mac support context switching from docker-for-desktop to Cloud instances in a matter of a Click

Starting from Docker for Mac 18.02 RC release, the context switching feature was introduced which helped developers and operators to switch from docker-for-desktop to any Cloud environment in just a matter of a “toggle”.

 

I have a detailed blog post published early this year which demonstrates this feature with crystal clear examples. Check it out.

Context Switching Made Easy under Kubernetes powered Docker for Mac 18.02.0

Other attractive features of Docker for Mac 18.04 includes –

  • Docker for Mac VM is entirely built with LinuxKit

How to enter into LinuxKit VM?

$screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
/ # cat /etc/issue

Welcome to LinuxKit

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
          {                       /  ===-
           \______ O           __/
             \    \         __/
              \____\_______/
/ # cat /etc/os-release
PRETTY_NAME="Docker for Mac"
/ #
linuxkit-025000000001:~# cat /etc/os-release
PRETTY_NAME="Docker for Mac"
linuxkit-025000000001:~# runc list
ID                 PID         STATUS      BUNDLE                                CREATED                          OWNER
000-metadata       0           stopped     /containers/onboot/000-metadata       2018-05-05T06:27:44.345735031Z   root
001-sysfs          0           stopped     /containers/onboot/001-sysfs          2018-05-05T06:27:44.768313965Z   root
002-binfmt         0           stopped     /containers/onboot/002-binfmt         2018-05-05T06:27:45.630283593Z   root
003-format         0           stopped     /containers/onboot/003-format         2018-05-05T06:27:46.341011253Z   root
004-extend         0           stopped     /containers/onboot/004-extend         2018-05-05T06:27:47.08889973Z    root
005-mount          0           stopped     /containers/onboot/005-mount          2018-05-05T06:27:55.334088074Z   root
006-swap           0           stopped     /containers/onboot/006-swap           2018-05-05T06:27:56.486815308Z   root
007-ip             0           stopped     /containers/onboot/007-ip             2018-05-05T06:28:03.894591249Z   root
008-move-logs      0           stopped     /containers/onboot/008-move-logs      2018-05-05T06:28:05.980232896Z   root
009-sysctl         0           stopped     /containers/onboot/009-sysctl         2018-05-05T06:28:06.15775421Z    root
010-mount-vpnkit   0           stopped     /containers/onboot/010-mount-vpnkit   2018-05-05T06:28:06.356833391Z   root
011-bridge         0           stopped     /containers/onboot/011-bridge         2018-05-05T06:28:06.551619273Z   root
linuxkit-025000000001:~# ctr tasks ls
  • Docker for Mac uses raw format VM disks for systems running APFS on SSD on High Sierra by default
  • DNS name docker.for.mac.host.internal should be used instead of docker.for.mac.localhost (still valid) for host resolution from containers.

Did you find this blog helpful?  Feel free to share your experience. Get in touch with me at twitter @ajeetsraina.

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

 

Demystifying Serverless & OpenFaas

Estimated Reading Time: 9 minutes

Is Function a new container? Why so buzz around Serverless computing like OpenFaas?

 

One of the biggest tech trend of 2018 has been a year of “shipping code and moving quickly”. It has been a year for developers not to think about infrastructure, not to worry about scaling, load-balancing etc. What developer need is shipping code, running code and moving quickly without any infrastructure concern.

If you look back 20 years from now, the first wave of computing started with x86 servers inside the datacenter. It was the very first wave of computing which democratized computing by becoming more and more accessible and that’s when the actual IT revolution started. The small and medium businesses(SMBs) could afford to run servers in their environments. The second wave of computing started with virtualization and the credit goes to VMware who actually introduced a very affordable hypervisor and then made it possible to run virtualization or VMs so one server with multiple VMs. The third & more recent wave is containerization which brought a new mechanism of packaging and deploying applications, paving way to some of the emerging patterns like microservices.

The most recent & very exciting wave of computing is serverless computing and this is a paradigm where we never have to deal with infrastructure. Though the name “serverless” is used but still it sounds “misnomer” as serverless computing still requires server. The name “serverless computing” is used because the server management and capacity planning decisions are completely hidden from the developer or operator. Serverless code can be used in conjunction with code deployed in traditional styles, such as microservices. Alternatively, applications can be written to be purely serverless and use no provisioned servers at all.

The whole idea about Serverless is even though we still have servers but we don’t really need to care about them anymore. Serverless means that you are not going to have server that you are going to manage. You don’t have anything running. Your code is lying around there. When somebody calls that particular code, then it gets triggered and will be executed. You write functions and that functions can be executed at that instance when there is something coming and triggering.

What is Function-as-a-Service(FaaS)?

Under serverless, we don’t really take an entire application and deploy like the way you do under PaaS but we take tiny code snippets that are written as stateless functions and deploy them in this environment So instead of packaging a large application, you actually break the application into one function at a time. You basically deploy them in this environment which is serverless computing, hence expected to write one function, test it and then upload it to serverless computing platform. After you have uploaded multiple number of independent isolated autonomous functions, you decide how you are going to connect all of them to compose an application. Sounds like microservices? Well that’s what it is. It can be rightly called as “Nano-Services” which is much more stateless than microservices

Is Function a Special Type of Container?

Function is small bits of code that can do one thing well & are easy to understand and maintain. These functions are then deployed as a single unit as code into your FaaS platform and these platform then deal with provisioning your infra, scaling, reliability, billing and security in any language which you care.

Function can be rightly called as a special type of container which has property like –

  • Short Running
  • Ephemeral
  • Stateless
  • Invoked
  • Single Purpose
  • Self-Contained

Introducing OpenFaas

OpenFaas is a framework for building Serverless applications. Functions in OpenFaas are actually Docker containers. It is a matured open source serverless software program that can run any CLI-driven binary program embedded in a Docker container. As an Open Source project, it has gained large-scale adoption within the community. OpenFaaS is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics.

OpenFaaS is an independent project created by Alex Ellis which is now being built and shaped by a growing community of contributors.

What does OpenFaas provides?

OpenFaaS provides :

  • Easy install (1minute!) – Ease of use through UI portal and one-click install
  • Multiple language support – Write functions in any language for Linux or Windows and package in Docker/OCI image format
  • Transparent autoscaling – Auto-scales as demand increases
  • No infrastructure headaches/concerns
  • Support for Docker Swarm and Kubernetes
  • Console UI to enable deployment, invocation of functions
  • Integrated with Prometheus Alerts
  • Async support
  • Market Serverless function repository
  • RESTful API
  • CLI tools to build & deploy functions to the cluster.

OpenFaaS is a framework for building serverless functions with Docker and Kubernetes which has first class support for metrics.

OpenFaas Stack – Under the Hood

OpenFaas stack consists of the following components:-

  • API Gateway
  • Function Watchdog
  • Prometheus

The API gateway and Prometheus instance runs as services while the function watchdog runs as function container. The below picture depicts API gateway and Prometheus stack under docker-compose.yml leveraged by OpenFaas deploy-stack.yml file.

 

 

Let us dive into each components separately:

API Gateway:

API Gateway is where you define all of your functions. The API Gateway is a RESTful microservice. It provides an external route into your functions and collects Cloud Native metrics through Prometheus. Your API Gateway will scale functions according to demand by altering the service replica count in the Docker Swarm or Kubernetes API. A UI is baked in allowing you to invoke functions in your browser and create new ones as needed.

 

                      Figure:2 – Swarm overlay connecting API gateway to Function containers

 

Function Watchdog ( fwatchdog)

Function Watchdog is embedded in every container and that allow every container to become serverless. It is a tiny Golang webserver. It provides an unmanaged and generic interface between the outside world and your function. Its job is to organize a HTTP request accepted on the API Gateway and to invoke your chosen application.

Each function container consists of a function watchdog, fwatchdog & certain function program written in any language(Python/Go/CSharp). The Function Watchdog is the entrypoint allowing HTTP requests to be forwarded to the target process via STDIN. The response is sent back to the caller by writing to STDOUT from your application.

It is important to note that the Dockerfile describing a function container must have the fprocess environment variable pointing to the function program name and arguments.

Reference: https://github.com/openfaas/faas/tree/master/watchdog

Prometheus:

Prometheus is an open-source systems monitoring and alerting toolkit used in this stack . It underpins the complete stack and collects statistic to build dashboard, certain metrics interpolation. Whenever certain functions have high traffic, it will scale out for you automatically using Docker Swarm and K8s API.

Orchestration Engines(Swarm & Kubernetes) & Docker Platform

All the above components run on top of Docker Swarm or Kubernetes orchestration engine The container runtime can be any modern version of Docker or containerd.

Want to test Drive OpenFaas?

Under this demo, I will showcase how to get started with OpenFaas in detailed manner. We will leverage PWD platform which is the fastest method to setup 5-node Swarm cluster. In case you are completely new to play-with-docker, you can follow the below step by step instructions –

  • Open https://labs.play-with-docker.com/
  • Click on login & start
  • Click on tool near the setting on the left side of PWD interface

  • Choose 3 Managers and 2 workers and allow it to bring up 5 node cluster

Setting up Visualizer tool:

https://github.com/ajeetraina/openfaas
cd openfaas/visualizer/
docker-compose up -d
$ docker ps
CONTAINER ID        IMAGE                             COMMAND             CREATED             STATUS              PORTS                    NAMES05b89b6b8aa9        dockersamples/visualizer:stable   "npm start"         56 seconds ago      U
p 55 seconds       0.0.0.0:8085->8080/tcp   visualizer_visualizer_1

This will setup a visualizer tool under port 8085:

As of now, there is no service up and running on Swarm and hence it will show up blank.

 

Cloning the OpenFaas Repository

git clone https://github.com/openfaas/faas
cd faas
./deploy_stack.sh

This single script will bring up complete OpenFaas stack.

Looks cool?

Click on port 8080 which appear on the top of PWD screen and it will redirect you to OpenFaas UI page as shown below:

 

As directed by UI page, let us head over to CLI and get faas-cli installed:

curl -sSL https://cli.openfaas.com | sudo sh
$ faas-cli list
Function                        Invocations     Replicas
func_wordcount                  0               1func_hubstats                   0               1
func_base64                     0               1func_echoit                     0               1
func_markdown                   0               1tcpdump                         8               1
func_nodeinfo                   1               1[manager1] (local) root@192.168.0.151 ~
$

OpenFaas by default comes with few already baked in functions like as shown below:

All you need is to choose a function(like func_nodeinfo) and click on “Invoke” on the right hand side, it will display node information.

Building Your First Serverless Function

Before we end up this blog, let us try building a Serverless function called “retweet-bot”. This function retweets all Tweets containing your search term. Let’s make this happen –

Clone this repository

In case you missed it out, clone the repository –

https://github.com/ajeetraina/openfaas

Writing Retweet Function

mkdir -p ~/retweet && \
  cd ~/retweet

The “Retweet” Python function using the CLI:

root@ubuntu18:~/retweet# faas-cli new --lang python retweet
Folder: retweet created.
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|


Function created in folder: retweet
Stack file written: retweet.yml
root@ubuntu18:~/retweet#

This creates 3 major files, 2 of them under retweet directory and 1 YAML file as shown:

retweet# tree
.
├── retweet.py
└── requirements.txt

0 directories, 2 files

Replace handler.py & requirements.txt with these above files and add config from the repository under the same location.

Displaying contents of retweet.yml

root@ubuntu18:~/retweet# cat retweet.yml
provider:
  name: faas
  gateway: http://127.0.0.1:8080

functions:
  retweet:
    lang: python
    handler: ./retweet
    image: retweet

Building the Function

cd ..
faas-cli build -f ./retweet.yml

Verifying the Image


docker images | grep retweet
ajeetraina/retweet          latest              027557a5185d        About a minute ago   83MB

Deploying the Retweet Function

faas-cli deploy -f ./retweet.yml
Deploying: retweet.

Deployed. 200 OK.
URL: http://127.0.0.1:8080/function/retweet

Now open up localhost:8080/ui and watch out for brand new retweet function. Clik on Invoke and there you find retweet bot active to display you hashtags.

 

 

Hurray ! We have built our first retweet Serverless function in just 2 minutes.

If you want to learn more about OpenFaas, head over to https://docs.openfaas.com.

Reference:

Introducing OpenUSM – Simplifying Server Management & Insight Log Analytics using Docker containers

Estimated Reading Time: 8 minutes

Its 2018 ! Let Containers Manage Your Datacenter..

Containers are changing the dynamics of modern data center. It is a growing technology that is drawing widespread attention across the Enterprise IT. One of the primary reason of the rise & adoption of containers is that it allows developers to move faster. Compared to VMs which takes minutes to stand up, containers take milliseconds and even microseconds. As organizations prioritize shipping new products and features faster, to keep up in the software-eaten world, developers are favoring technology that allows them to scale applications and deploy resources much faster than what traditional VMs on public and private clouds can support.

Docker containers bring variety of cost and performance benefits. It brings ability to run multiple applications on the same server or OS without a hypervisor which eliminates the drag of the hypervisor on system resources, which means that your workloads have a lighter footprint – the container footprint is zero, because it’s simply a boundary of permissions and resources within Linux. They fire up and decommission very rapidly compared to virtual machines – a perfect fit for the ephemeral nature of today’s short-lived workloads, which are often tied to real world events.

Docker is an open platform which make application and workloads more portable and distributed in an effective and standardized way. Combining Docker containers, micro services along with software-defined infrastructure makes the datacenter more agile and quick resource reallocation. Hence, this architecture works well to improve the datacenter operations.

Most DellEMC Server Management software offerings, as well as the entire Software Defined Infrastructure, are built upon standard implementation using RESTful architecture called Redfish. Redfish is a next generation system management standard using a data model representation inside a hypermedia RESTful interface. The data model is defined in terms of a standard, machine-readable schema, with the payload of the messages expressed in JSON and the protocol using OData v4. Since it is a hypermedia API, Redfish is capable of representing a variety of implementations by using a consistent interface. It has mechanisms for discovering and managing data center resources, handling events, and managing long-lived tasks. It is easy to implement, easy to consume and offer scalability advantages over old technologies. Redfish is a RESTful interface over HTTPS in JSON format based on ODATA v4 usable by clients, scripts, and browser-based GUIs

 

What is OpenUSM?

 

 

OpenUSM stands for Open Universal Systems Manager. It is basically a multi-tool product like “a Swiss army knife”. It is a suite of open source tools & scripts which purely uses containers & related tools to perform server management tasks, monitoring & insight Log Analytics. It is 100% container based solution which heavily uses Docker for building microservices for system management tasks like BIOS token change, Firmware update. It is completely an out-of-band system management solution purely based on Redfish API Interface. It is a platform agnostic solution (can be run from laptop, server or cloud) and works on any of Linux or Windows platform with Docker Engine running on top of it.

OpenUSM is a project which was created by Ajeet Singh Raina & Avinash Bendigeri, a DellEMC employee 3-months back.

 

Value Proposition:

OpenUSM follow an easy deployment model. It uses developer’s tools like Docker Compose & CLI to bring up microservices which ensures that system management tasks can be achieved flawlessly. It uses modern tools & technologies and integrates well with near real-time search analytics tools like ELK stack. It enables Sensor Log Analytics & visualization for operations team using the popular Grafana tool. It can scale both vertically & horizontally. As it is completely open source, you are free to build and customize based on your needs and holds a plug-and-play components and functionalities.

Technology Overview of OpenUSM:

OpenUSM is an integrated container solution which brings 3 basic functionalities– system management, monitoring and insight log analytics. System Management stack sits at the middle of the architecture which uses Python, Redfish, Django, Docker & Docker compose to enable system management tasks. On the left hand side of the architecture, we have monitoring stack which uses Prometheus, Grafana, Alert-Manager and Pushgateway to retrieve GPU and Sensor metrics. On the right hand side, it consists of open source version of Elasticsearch, Logstash & Kibana for sensors, lifecycle controller & SEL logs.

Understanding OpenUSM System Management WorkFlow

OpenUSM uses “Container-Per-Server (CPS)” model. For each server management tasks, there are scripts which when executed builds and run Docker containers against each of server platform. It purely uses Redfish API to communicate directly with Dell iDRAC, collects iDRAC/LC logs and pushes it to syslog server. Logstash collects the syslog server and pushes to elasticsearch and finally it gets visualized through Kibana. OpenUSM uses Prometheus Stack for monitoring System components like GPU/CPU monitoring using nvidia-docker & node exporter.

 

Under this blog post, I am going to demonstrate how OpenUSM simplifies the overall system management tasks with the help of Docker containers & Redfish. For this demonstration, I will leverage Ubuntu 17.10 VM running on my ESXi 6.0 system. This code should work on any of Linux & Windows platform too.

The source code for this project is open to the public and is available under https://github.com/openusm/openusm

 

Cloning the Repository

git clone https://github.com/openusm/opensum

Bootstrapping Docker

If you have Docker already installed on your system, you can skip to next step. If not, run the below command to install Docker & Docker Compose on your system.

sh bootstrap.sh install_docker

Based on your network connectivity, this step will take 1-2 minutes to complete.

Bootstrapping ELK

OpenUSM is 100% containerized solution and hence we will be running ELK inside Docker containers. To keep it simple, we designed a docker-compose file which can get you started in a matter of seconds.

Execute the same bootstrap file to bring up ELK stack as shown below:

sh bootstrap.sh provision_elk

Just wait for 30-40 seconds to get ELK stack up and running.

Verifying the ELK services

Run the below command to check if ELK services are up and running:

sh bootstrap.sh logs

Pushing DellEMC iDRAC Logs to ELK Stack

Under this section, I am going to demonstrate how OpenUSM makes it so easy to push logs of DellEMC system management tasks to a centralized  ELK stack and get it visualized via Kibana . Let us pick up a simple “BIOS Token Change” functionality and apply it for multitude of DellEMC servers.

To keep it simple, we designed a script named “bios-token.py” which is placed under the root of OpenUSM GIT repository. Let us first look at various parameters which can be supplied with bios-token.py script –

python bios-token.py --help
usage: bios-token.py [-h] [--verbose] [-i IDRAC] [-n NFS] [-s SHARE]
[-c CONFIG] [-f IPS]
 
Welcome to Universal Systems Manager Bios Token Change
 
optional arguments:
-h, --help            show this help message and exit
--verbose             Turn on verbose logging
-i IDRAC, --idrac IDRAC
iDRAC IP of the Host
-n NFS, --nfs NFS     NFS server IP address
-s SHARE, --share SHARE
NFS Share folder
-c CONFIG, --config CONFIG
XML File to be imported
-f IPS, --ips IPS     IP files to be updated

As shown above, the script is targeted both at a single Dell server as well as multitude of DellEMC servers via a plain text file. We are currently looking at Autodiscovery feature to automate this functionality. One need to provide NFS server IP, share name and BIOS token configuration files as argument to execute it successfully. Once this script is invoked, it creates as many number of Docker containers per DellEMC servers, collects iDRAC logs from each of servers and pushes it to syslog server which runs inside Docker container. Logstash collects it from syslog and dumps it into Elasticsearch to get it visualized under Kibana UI.

bios-token.py -f ips.txt -s /var/nfsshare -c biosconfig.xml -n <NFS server-IP>

where,

/var/nfsshare => NFS share

ips.txt => list of DellEMC iDRAC IP

biosconfig.xml => XML definining the BIOS tokens entry

By now, you should be able to see iDRAC logs visualized under Kibana UI. We can perform ample amount of customization around Kibana UI to display the logs per server basis.

 

Insight Log Analytics (LC, SEL & Sensor Logs)

Unpacking OpenUSM secrets further, this marks as an interesting use case and robust capability around insight Log Analytics. DellEMC server generates varieties of logs like system event logs (SEL), RAID controller logs & Lifecycle controller (LC) logs. When a system event occurs on a managed system, it is recorded in the System Event Log (SEL). The SEL page displays a system health indicator, a time stamp, and a description for each event logged. The same SEL entry is also available in the Lifecycle Controller (LC) log.

Considering a certain use cases where datacenter administrator want to collect LC logs for the last 1 year, it definitely requires a robust and modern tool to collect such huge data and perform analysis on top of the specific software.

We recently designed a script which simplifies such log analytics capability using Docker, Redfish & ELK. You can find the “lclogexporter.py” script under the root of GITHUB repository:

python lclogexporter.py -f ips.txt -ei <ip-address of ELK> -eu elastic -ep changeme

 

This script requires elasticsearch IP address, credentials & list of iDRAC IPs to get all iDRAC LC logs pushed to ELK stack. Whenever you execute this script, it will build a Docker image called “openusm-analytics” first and run this container which automatically pushes all LC, SEL and Sensors logs to ELK stack.

Below is the Kibana UI visualizing Pie-Chart for Dell Lifecycle Controller logs collected for the last 1 year timeframe.

 

Insight Log Metrics (Sensor Logs) using Grafana

 

Did you find OpenUSM interesting? OpenUSM is just 3 months old project and we are looking out for contributors across the globe. If you think the project really looks cool, come & join us to make it more robust. We welcome your participation..

Under the Hood: Demystifying Docker Enterprise Edition 2.0 Architecture

Estimated Reading Time: 8 minutes

The Only Kubernetes Solution for Multi-Linux, Multi-OS and Multi-Cloud Deployments is right here…

Docker Enterprise Edition(EE) 2.0 Final GA release is available. One of the most promising feature announced with this release includes Kubernetes integration as an optional orchestration solution, running side-by-side with Docker Swarm. Not only this, this release includes Swarm Layer 7 routing improvements, Registry image mirroring, Kubernetes integration to Docker Trusted Registry & Kubernetes integration to Docker EE access controls. With this new release, organizations will be able to deploy applications with either Swarm or fully-conformant Kubernetes while maintaining the consistent developer-to-IT workflow.

Docker EE is more than just a container orchestration solution; it is a full lifecycle management solution for the modernization of traditional applications and microservices across a broad set of infrastructure platforms. It is a Containers-as-a-Service(CaaS) platform for IT that manages and secures diverse applications across disparate infrastructure, both on-premises and in the cloud. Docker EE provides an integrated, tested and certified platform for apps running on enterprise Linux or Windows operating systems and Cloud providers. It is tightly integrated to the underlying infrastructure to provide a native, easy to install experience and an optimized Docker environment.

Docker EE 2.0 GA consists of 3 major components which together enable a full software supply chain, from image creation, to secure image storage, to secure image deployment.

  • Universal Control Plane 3.0.0 (application and cluster management) – Deploys applications from images, by managing orchestrators, like Kubernetes and Swarm. UCP is designed for high availability (HA). You can join multiple UCP manager nodes to the cluster, and if one manager node fails, another takes its place automatically without impact to the cluster.
  • Docker Trusted Registry 2.5.0 – The production-grade image storage solution from Docker &
  • EE Engine 17.06.2- The commercially supported Docker engine for creating images and running them in Docker containers.

Tell me more about Kubernetes Support..

Kubernetes in Docker EE fully supports all Docker EE features, including role-based access control, LDAP/AD integration, scanning, signing enforcement, and security policies.

Kubernetes features on Docker EE include:

  • Kubernetes orchestration full feature set
  • CNCF Certified Kubernetes conformance
  • Kubernetes app deployment by using web UI or CLI
  • Compose stack deployment for Swarm and Kubernetes apps
  • Role-based access control for Kubernetes workloads
  • Pod-based autoscaling, to increase and decrease pod count based on CPU usage
  • Blue-Green deployments, for load balancing to different app versions
  • Ingress Controllers with Kubernetes L7 routing
  • Interoperability between Swarm and Kubernetes workloads for networking and storage.

But wait.. What type of workload shall I run on K8s and what on Swarm?

One of the question raised multiple times in last couple of Docker Meetup was – Good that now we have K8s and Swarm as multi-orchestrator under the single Enterprise Engine. But what type of workload shall I run on Kubernetes and what on Docker Swarm?

 

As shown above, even though there is a bigger overlap in terms of Microservices, Docker Inc. recommends specific types of workloads for both Swarm and Kubernetes.
Kubernetes works really great for large-scale workloads. It is designed to address some of the difficulties that are inherent in managing large-scale containerized environments. Example: blue-green deployments on Cloud Platform, highly resilient infrastructure with zero downtime deployment capabilities, automatic rollback, scaling, and self-healing of containers (which consists of auto-placement, auto-restart, auto-replication , and scaling of containers on the basis of CPU usage). Hence, it can be used in a variety of scenarios, from simple ones like running WordPress instances on Kubernetes, to scaling Jenkins machines, to secure deployments with thousands of nodes.

Docker Swarm, on the other hand, do provide most of the functionality which Kubernetes provides today. With notable features like easy & fast to install & configure, quick container deployment and scaling even in very large clusters, high availability through container replication and service redundancy, automated internal load balancing, process scheduling, Automatically configured TLS authentication and container networking, service discovery, routing mesh etc. Docker Swarm is best suited for MTA program – modernizing your traditional legacy application(.Net app running on Windows Server 2016)  by containerization on Docker Enterprise Edition. By doing this, you reduce the total resource requirements to run your application. This increases operational efficiency and allows you to consolidate your infrastructure.

Under this blog post, I will deep dive into Docker EE Architecture covering its individual components, how they work together and how the platform manages containers.

Let us begin with its architectural diagram –

 

At the base of the architecture, we have Docker Enterprise Engine. All the nodes in the cluster runs Docker Enterprise Edition as the base engine. Currently the stable release is 17.06 EE. It is a lightweight and powerful containerization technology combined with a work flow for building and containerizing your applications.

Sitting on top of the Docker Engine is what we call “Enterprise Edition Stack” where all of these components run as containers(shown below in the picture), except Swarm Mode. Swarm Mode is integrated into Docker Engine and you can turn it on/off based on your choice.Swarm Mode is enabled when Universal Control Plane (UCP) is up & running.

 

UCP depicting ucp-etcd, ucp-hyperkube & ucp-agent containers

In case you’re new to UCP, Docker Universal Control Plane is a solution designed to deploy, manage and scale Dockerized applications and infrastructure. As a Docker native solution, full support for the Docker API ensures seamless transition of your app from development to test to production – no code changes required, support for the Docker tool-set and compatibility with the Docker ecosystem.

 

 

From infrastructure clustering and container scheduling with the embedded Docker Swarm Mode, Kubernetes to multi-container application definition with Docker Compose and image management with Trusted Registry, Universal Control Plane simplifies the process of managing infrastructure, deploying, managing applications and monitoring their activity, with a mix of graphical dashboards and Docker command line driven user interface

There are 3 orchestrators sitting on top of Docker Enterprise Engine – Docker Swarm Mode, Classic Swarm & Kubernetes. For both Classic Swarm and Kubernetes, there is the same underlying etcd instance which is highly available based on setup you have in your cluster.

Docker EE provides access to the full API sets of three popular orchestrators:

  • Kubernetes: Full YAML object support
  • SwarmKit: Service-centric, Compose file version 3
  • “Classic” Swarm: Container-centric, Compose file version 2

Docker EE proxies the underlying API of each orchestrator, giving you access to all of the capabilities of each orchestrator, along with the benefits of Docker EE, like role-based access control and Docker Content Trust.

To manage lifecycle of orchestrator, we have a component called Agents and reconciler which manages both the promotion and demotion flows as well as certification renewal rotation and deal with patching and upgrading.

Agent is the core component of UCP and is a globally-scheduled service. When you install UCP on a node, or join a node to a swarm that’s being managed by UCP, the agent service starts running on that node. Once this service is running, it deploys containers with other UCP components, and it ensures they keep running. The UCP components that are deployed on a node depend on whether the node is a manager or a worker. In nutshell, it monitors the node and ensures the right UCP services are running.

Another important component is Reconciler. When UCP agent detects that the node is not running the right UCP components, it starts the reconciler container to converge the node to its desired state. It is expected for the UCP reconciler container to remain in an exited state when the node is healthy.

In order to integrate with K8s and support authentication, Docker Team built open ID connect provider. In case you’re completely new to OIDC, OpenID Connect is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an inter-operable and REST-like manner.

As Swarm Mode is designed, we have central certificate authority that manages all the certificates and cryptographic node identity of the cluster. All component uses mutual TLS communication and they are using nodes which are issue by certificate authority. We are using unmodified version of Kubernetes.

Sitting on the top of the stack, there is GUI which uses Swarm APIs and Kubernetes APIs. Right next to GUI, we have Docker Trusted Registry which is an enterprise-ready on-premises service for storing, distributing and securing images. It gives enterprises the ability to ensure secure collaboration between developers and sysadmins to build, ship and run applications. Finally we have Docker & Kubernetes CLI capability integrated. Please note that this uses un-modified version of Kubernetes API.

Try Free Hosted Trial

If you’re interested in a fully conformant Kubernetes environment that is ready for the enterprise,    https://trial.docker.com/

Did you find this blog helpful? Feel free to share your experience. Get in touch with me on Twitter –  @ajeetsraina

Docker’s Birthday Celebration in Bangalore – The Fifth Kind

Estimated Reading Time: 5 minutes

“As you grow up, make sure you have more networking opportunities than chances, and more collaborative approach than just an acquaintance. Reason  ~ “Sharing is Caring”.

Docker’s Birthday Celebration is not just about cakes, food and party. It’s actually a global tradition that is near and dear to our heart because it gives each one of us an opportunity to express our gratitude to our community of contributors, customers, partners and users. The goal of this global celebration is to welcome those new to Docker and bring them up to speed through hands-on labs with the help of more advanced users who act as mentors on-site.

 

This year, celebrations all over the world took place March 19-25, 2018. Yesterday we celebrated Docker’s 5th Birthday in Bangalore at Altimetrik Office, Electronic City. This time I was lucky enough to get chance to lead this event.

I started planning for this event in the first week of March. Started interlocking with  Community Marketing Co-ordinator, Docker Inc. and ordered for Docker Mentor T-shirts, birthday stickers & Docker crafted Balloons. It just took 1 week time to get it shipped to Bangalore address. Quite quick, isn’t it? A Special thanks to Lisa McNikol for helping me with these stuffs. Next, Neependra Khare, Founder of CloudYuga & Docker Bangalore Meetup organizer initiated a thread for looking out Mentors for the event. Around 10+ community members turned up to mentor the participants for lab sessions. I started interlocking with them for the lab preparation. A BIG SHOUT OUT to each and every mentors (names listed towards the end) as I find them quite interested and were very quick in completing the testing with the lab sessions. Few of the issues reported were fixed and handled smoothly.

Next, it was time to work with the organizers at Altimetric India. A special thanks to Sonali Baluni, Altimetrik Employee for multiple interactions on phone and emails for planning this event. Over 30+ balloons, 5kg Docker Birthday Cake, goodies for organizer and participants, smoothening WiFi connectivity & Backups were planned for the day.

Over 200+ participants signed up while 50+ turned up for this event. This was expected as we just completed Joint Meetup with 8 vibrant groups last weekend in Walmart Office. I reached venue at around 8:30 AM and was stoked to find wonderful ambience in place. Worked with organizers to arrange Docker T-shirts and Balloons for the participants, distributed Mentors T-Shirts so that they can get ready for the celebration.

I started the Birthday celebration with an attention grabber –

 

Talked about the agenda for the day which included more networking, mentorship & lab practice as shown –

Next, I asked each of Docker Mentors to introduce themselves to the group. Mentors are group of experts targeted to help the participants with the lab sessions.

 

Most of the participants followed the instructions on their own and where ever needed, mentors helped the participants. There were around 5 Lab sessions categorised under Docker 101, MTA with Java and .Net applications, Kubernetes and Hybrid OS applications. You can find it under https://github.com/dockersamples/docker-fifth-birthday

 

At 11:30 sharp, we did a cake cutting. We gave chance to youngest member in the group to do this honour.

This time I focused on networking and spent considerable amount of time interlocking with the participants. I met with new Community members and tried to understand how they are using Docker. Few of interesting questions from the community were –

  1. What is Serverless?
  2. How to get started with OpenFaas?
  3. How to become Docker Mentor?
  4. Why we switched to Kubernetes? Issues around Docker EE UCP.
  5. How to containerise 20-years old Monolithic Legacy Application inside my organization?
  6. What is RexRay & its architecture?
  7. When can we expect K8s under Docker CE on Linux Platform?
  8. A Query around AWX – An Alternative to RedHat’s Ansible Tower?
  9. When shall I use Kubernetes & when to use Swarm?

I am planning to come up with blog post around most of the queries in the near future.

Thank You –

  • Mentors (Suraj Narwade, Yogesh Kad, Mamta Jha, Shanthi Kumar, Jibin George, Shankar, Kumara K, Vishan Ghule, Y Ramesh Naik, Savitha Ashture, Saurav Kumar, Swapnasagar Pradhan – For effective Mentorship
  • Mano Marks, Elton & Marcos Nills – For the robust PWD Platform for Lab workshop
  • Sonali Baluni & Team, Organizer ( Altimetrik India)
  • Lisa McNikol – Community Marketing Co-ordinator, Docker Inc.
  • Jenny Burcio – Sr. Program Manager, Developer Relations and Advocacy, Docker Inc.
  • All Participants

What I liked most about this Meetup is the energy and curiosity that the participants have. It was a blast discussing with, answering and questioning all the keen minds that make up this community.

Dockercon 2018 is happening this June 12-15. I have plan to attend this event.  Would be completely excited to meet more members of the Docker community over at DockerCon in June and hope to bring home interesting knowledge and tips & tricks for the larger Docker Bangalore Meetup to gain from.

Thanks to all you there who participated to make this event a blast !