How to Deploy Apache Kafka on AWS Platform using Docker Swarm Mode?

Estimated Reading Time: 7 minutes

I am thrilled and excited to start a new open source project called “Pico”. Pico is a beta project which is targeted at object detection and analytics using Apache Kafka, Docker, Raspberry Pi & AWS Rekognition Service. The whole idea of Pico project is to simplify object detection and analytics process using few bunch of Docker containers. A cluster of Raspberry Pi nodes installed at various location points are coupled with camera modules and sensors with motion detection activated on them. Docker containers running on these Raspberry Pis are able to convert these nodes into CCTV camera. After producing images of all these cameras, the real-time data are then consumed on any of the five containers because of the replication factor of Kafka. The camera captured video streams and processed by Apache Kafka. The data is consumed inside a different container which runs on all of these nodes. AWS Rekognition analyses the real time video data & searches object on screen against a collection of objects.

In my past blog, I already demonstrated how to convert Raspberry Pi into CCTV camera using Docker container. The next target was to understand what is Apache Kafka and how to implement it on AWS Cloud so that the real time data can be sent to AWS Rekognition Service. I spent considerable amount of time understanding the basics of Apache Kafka before I jump directly into Docker Compose to containerize the various services which falls under this piece of software stack.

Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation. It is written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Apache Kafka is a distributed, partitioned, and replicated publish-subscribe messaging system that is used to send high volumes of data, in the form of messages, from one point to another. It replicates these messages across a cluster of servers in order to prevent data loss and allows both online and offline message consumption. This in turn shows the fault-tolerant behaviour of Kafka in the presence of machine failures that also supports low latency message delivery. In a broader sense, Kafka is considered as a unified platform which guarantees zero data loss and handles real-time data feeds.

Architecture of Apache Kafka

The overall architecture of Kafka is shown below.

It is composed of three server machines which together act as a cluster computing platform. In a typical Kafka cluster, each server is configured to behave as a single broker system that shows the persistence and replication of message data. In other words, we can say that there is more than one broker in a typical Kafka cluster. Essentially, broker is the key component of Kafka cluster which is basically responsible for maintaining published data. Each broker instance can easily handle thousands of reads and writes per topic, as they have a stateless behavior.



At a basic level, Kafka broker uses topics to handle message data. The topic is first created and then divided into multiple partitions in order to balance load. The above diagram illustrates the basic concept of topic which is divided into three partitions. Each partition has multiple offsets in which messages are stored. As an example, suppose that the topic has a replication factor of value β€˜3’, then Kafka will create three identical replicas of each partition regarding the topic and distribute them across the cluster. In order to balance load and maintaining data replication, each broker stores one or more partition replicas. Suppose that there are N brokers and N number of partitions then each broker will store one partition.

What’s the role of Zookeeper?

Kafka uses Zookeeper to maintain cluster state. Zookeeper is a synchronization and coordination service for managing Kafka brokers and its main functionality is to perform leader election across multiple broker instances. Under zookeeper, one server acts as a leader and the other two servers act as followers. Leader node handles all reads and writes per partition. Follower node just follows the instructions given by the leader node. If the leader fails, then the follower node will be automatically appointed as a new leader.

Benefits of Apache Kafka

Kafka has the following benefits.

  • Durability: Kafka allows messages to persist on the disk in order to prevent data loss. It uses distributed commit log for replicating messages across the cluster, and thus making it a durable system.
  • Scalability: It can easily be expanded without any downtime. Since a single Kafka cluster is acting as a central backbone for handling the large organization, we can elastically spread it to multiple clusters.
  • Reliability: It is reliable over time, as it is considered as a distributed, repli- cated, and fault tolerant messaging system.
  • Efficiency: Kafka publishes and subscribes messages efficiently which shows high system throughput. It can store terabytes of messages without any performance impact.

Under this blog post, I will showcase how to implement Apache Kafka on 2 Node Docker Swarm Cluster running on AWS via Docker Desktop.

Pre-requisites:

  • Docker Desktop for Mac
  • AWS Account ( You will require t2.medium instances for this)
  • AWS CLI installed

Adding Your Credentials:

[Captains-Bay]🚩 >  cat ~/.aws/credentials
[default]
aws_access_key_id = XXXA 
aws_secret_access_key = XX

Verifying AWS Version

[Captains-Bay]🚩 >  aws --version
aws-cli/1.11.107 Python/2.7.10 Darwin/17.7.0 botocore/1.5.70

Setting up Environmental Variable

[Captains-Bay]🚩 >  export VPC=vpc-ae59f0d6
[Captains-Bay]🚩 >  export REGION=us-west-2a
[Captains-Bay]🚩 >  export SUBNET=subnet-827651c9
[Captains-Bay]🚩 >  export ZONE=a
[Captains-Bay]🚩 >  export REGION=us-west-2

Building up First Node using Docker Machine

[Captains-Bay]🚩 >  docker-machine create  --driver amazonec2  --amazonec2-access-key=${ACCESS_KEY_ID}  --amazonec2-secret-key=${SECRET_ACCESS_KEY} --amazonec2-region=us-west-2 --amazonec2-vpc-id=vpc-ae59f0d6 --amazonec2-ami=ami-78a22900 --amazonec2-open-port 2377 --amazonec2-open-port 7946 --amazonec2-open-port 4789 --amazonec2-open-port 7946/udp --amazonec2-open-port 4789/udp --amazonec2-open-port 8080 --amazonec2-open-port 443 --amazonec2-open-port 80 --amazonec2-subnet-id=subnet-72dbdb1a --amazonec2-instance-type=t2.micro kafka-swarm-node1

Listing out the Nodes

[Captains-Bay]🚩 >  docker-machine ls
NAME                ACTIVE   DRIVER      STATE     URL                         SWARM   DOCKER     ERRORS
kafka-swarm-node1   -        amazonec2   Running   tcp://35.161.106.158:2376           v18.09.6   
kafka-swarm-node2   -        amazonec2   Running   tcp://54.201.99.75:2376             v18.09.6 

Initialiating Docker Swarm Manager Node

ubuntu@kafka-swarm-node1:~$ sudo docker swarm init --advertise-addr 172.31.53.71 --listen-addr 172.31.53.71:2377
Swarm initialized: current node (yui9wqfu7b12hwt4ig4ribpyq) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-xxxxxmr075to2v3k-decb975h5g5da7xxxx 172.31.53.71:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Adding Worker Node

ubuntu@kafka-swarm-node2:~$ sudo docker swarm join --token SWMTKN-1-2xjkynhin0n2zl7xxxk-decb975h5g5daxxxxxxxxn 172.31.53.71:2377
This node joined a swarm as a worker.

Verifying 2-Node Docker Swarm Mode Cluster

ubuntu@kafka-swarm-node1:~$ sudo docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
yui9wqfu7b12hwt4ig4ribpyq *   kafka-swarm-node1   Ready               Active              Leader              18.09.6
vb235xtkejim1hjdnji5luuxh     kafka-swarm-node2   Ready               Active                                  18.09.6

Installing Docker Compose

curl -L https://github.com/docker/compose/releases/download/1.25.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0   2212      0 --:--:-- --:--:-- --:--:--  2211
100 15.5M  100 15.5M    0     0  8693k      0  0:00:01  0:00:01 --:--:-- 20.1M
root@kafka-swarm-node1:/home/ubuntu/dockerlabs/solution/kafka-swarm# chmod +x /usr/local/bin/docker-compose
root@kafka-swarm-node1:/home/ubuntu/dockerlabs/solution/kafka-

ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker-compose version
docker-compose version 1.25.0-rc1, build 8552e8e2
docker-py version: 4.0.1
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.0j  20 Nov 2018

Building up Kafka Application

git clone https://github.com/collabnix/dockerlabs
cd dockerlabs/solution/kafka-swarm
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker stack deploy -c docker-compose.yml mykafka
Creating network mykafka_default
Creating service mykafka_zkui
Creating service mykafka_broker
Creating service mykafka_manager
Creating service mykafka_producer
Creating service mykafka_zookeeper
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$

Verifying Apache Kafka Stack

ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker stack lsNAME                SERVICES            ORCHESTRATOR
mykafka             5                   Swarm

Verifying Apache Kafka Services

ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                                                     PORTS
t04p6i8zky4z        mykafka_broker      replicated          0/3                 qnib/plain-kafka:2018-04-25_1.1.0                                                         *:9092->9092/tcp
r5f0x9clnwix        mykafka_manager     replicated          1/1                 qnib/plain-kafka-manager:2018-04-25                                                       *:9000->9000/tcp
jzwvrt4df66b        mykafka_producer    replicated          3/3                 qnib/golang-kafka-producer:2018-05-01.12                                                  
09lkbevsktt9        mykafka_zkui        replicated          1/1                 qnib/plain-zkui@sha256:30c4aa1236ee90e4274a9059a5fa87de2ee778d9bfa3cb48c4c9aafe7cfa1a13   *:9090->9090/tcp
b1hqfk1vc4lu        mykafka_zookeeper   replicated          1/1                 qnib/plain-zookeeper:2018-04-25                                                           *:2181->2181/tcp
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ 

In my next blog post, I will talk about Docker compose file which automates the overall Pico project right from video streaming captured from Raspberry Pi to object detection and analytics via AWS Rekognition Service. Stay tuned !

Credits:

How to create a Local Private Docker Registry on Play with Docker in 5 Minutes?

Estimated Reading Time: 4 minutes

DockerHub is a service provided by Docker for finding and sharing container images with your team. It is the world’s largest repository of container images with an array of content sources including container community developers, open source projects and independent software vendors (ISV) building and distributing their code in containers. Users get access to free public repositories for storing and sharing images or can choose subscription plan for private repos.

One can easily pull the Docker Image from Dockerhub and run application in their environment flawlessly. But in case you want to setup a private registry, it is still possible to accomplish. Under this blog post, I will demonstrate how to build a private registry on Play with Docker in just 5 minutes.

Caution – Please note that Play with Docker platform is just for demo or training purpose. As the instance wipe away after 4 hours of limited time, it’s better to save your data before it gets wiped up.

Tested Infrastructure

PlatformNumber of InstanceReading Time
Play with Docker15 min

Pre-requisite

  • Create an account with DockerHub
  • Open PWD Platform on your browser
  • Click on Add New Instance on the left side of the screen to bring up Alpine OS instance on the right side

Create a directory to permanently store images.

$ mkdir -p /opt/registry/data

Authenticate with DockerHub

$docker login

Start the registry container.

$ docker run -d \
  -p 5000:5000 \
  --name registry \
  -v /opt/registry/data:/var/lib/registry \
  --restart always \
  registry:2

Display running containers.

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS
                    NAMES
3a056bf96c6d        registry:2          "/entrypoint.sh /etc…"   About an hour ago   Up About an hour    0.0.0
.0:5000->5000/tcp   registry

Pull Debian Stretch image from official repository.

$ docker pull debian:stretch

stretch: Pulling from library/debian
723254a2c089: Pull complete
Digest: sha256:0a5fcee6f52d5170f557ee2447d7a10a5bdcf715dd7f0250be0b678c556a501b
Status: Downloaded newer image for debian:stretch

Tag local Debian Stretch image with an additional tag – local repository address.

$ docker tag debian:stretch localhost:5000/debian:stretch

Push image to the local repository.

[node1] (local) root@192.168.0.23 ~
$ docker push localhost:5000/debian:stretch
The push refers to repository [localhost:5000/debian]
90d1009ce6fe: Pushed
stretch: digest: sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe size: 529

Remove local images.

[node1] (local) root@192.168.0.23 ~
$ docker image remove debian:stretch
Untagged: debian:stretch
Untagged: debian@sha256:df6ebd5e9c87d0d7381360209f3a05c62981b5c2a3ec94228da4082ba07c4f05
[node1] (local) root@192.168.0.23 ~
$ docker image remove localhost:5000/debian:stretch
Untagged: localhost:5000/debian:stretch
Untagged: localhost:5000/debian@sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe
Deleted: sha256:4879790bd60d439cfe39c063660eef7af525d5f6f1cbb701a14c7cfc11cbfcf7

Pull Debian Stretch image from local repository.

[node1] (local) root@192.168.0.23 ~
$ docker pull localhost:5000/debian:stretch
stretch: Pulling from debian
54f7e8ac135a: Pull complete
Digest: sha256:38236c068c393272ad02db100e09cac36a5465149e2924a035ee60d6c60c38fe
Status: Downloaded newer image for localhost:5000/debian:stretch

List stored images.

[node1] (local) root@192.168.0.23 ~
$ docker image ls
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
localhost:5000/debian   stretch             4879790bd60d        12 days ago         101MB
registry                2                   2e2f252f3c88        2 months ago        33.3MB

Shared local registry

Create a directory to permanently store images.

$ mkdir -p /srv/registry/data

Create a directory to permanently store certificates and authentication data.

$ mkdir -p /srv/registry/security

Store domain and intermediate certificates using /srv/registry/security/registry.crt file, private key using /srv/registry/security/registry.key file. Use valid certificate and do not waste time with self-signed one. This step is required do use basic authentication.

Install apache2-utils to use htpasswd utility.

[node1] (local) root@192.168.0.23 ~
$ apk add apache2-utils
OK: 302 MiB in 110 packages

Create initial username and password. The only supported password format is bcrypt.

$ : | sudo tee /srv/registry/security/htpasswd
[node1] (local) root@192.168.0.23 ~
$ echo "password" | sudo htpasswd -iB /srv/registry/security/htpasswd username
Adding password for user username

Adding password for user username

$(local) root@192.168.0.23 ~ $ cat /srv/registry/security/htpasswd username:$2y$05$q9R5FSNYpAppB4Vw/AGWb.RqMCGE8DmZ4q5HZC/1wC87oTWyvB9vy 

Stop and Remove all old containers

$ docker stop $(docker ps -a -q)
3a056bf96c6d
$ docker rm -f $(docker ps -a -q) 
3a056bf96c6d 

Start the registry container.

[node1] (local) root@192.168.0.23 ~
$ docker run -d   -p 443:5000   --name registry   -v /srv/registry/data:/var/lib/registry   -v /srv/registry/security:/etc/security   -e REGISTRY_HTTP_TLS_CERTIFICATE=/etc/security/registry.crt   -e REGISTRY_HTTP_TLS_KEY=/etc/security/registry.key   -e REGISTRY_AUTH=htpasswd   -e REGISTRY_AUTH_HTPASSWD_PATH=/etc/security/htpasswd   -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm"   --restart always   registry:2
e7755af8cbd70ea84ab77237a87cb97fd1abb18c7726fbc116c40f081d3b7098

Display running containers.

[node1] (local) root@192.168.0.23 ~
$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS        PORTS               NAMES
e7755af8cbd7        registry:2          "/entrypoint.sh /etc…"   About a minute ago   Restarting (1) 22 seconds ago                       registry

Pull Debian Stretch image from official repository.

$ docker pull debian:stretch

stretch: Pulling from library/debian
723254a2c089: Pull complete
Digest: sha256:0a5fcee6f52d5170f557ee2447d7a10a5bdcf715dd7f0250be0b678c556a501b
Status: Downloaded newer image for debian:stretch

Tag local Debian Stretch image with an additional tag – local repository address.

$ docker tag debian:stretch registry.collabnix.com/debian:stretch

This time you need to provide login credentials to use local repository.

$ docker push registry.collabnix.com/debian:stretch

e27a10675c56: Preparing
no basic auth credentials
$ docker pull registry.collabnix.com/debian:stretch

Error response from daemon: Get https://registry.collabnix.com/v2/debian/manifests/stretch: no basic auth credentials

Log in to the local registry.

$ docker login --username username registry.collabnix.com
Password: ********

Login Succeeded

Push image to the local repository.

$ docker push registry.collabnix.com/debian:stretch
The push refers to repository [registry.collabnix.com/debian]
e27a10675c56: Pushed
stretch: digest: sha256:02741df16aee1b81c4aaff4c48d75cc2c308bade918b22679df570c170feef7c size: 529

Remove local images.

$ docker image remove debian:stretch

Untagged: debian:stretch
Untagged: debian@sha256:0a5fcee6f52d5170f557ee2447d7a10a5bdcf715dd7f0250be0b678c556a501b
$ docker image remove registry.collabnix.com/debian:stretch

Untagged: registry.collabnix.com/debian:stretch
Untagged: registry.sl.collabnix.com/debian@sha256:02741df16aee1b81c4aaff4c48d75cc2c308bade918b22679df570c170feef7c
Deleted: sha256:da653cee0545dfbe3c1864ab3ce782805603356a9cc712acc7b3100d9932fa5e
Deleted: sha256:e27a10675c5656bafb7bfa9e4631e871499af0a5ddfda3cebc0ac401dfe19382

Pull Debian Stretch image from local repository.

$ docker pull registry.collabnix.com/debian:stretch

stretch: Pulling from debian
723254a2c089: Pull complete
Digest: sha256:02741df16aee1b81c4aaff4c48d75cc2c308bade918b22679df570c170feef7c
Status: Downloaded newer image for registry.collabnix.com/debian:stretch

List stored images.

$ docker image ls

REPOSITORY                           TAG                 IMAGE ID            CREATED             SIZE
registry                             2                   d1fd7d86a825        4 weeks ago         33.3MB
registry.collabnix.com/debian   stretch             da653cee0545        2 months ago        100MB
hello-world                          latest              f2a91732366c        2 months ago     

Running Docker Containers on EC2 A1 Instances powered by Arm-Based AWS Graviton Processors

Estimated Reading Time: 7 minutes

2 week back, I wrote a blog post on how Developers can now build ARM containers on Docker Desktop using docker buildx CLI Plugin. Usually developers are restricted to build Arm-based application right on top of Arm-based system.Using this plugin, developers can build their application for Arm platform right on their laptop(x86) and then deploy onto the Cloud flawlessly without any cross-compilation pain anymore.

Wait…Did you say “ARM containers on Cloud?”

Yes, you heard it right. It is possible to deploy Arm containers on Cloud. Thanks to new Amazon EC2 A1 instances powered by custom AWS Graviton processors based on the Arm architecture, which brings Arm to the public cloud as a first class citizen. Docker Developers can now build ARM containers on AWS Cloud Platform.

A Brief about AWS Graviton Processors..

Amazon announced the availability of EC2 instances on its Arm-based servers during AWS re:Invent(December 2018). AWS Graviton processors are a new line of processors that are custom designed by AWS targeted in building platform solutions for cloud applications running at scale.The Graviton based instances are known as EC2 A1. These instances are targeted at scale-out workloads and applications such container based microservices, web sites, and scripting language-based applications (e.g., Ruby, Python, etc.)

EC2 A1 instances are built on the AWS Nitro System, a combination of dedicated hardware and lightweight hypervisor, which maximizes resource efficiency for customers while still supporting familiar AWS and Amazon EC2 instance capabilities such as EBS, Networking, and AMIs. Amazon Linux 2, Red Hat Enterpise Linux (RHEL), Ubuntu and ECS optimized AMIs are available today for A1 instances.  Built around Arm cores and making extensive use of custom-built silicon, the A1 instances are optimized for performance and cost.

Under this blog post, I will showcase how to deploy Containers on AWS EC2 A1 instance using Docker Machine running on Docker Desktop for Windows.

Pre-requisites:

My Image
  • Click on “Register for Public Beta”. This will open up various options to test drive Docker products
My Image
  • Don’t forget to Select “Docker Desktop CE with Multi-Arch images (Arm Enabled) – Edge Release Amazon Cloud Credits available for limited time” option.
  • Enter your details and this will open.
  • You will see an option to sign up for credits for Amazon EC2 A1 instances via https://www.surveymonkey.com/r/DockerCon19AWS.
  • Click on Sign Up

Creating AWS Account

  • Go to aws.amazon.com and create Free Tier Account
  • By now, you must have received email from Amazon on Free Credits of $50.
  • Open up https://aws.amazon.com/amazoncredits and add the Promo Code

Creating AWS A1 Instance

We will use Docker Desktop for Windows which comes installed with Docker Machine to bring up ARM instances quickly.

Go to My Security Credentials under your Account and Click “Access Keys” shown below to display Access Key IDs.


Run the below command to set the environmental variable for ACCESS_KEY_ID as well as SECRET_ACCESS_KEY ID.

PS C:\Users\Ajeet_Raina> set ACCESS_KEY_ID=XXX
PS C:\Users\Ajeet_Raina> set SECRET_ACCESS_KEY=XX

Running Docker Machine to bring up our first Docker Node on AWS A1 ARM instance

Docker Desktop for Windows comes with Docker Machine by default and there is NO need to install it separately.

PS C:\Users\Ajeet_Raina> docker-machine create  --driver amazonec2  --amazonec2-access-key=${ACCESS_KEY_ID}  --amazonec2-secret-key=${SECRET_ACCESS_KEY} --amazonec2-region=us-west-2 --amazonec2-vpc-id=vpc-ae59f0d6 --amazonec2-ami=ami-0db180c518750ee4f  --amazonec2-instance-type=a1.medium arm-node1

By now, you should be able to see arm-node1 up and running on your AWS environment.


Listing out the ARM Nodes

PS C:\Users\Ajeet_Raina> docker-machine ls
NAME        ACTIVE   DRIVER      STATE     URL                         SWARM   DOCKER     ERRORS
arm-node1   -        amazonec2   Running   tcp://34.218.208.175:2376           v18.09.6
PS C:\Users\Ajeet_Raina>

Login into the first Node

You can use docker-machine ssh to login into the AWS EC2 A1 instance directly.

PS C:\Users\Ajeet_Raina> docker-machine ssh arm-node1
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1028-aws aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu May 16 04:35:16 UTC 2019

  System load:  0.06              Processes:              116
  Usage of /:   9.1% of 15.34GB   Users logged in:        0
  Memory usage: 10%               IP address for ens5:    172.31.60.52
  Swap usage:   0%                IP address for docker0: 172.17.0.1


  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

178 packages can be updated.
86 updates are security updates.




This node comes with Docker 18.09.6 installed.

ubuntu@arm-node1:~$ sudo docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:40:48 2019
 OS/Arch:           linux/arm64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 02:00:10 2019
  OS/Arch:          linux/arm64
  Experimental:     false
ubuntu@arm-node1:~$




Checking the Node IP

PS C:\Users\Ajeet_Raina> docker-machine ip arm-node1
34.218.208.175

Running ARM-based Portainer v1.20.2 Container

Before we run Portainer, we need to ensure that the port 9000 is open for accessibility.

Click on Actions > Inbound Rules and add 9000 for Portainer. Allowing “All TCP” from 0-65535 is just for testing purpose and not recommended for the production environment.


ubuntu@ip-172-31-62-91:~$ sudo docker run --rm mplatform/mquery portainer/portainer
Unable to find image 'mplatform/mquery:latest' locally
latest: Pulling from mplatform/mquery
db6020507de3: Pull complete
713cdc222639: Pull complete
Digest: sha256:e15189e3d6fbcee8a6ad2ef04c1ec80420ab0fdcf0d70408c0e914af80dfb107
Status: Downloaded newer image for mplatform/mquery:latest
Image: portainer/portainer
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm
   - linux/arm64
   - linux/ppc64le
   - windows/amd64:10.0.14393.2551
   - windows/amd64:10.0.16299.967
   - windows/amd64:10.0.17134.590
   - windows/amd64:10.0.17763.253

Initialising Docker Swarm Mode on Arm-based EC1 instance

Follow the below steps to setup 2 Node Docker Swarm Mode cluster on AWS Platform using Docker Machine.

PS C:\Users\Ajeet_Raina> docker-machine create  --driver amazonec2  --amazonec2-access-key=${ACCESS_KEY_ID}  --amazonec2-secret-key=${SE
CRET_ACCESS_KEY} --amazonec2-region=us-west-2 --amazonec2-vpc-id=vpc-ae59f0d6 --amazonec2-ami=ami-0db180c518750ee4f --amazonec2-open-por
t 2377 --amazonec2-open-port 7946 --amazonec2-open-port 4789 --amazonec2-open-port 7946/udp --amazonec2-open-port 4789/udp --amazonec2-open-port 8080 --amazonec2-open-port 443 --amazonec2-open-port 80 --amazonec2-subnet-id=subnet-827651c9 --amazonec2-instance-type=a1.medi
um arm-swarm-node2
Running pre-create checks...
Creating machine...
(arm-swarm-node2) Launching instance...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...

You can open all ports on AWS using the below command:

PS C:\Users\Ajeet_Raina> aws ec2 authorize-security-group-ingress --group-name docker-machine --protocol -1 --cidr 0.0.0.0/0

Initialising Docker Swarm Manager


PS C:\Users\Ajeet_Raina> docker-machine ssh arm-swarm-node1 sudo docker swarm init
Swarm initialized: current node (oqk875mcldbn28ce2rip31fg5) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-6bw0zfd7vjpXX17usjhccjlg3rs 172.31.50.5:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

PS C:\Users\Ajeet_Raina> docker-machine ssh arm-swarm-node2 sudo docker swarm join --token SWMTKN-1-6XX23ye817usjhccjlg3rs 172.31.50.5:2377
This node joined a swarm as a worker.

Adding Worker Node

PS C:\Users\Ajeet_Raina> docker-machine ssh arm-swarm-node2 sudo docker swarm join --token SWMTKN-1-6bw0zfXXXhccjlg3rs 172.31.50.5:2377
This node joined a swarm as a worker.

Verifying 2-Node Swarm Cluster

ubuntu@arm-swarm-node1:~$ sudo docker node ls
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
oqk875mcldbn28ce2rip31fg5 *   arm-swarm-node1     Ready               Active              Leader              18.09.6
f3rwuj6f6mghte3630car83ia     arm-swarm-node2     Ready               Active                                  18.09.6
ubuntu@arm-swarm-node1:~$

Building Up Portainer Application Stack

ubuntu@ip-172-31-62-91:~$ sudo docker stack deploy --compose-file=portainer-agent-stack.yml portainer
Creating network portainer_agent_network
Creating service portainer_portainer
Creating service portainer_agent
ubuntu@ip-172-31-62-91:~$

Listing out Portainer Stack

ubuntu@arm-node1:~$ sudo docker stack ls
NAME                SERVICES            ORCHESTRATOR
portainer           2                   Swarm
ubuntu@arm-node1:~$ sudo docker service ls
ID                  NAME                  MODE                REPLICAS            IMAGE                        PORTS
k5651aoxgqhk        portainer_agent       global              1/1                 portainer/agent:latest       
yoembxxj25k8        portainer_portainer   replicated          1/1                 portainer/portainer:latest   *:9000->9000/tcp

Viewing Portainer Dashboard

Portainer UI showing a Single Node Swarm Mode Cluster

In my future post, I am going to showcase how I leveraged buildx CLI plugin & AWS EC2 A1 instance to build in-house project called “Pico” for Deep Learning using Apache Kafka, IoT & Amazon Rekognition Service. Stay tuned !