Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation. It is written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
Apache Kafka is a distributed, partitioned, and replicated publish-subscribe messaging system that is used to send high volumes of data, in the form of messages, from one point to another. It replicates these messages across a cluster of servers in order to prevent data loss and allows both online and offline message consumption. This in turn shows the fault-tolerant behavior of Kafka in the presence of machine failures that also supports low latency message delivery. In a broader sense, Kafka is considered as a unified platform which guarantees zero data loss and handles real-time data feeds.
In order to understand the functionality of Kafka, there are few methodologies that need to be discussed before moving deep into the architecture. A stream of messages is divided into particular categories called topics. These messages are published to specific topics using dedicated processes which we call producers. A separate set of processes called consumers are used to pull data from the brokers by subscribing to one or more topics. At a high level, we can say that the producers send messages to the Kafka cluster which then consumed by the consumers
Let us spend few minutes in understanding what is streaming all about? A streaming platform has three major capabilities:
- Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
- Store streams of records in a fault-tolerant durable way.
- Process streams of records as they occur.
Said that, Kafka is generally used for two broad classes of applications:
- Building real-time streaming data pipelines that reliably get data between systems or applications
- Building real-time streaming applications that transform or react to the streams of data
Kafka is run as a cluster on one or more servers that can span multiple datacenters. The Kafka cluster stores streams of records in categories called topics. Each record consists of a key, a value, and a timestamp.
Core APIs of Apache Kafka
source ~ https://kafka.apache.org
- The Producer API allows an application to publish a stream of records to one or more Kafka topics.
- The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
- The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
- The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.
Under this blog post, I will showcase how to implement Apache Kafka on 2- Node Docker Swarm Cluster running on AWS via Docker Desktop.
Pre-requisites:
- Docker Desktop for Mac
- AWS Free Tier Account
- AWS CLI installed
Adding Your Credentials:
[Captains-Bay]🚩 > cat ~/.aws/credentials
[default]
aws_access_key_id = XXXA
aws_secret_access_key = XXVerifying AWS Version
[Captains-Bay]🚩 > aws --version
aws-cli/1.11.107 Python/2.7.10 Darwin/17.7.0 botocore/1.5.70Setting up Environmental Variable
[Captains-Bay]🚩 > export VPC=vpc-ae59f0d6
[Captains-Bay]🚩 > export REGION=us-west-2a
[Captains-Bay]🚩 > export SUBNET=subnet-827651c9
[Captains-Bay]🚩 > export ZONE=a
[Captains-Bay]🚩 > export REGION=us-west-2Building up First Node using Docker Machine
[Captains-Bay]🚩 > docker-machine create --driver amazonec2 --amazonec2-access-key=${ACCESS_KEY_ID} --amazonec2-secret-key=${SECRET_ACCESS_KEY} --amazonec2-region=us-west-2 --amazonec2-vpc-id=vpc-ae59f0d6 --amazonec2-ami=ami-78a22900 --amazonec2-instance-type=t2.micro kafka-swarm-node1Listing out the Nodes
[Captains-Bay]🚩 > docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
kafka-swarm-node1 - amazonec2 Running tcp://35.161.106.158:2376 v18.09.6
kafka-swarm-node2 - amazonec2 Running tcp://54.201.99.75:2376 v18.09.6 Initialiating Docker Swarm Manager Node
ubuntu@kafka-swarm-node1:~$ sudo docker swarm init --advertise-addr 172.31.53.71 --listen-addr 172.31.53.71:2377
Swarm initialized: current node (yui9wqfu7b12hwt4ig4ribpyq) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-xxxxxmr075to2v3k-decb975h5g5da7xxxx 172.31.53.71:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.Adding Worker Node
ubuntu@kafka-swarm-node2:~$ sudo docker swarm join --token SWMTKN-1-2xjkynhin0n2zl7xxxk-decb975h5g5daxxxxxxxxn 172.31.53.71:2377
This node joined a swarm as a worker.Verifying 2-Node Docker Swarm Mode Cluster
ubuntu@kafka-swarm-node1:~$ sudo docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
yui9wqfu7b12hwt4ig4ribpyq * kafka-swarm-node1 Ready Active Leader 18.09.6
vb235xtkejim1hjdnji5luuxh kafka-swarm-node2 Ready Active 18.09.6
Installing Docker Compose
curl -L https://github.com/docker/compose/releases/download/1.25.0-rc1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 617 0 617 0 0 2212 0 --:--:-- --:--:-- --:--:-- 2211
100 15.5M 100 15.5M 0 0 8693k 0 0:00:01 0:00:01 --:--:-- 20.1M
root@kafka-swarm-node1:/home/ubuntu/dockerlabs/solution/kafka-swarm# chmod +x /usr/local/bin/docker-compose
root@kafka-swarm-node1:/home/ubuntu/dockerlabs/solution/kafka-
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker-compose version
docker-compose version 1.25.0-rc1, build 8552e8e2
docker-py version: 4.0.1
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.0j 20 Nov 2018Building up Kafka Application
git clone https://github.com/collabnix/dockerlabs
cd dockerlabs/solution/kafka-swarmubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker stack deploy -c docker-compose.yml mykafka
Creating network mykafka_default
Creating service mykafka_zkui
Creating service mykafka_broker
Creating service mykafka_manager
Creating service mykafka_producer
Creating service mykafka_zookeeper
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$Verifying Apache Kafka Stack
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker stack lsNAME SERVICES ORCHESTRATOR
mykafka 5 SwarmVerifying Apache Kafka Services
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ sudo docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
t04p6i8zky4z mykafka_broker replicated 0/3 qnib/plain-kafka:2018-04-25_1.1.0 *:9092->9092/tcp
r5f0x9clnwix mykafka_manager replicated 1/1 qnib/plain-kafka-manager:2018-04-25 *:9000->9000/tcp
jzwvrt4df66b mykafka_producer replicated 3/3 qnib/golang-kafka-producer:2018-05-01.12
09lkbevsktt9 mykafka_zkui replicated 1/1 qnib/plain-zkui@sha256:30c4aa1236ee90e4274a9059a5fa87de2ee778d9bfa3cb48c4c9aafe7cfa1a13 *:9090->9090/tcp
b1hqfk1vc4lu mykafka_zookeeper replicated 1/1 qnib/plain-zookeeper:2018-04-25 *:2181->2181/tcp
ubuntu@kafka-swarm-node1:~/dockerlabs/solution/kafka-swarm$ 
