To implement a microservice architecture and a multi-cloud strategy, Kubernetes today has become a key enabling technology. The bedrock of Kubernetes remains the orchestration and management of Linux containers, to create a powerful distributed system for deploying applications across a hybrid cloud environment. Said that, Kubernetes has become the de-facto standard container orchestration framework for cloud-native deployments. Development teams have turned to Kubernetes to support their migration to new microservices architectures and a DevOps culture for continuous integration and continuous deployment.
Why Docker & Kubernetes on IoT devices?
Today many organizations are going through a digital transformation process. Digital transformation is the integration of digital technology into almost all areas of a business, fundamentally changing how you operate and deliver value to customers. It’s basically a cultural change. The common goal for all these organization is to change how they connect with their customers, suppliers and partners. These organizations are taking advantage of innovations offered by technologies such as IoT platforms, big data analytics, or machine learning to modernize their enterprise IT and OT systems. They realize that the complexity of development and deployment of new digital products require new development processes. Consequently, they turn to agile development and infrastructure tools such as Kubernetes.
At the same time, that there has been a major increase in the demand for Kubernetes outside the datacenter. Kubernetes is pushing out of the data center into stores and factories. DevOps teams do find Kubernetes quite interesting as it provides predictable operations and a cloud-like provisioning experience on just about any infrastructure.
Docker containers & Kubernetes are an excellent choice for deploying complex software to the Edge. The reasons are listed below:
- Containers are awesome
- Consistent across a wide variety of Infrastructure
- Capable of standalone or clustered operations
- Easy to upgrade and/or replace containers
- Support for different infrastructure configs(storage,CPU etc.)
- Strong Ecosystem(Monitoring, logging, CI, management etc.)
Introducing K3s – A Stripped Down version of Kubernetes
K3s is a brand new distribution of Kubernetes that is designed for teams that need to deploy applications quickly and reliably to resource-constrained environments. K3s is a Certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
K3s is lightweight certified K8s distribution built for production operations. It is just 40MB binary with 512MB memory consumption. It is based on a single process w/t integrated K8s master, Kubelet and containerd. It includes SQLite in addition to etcd. Simultaneously released for x86_64, ARM64 and ARMv7. It is an open Source project, not yet Rancher product. It is wrapped in a simple package that reduces the dependencies and steps needed to run a production Kubernetes cluster. Packaged as a single binary, k3s makes installation and upgrade as simple as copying a file. TLS certificates are automatically generated to ensure that all communication is secure by default.
k3s bundles the Kubernetes components (kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, kube-proxy) into combined processes that are presented as a simple server and agent model. Running k3s server will start the Kubernetes server and automatically register the local host as an agent. This will create a one node Kubernetes cluster. To add more nodes to the cluster just run k3s agent --server ${URL} --token ${TOKEN} on another host and it will join the cluster. It’s really that simple to set up a Kubernetes cluster with k3s.
Minimum System Requirements
- Linux 3.10+
- 512 MB of ram per server
- 75 MB of ram per node
- 200 MB of disk space
- x86_64, ARMv7, ARM64
Under this blog post, I will showcase how to get started with K3s on 2-Node Raspberry Pi’s cluster.
Prerequisite:
Hardware:
- Raspberry Pi 3 ( You can order it from Amazon in case you are in India for 2590 INR)
- Micro-SD card reader ( I got it from here )
- Any Windows/Linux/MacOS
- HDMI cable ( I used the HDMI cable of my plasma TV)
- Internet Connectivity(WiFi/Broadband/Tethering using Mobile) – to download Docker 18.09.0 package
- Keyboard & mouse connected to Pi’s USB ports
Software:
- SD-Formatter – to format microSD card (in case of Windows Laptop)
- Win32DiskImager(in case you have Windows OS running on your laptop) – to burn Raspbian format directly into microSD card.(No need to extract XZ using any tool). You can use Etcher tool if you are using macbook.
Steps to Flash Raspbian OS on Pi Boxes:
- Format the microSD card using SD Formatter as shown below:
2. Download Raspbian OS from here and use Win32 imager(in case you are on Windows OS running on your laptop) to burn it on microSD card.
3. Insert the microSD card into your Pi box. Now connect the HDMI cable from one end of Pi’s HDMI slot to your TV or display unit and mobile charger(recommended 5.1V@1.5A).
4. Let the Raspbian OS boot up on your Pi box. It takes hardly 2 minutes for OS to come up.
5. Configure WiFi via GUI. All you need is to input the right password for your WiFi.
6. The default username is “pi” and password is “raspberry”. You can use this credentials to login into the Pi system.
7. You can use “FindPI” Android application to search for IP address if you don’t want to look out for Keyboard or mouse to search for the right IP address.
Enable SSH to perform remote login
To login via your laptop, you need to allow SSH service running. You can verify IP address command via ifconfig command.
[Captains-Bay]? > ssh pi@192.168.1.5
pi@192.168.1.5's password:
Linux raspberrypi 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Feb 26 12:30:00 2019 from 192.168.1.4
pi@raspberrypi:~ $ sudo su
root@raspberrypi:/home/pi# cd
Verifying Raspbian OS Version
root@raspberrypi:~# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
root@raspberrypi:~#
</code></pre>Enable container features in Kernel
Edit /boot/cmdline.txt on both the Raspberry Pi nodes and add the following to the end of the line:
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
Reboot the devices.
Installing K3s
root@raspberrypi:~# curl -sfL https://get.k3s.io | sh -
[INFO] Finding latest release
[INFO] Using v0.2.0 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v0.2.0/sha256sum-arm.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v0.2.0/k3s-armhf
^Croot@raspberrypi:~# wget https://github.com/rancher/k3s/releases/download/v0.2.0/k3s-mhf && \
> chmod +x k3s-armhf && \
> sudo mv k3s-armhf /usr/local/bin/k3s
--2019-03-28 22:47:22-- https://github.com/rancher/k3s/releases/download/v0.2.0/k3s-armhf
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/135516270/4010d900-41db-11e9-9992-cc2248364eac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190328T171725Z&X-Amz-Expires=300&X-Amz-Signature=75c5a361f0219d443dfa0754250c852257f1b8512e54094da0bcc6fbb92327cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dk3s-armhf&response-content-type=application%2Foctet-stream [following]
--2019-03-28 22:47:25-- https://github-production-release-asset-2e65be.s3.amazonaws.com/135516270/4010d900-41db-11e9-9992-cc2248364eac?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190328%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190328T171725Z&X-Amz-Expires=300&X-Amz-Signature=75c5a361f0219d443dfa0754250c852257f1b8512e54094da0bcc6fbb92327cc&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dk3s-armhf&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.0.56
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.0.56|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34684224 (33M) [application/octet-stream]
Saving to: ‘k3s-armhf’
k3s-armhf 100%[========================>] 33.08M 93.1KB/s in 8m 1s
2019-03-28 22:55:28 (70.4 KB/s) - ‘k3s-armhf’ saved [34684224/34684224]
Boostrapping Your K3s Server
root@raspberrypi:~# sudo k3s server
INFO[2019-03-29T10:52:06.995054811+05:30] Starting k3s v0.2.0 (2771ae1)
INFO[2019-03-29T10:52:07.082595332+05:30] Running kube-apiserver --watch-cache=false --cert-dir /var/lib/rancher/k3s/server/tls/temporary-certs --allow-privileged=true --authorization-mode Node,RBAC --service-account-signing-key-file /var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range 10.43.0.0/16 --advertise-port 6445 --advertise-address 127.0.0.1 --insecure-port 0 --secure-port 6444 --bind-address 127.0.0.1 --tls-cert-file /var/lib/rancher/k3s/server/tls/localhost.crt --tls-private-key-file /var/lib/rancher/k3s/server/tls/localhost.key --service-account-key-file /var/lib/rancher/k3s/server/tls/service.key --service-account-issuer k3s --api-audiences unknown --basic-auth-file /var/lib/rancher/k3s/server/cred/passwd --kubelet-client-certificate /var/lib/rancher/k3s/server/tls/token-node.crt --kubelet-client-key /var/lib/rancher/k3s/server/tls/token-node.key
INFO[2019-03-29T10:52:08.094785384+05:30] Running kube-scheduler --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --port 10251 --address 127.0.0.1 --secure-port 0 --leader-elect=false
INFO[2019-03-29T10:52:08.105366477+05:30] Running kube-controller-manager --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file /var/lib/rancher/k3s/server/tls/service.key --allocate-node-cidrs --cluster-cidr 10.42.0.0/16 --root-ca-file /var/lib/rancher/k3s/server/tls/token-ca.crt --port 10252 --address 127.0.0.1 --secure-port 0 --leader-elect=false
Flag --address has been deprecated, see --bind-address instead.
INFO[2019-03-29T10:52:10.410557414+05:30] Listening on :6443
INFO[2019-03-29T10:52:10.519075956+05:30] Node token is available at /var/lib/rancher/k3s/server/node-token
INFO[2019-03-29T10:52:10.519226216+05:30] To join node to cluster: k3s agent -s https://192.168.43.134:6443 -t ${NODE_TOKEN}
INFO[2019-03-29T10:52:10.543022102+05:30] Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml
INFO[2019-03-29T10:52:10.548766216+05:30] Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml
I encountered the below error message while running K3s server for the first time.
INFO[2019-04-04T15:52:44.710450122+05:30] Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: connection refused"
containerd: exit status 1
You can fix it by editing /etc/hosts and adding the right entry for your Pis boxes
127.0.0.1 raspberrypi-node3
By now, you should be able to get k3s nodes listed.
root@raspberrypi:~# sudo k3s kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
raspberrypi Ready <none> 2m13s v1.13.4-k3s.1 192.168.43.134 <none> Raspbian GNU/Linux 9 (stretch) 4.14.98-v7+ containerd://1.2.4+unknown
Listing K3s Nodes
root@raspberrypi:~# k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
raspberrypi Ready <none> 2m26s v1.13.4-k3s.1
Listing K3s Pods
root@raspberrypi:~# k3s kubectl get po
No resources found.
root@raspberrypi:~# k3s kubectl get po,svc,deploy
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 11h
root@raspberrypi:~#
containerd and Docker
k3s by default uses containerd. If you want to use it with Docker, all you just need to run the agent with the --docker flag
k3s agent -s ${SERVER_URL} -t ${NODE_TOKEN} --docker &Running Nginx Pods
To launch a pod using the container image nginx and exposing a HTTP API on port 80, execute:
root@raspberrypi:~# k3s kubectl run mynginx --image=nginx --replicas=3 --port=80
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/mynginx created
Listing the Nginx Pods
You can now see that the pod is running:
root@raspberrypi:~# k3s kubectl get po
NAME READY STATUS RESTARTS AGE
mynginx-84b8d48d44-ggpcp 1/1 Running 0 119s
mynginx-84b8d48d44-hkdg8 1/1 Running 0 119s
mynginx-84b8d48d44-n4r6q 1/1 Running 0 119s
Exposing the Deployment
Create a Service object that exposes the deployment:
root@raspberrypi:~# k3s kubectl expose deployment mynginx --port 80
service/mynginx exposed
Verifying the endpoints controller for Pods
The below command verifies if endpoints controller has found the correct Pods for your Service:
root@raspberrypi:~# k3s kubectl get endpoints mynginx
NAME ENDPOINTS AGE
mynginx 10.42.0.10:80,10.42.0.11:80,10.42.0.12:80 17s
Testing if Nginx application is up & running:
root@raspberrypi:~# curl 10.42.0.10:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Adding a new Node to K3s Cluster
To add more nodes to the cluster just run k3s agent --server ${URL} --token ${TOKEN} on another host and it will join the cluster. It’s really that simple to set up a Kubernetes cluster with k3s.
To test drive K3s on multi-node cluster, first you will need to copy the token which is stored on the below location:
root@raspberrypi:~# cat /var/lib/rancher/k3s/server/node-token
K108b8e370b380bea959e8017abea3e540d1113f55df2c3f303ae771dc73fc67aa3::node:42e3dfc68ee27cf7cbdae5e4c8ac91b2
root@raspberrypi:~#
Create a variable NODETOKEN with token ID and then pass it directly with k3s agent command as shown below:
root@pi-node1:~# NODETOKEN=K108b8e370b380bea959e8017abea3e540d1113f55df2c3f303ae771dc73fc67aa3::node:42e3dfc68ee27cf7cbdae5e4c8ac91b2
root@pi-node1:~# k3s agent --server https://192.168.1.5:6443 --token ${NODETOKEN}
INFO[2019-04-04T23:09:16.804457435+05:30] Starting k3s agent v0.3.0 (9a1a1ec)
INFO[2019-04-04T23:09:19.563259194+05:30] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-04-04T23:09:19.563629400+05:30] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
INFO[2019-04-04T23:09:19.613809334+05:30] Connecting to wss://192.168.1.5:6443/v1-k3s/connect
INFO[2019-04-04T23:09:19.614108395+05:30] Connecting to proxy url="wss://192.168.1.5:6443/v1-k3s/connect"
FATA[2019-04-04T23:09:19.907450499+05:30] Failed to start tls listener: listen tcp 127.0.0.1:6445: bind: address already in use
root@pi-node1:~# pkill -9 k3s
root@pi-node1:~# k3s agent --server https://192.168.1.5:6443 --token ${NODETOKEN}
INFO[2019-04-04T23:09:45.843235117+05:30] Starting k3s agent v0.3.0 (9a1a1ec)
INFO[2019-04-04T23:09:48.272160155+05:30] Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log
INFO[2019-04-04T23:09:48.272542392+05:30] Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
/run/k3s/containerd/containerd.sock: connect: connection refused"
INFO[2019-04-04T23:09:49.321863688+05:30] Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet
INFO[2019-04-04T23:09:50.347628159+05:30] Connecting to wss://192.168.1.5:6443/v1-k3s/connect
Listing the k3s Nodes
root@raspberrypi:~# k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
pi-node1 Ready <none> 118s v1.13.5-k3s.1
pi-node2 Ready <none> 108m v1.13.5-k3s.1
Setting up Nginx
As shown earlier, we will go ahead and test Nginx application on top of K3s cluster nodes
root@raspberrypi:~# k3s kubectl run mynginx --image=nginx --replicas=3 --port=80
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/mynginx created
Verifying the endpoints controller for Pods
root@raspberrypi:~# k3s kubectl expose deployment mynginx --port 80
service/mynginx exposed
Test driving Kubernetes Dashboard
root@node1:/home/pi# k3s kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
root@node1:/home/pi#
We require kubectl proxy to get it accessible on our Web browser. This command creates a proxy server or application-level gateway between localhost and the Kubernetes API Server. It also allows serving static content over specified HTTP path. All incoming data enters through one port and gets forwarded to the remote kubernetes API Server port, except for the path matching the static content path.
root@node1:/home/pi# k3s kubectl proxy
Starting to serve on 127.0.0.1:8001By now, you should be able to access Dashboard via 8001 port on your Raspberry Pi system browser.
Cleaning up
kubectl delete --all pods
pod "mynginx-84b8d48d44-9ghrl" deleted
pod "mynginx-84b8d48d44-bczsv" deleted
pod "mynginx-84b8d48d44-qqk9p" deletedHope you find this blog useful. In my future blog post, I will be talking about K3s Internals in detail.
References:
Comments are closed.