If you are looking out for lightweight Kubernetes which is easy to install and perfect for Edge, IoT, CI and ARM, then look no further. K3s is the right solution for you. K3s is a certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances.
The K3s GITHUB repository has already crossed 9000+ stars. With over 600+ forks & 50+ contributors, this project is gaining a lot of momentum across the developers. Few of notable feature offered by k3s are:
- Single-liner installation
- A binary of less than 40 MB
- Both ARM64 and ARMv7 are supported with binaries and multiarch images available for both
- Can be deployed seamlessly over Raspberry Pi 3 & 4
- SQLite3 as the default storage mechanism. etcd3 is still available, but not the default.
Early this year, I wrote a blog post around how to build up K3s cluster on Raspberry Pi 3.
Under this blog, I will showcase how to build 2-node K3s cluster on NVIDIA Jetson Nano without any compilation pain.
Prerequisite:
- Unboxing Jetson Nano Pack
- Preparing your microSD card
To prepare your microSD card, you’ll need a computer with Internet connection and the ability to read and write SD cards, either via a built-in SD card slot or adapter.
- Download the Jetson Nano Developer Kit SD Card Image, and note where it was saved on the computer.
- Write the image to your microSD card( atleast 16GB size) by following the instructions below according to the type of computer you are using: Windows, Mac, or Linux. If you are using Windows laptop, you can use SDFormatter software for formatting your microSD card and Win32DiskImager to flash Jetson Nano Image. In case you are using Mac, you will need Etcher software.
- To prepare your microSD card, you’ll need a computer with Internet connection and the ability to read and write SD cards, either via a built-in SD card slot or adapter
The Jetson Nano SD card image is of 12GB(uncompressed size).
Next, It’s time to remove this tiny SD card from SD card reader and plugin it to Jetson Board to let it boot.
Verifying OS running on Jetson Nano
jetson@jetson-desktop:~$ sudo cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
jetson@jetson-desktop:~$Setting up hostname
sudo vi /etc/hostname
master1.dell.com
Reboot the system.
Installing K3s
jetson@master1:~$ sudo curl -sfL https://get.k3s.io | sh -
[INFO] Finding latest release
[INFO] Using v0.9.1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v0.9.1/sha256sum-arm64.txt
[INFO] Downloading binary https://github.com/rancher/k3s/releases/download/v0.9.1/k3s-arm64
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Listing the Nodes
jetson@master1:~$ sudo k3s kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
jetson-desktop Ready master 3m55s v1.15.4-k3s.1 192.168.1.3 <none> Ubuntu 18.04.2 LTS 4.9.140-tegra containerd://1.2.8-k3s.1
jetson@master1:~$
Verifying Kubernetes Cluster Information
sudo k3s kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Listing out all Kubernetes Namespaces
$ sudo k3s kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-66f496764-bf7qn 1/1 Running 0 79s
kube-system pod/traefik-d869575c8-wsq2b 0/1 ContainerCreating 0 12s
kube-system pod/svclb-traefik-gtqpd 0/3 ContainerCreating 0 12s
kube-system pod/helm-install-traefik-4tjpc 0/1 Completed 0 79s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-system service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 99s
default service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 97s
kube-system service/traefik LoadBalancer 10.43.140.218 <pending> 80:31655/TCP,443:31667/TCP,8080:31486/TCP 13s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/svclb-traefik 1 1 0 1 0 <none> 13s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 1/1 1 1 99s
kube-system deployment.apps/traefik 0/1 1 0 13s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-66f496764 1 1 1 80s
kube-system replicaset.apps/traefik-d869575c8 1 1 0 13s
NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik 1/1 69s 98s
jetson@jetson-desktop:~$
Deploying NGINX on k3s
$ sudo k3s kubectl run mynginx --image=nginx --replicas=3 --port=80
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/mynginx created
Listing the NGINX Pods
sudo k3s kubectl get po
NAME READY STATUS RESTARTS AGE
mynginx-568f57494d-8jpwq 0/1 ContainerCreating 0 69s
mynginx-568f57494d-czl9x 0/1 ContainerCreating 0 69s
mynginx-568f57494d-pnphb 0/1 ContainerCreating 0 69s
Viewing Nginx Pod details
sudo k3s kubectl describe po mynginx-568f57494d-8jpwq
Name: mynginx-568f57494d-8jpwq
Namespace: default
Priority: 0
Node: jetson-desktop/192.168.1.3
Start Time: Mon, 07 Oct 2019 20:57:14 +0530
Labels: pod-template-hash=568f57494d
run=mynginx
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/mynginx-568f57494d
Containers:
mynginx:
Container ID:
Image: nginx
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-zjsrt (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-zjsrt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-zjsrt
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 98s default-scheduler Successfully assigned default/mynginx-568f57494d-8jpwq to jetson-desktop
Normal Pulling 94s kubelet, jetson-desktop Pulling image "nginx"
Ensuring that Pods are in Running State
jetson@jetson-desktop:~$ sudo k3s kubectl get po
NAME READY STATUS RESTARTS AGE
mynginx-568f57494d-pnphb 1/1 Running 0 113s
mynginx-568f57494d-czl9x 1/1 Running 0 113s
mynginx-568f57494d-8jpwq 1/1 Running 0 113s
jetson@jetson-desktop:~$
Exposing NGINX Port
$ sudo k3s kubectl expose deployment mynginx --port 80
service/mynginx exposed
Verifying the Endpoints
:~$ sudo k3s kubectl get endpoints mynginx
NAME ENDPOINTS AGE
mynginx 10.42.0.6:80,10.42.0.7:80,10.42.0.8:80 27s
Verifying if Nginx is accessible
$ sudo curl 10.42.0.6
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
jetson@jetson
Joining Nodes
I assume that the worker node hostname is “worker1.dell.com”
sudo cat /var/lib/rancher/k3s/server/node-token
password for jetson: K1062243e4f7c5bef777a082ae9919d3d2bf13d446bff423275XXXXd08::node:a761a49be1XXXXXXc7c9cc7ccd9baaf
Installing K3s on Worker Node and joining to the Master Node
jetson@worker1:~$ sudo curl -sfL https://get.k3s.io | K3S_URL=https://master1.dell.com:6443 K3S_TOKEN=K10a2fa03edec41872f9b4068ddcfb9afaf329e24bf25ca33a0dc879b3e02ea9805::node:f9613e3b73d8184277493911ddca4e6a sh -
[INFO] Finding latest release
[INFO] Using v0.9.1 as release
[INFO] Downloading hash https://github.com/rancher/k3s/releases/download/v0.9.1/sha256sum-arm64.txt
[INFO] Skipping binary downloaded, installed k3s matches hash
[INFO] Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO] Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO] systemd: Starting k3s-agent
jetson@master1:~$ sudo k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1.dell.com Ready master 7m45s v1.15.4-k3s.1
worker1.dell.com Ready worker 24s v1.15.4-k3s.1
jetson@master1:~$
Uninstalling k3s
/usr/local/bin/k3s-uninstall.sh
Setting up Kubernetes Dashboard
jetson@master1:~$ cat k3s-dashboard.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
jetson@master1:~$ sudo k3s kubectl apply -f k3s-dashboard.yaml
serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
jetson@master1:~$ cat k3s-dashboard.yaml
Verifying if Kubernetes Dashboard Pods are up and Running
jetson@master1:~$ sudo k3s kubectl get po,svc,deploy -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-66f496764-dgftj 1/1 Running 0 12m
pod/helm-install-traefik-skl9c 0/1 Completed 0 12m
pod/svclb-traefik-bxxpf 3/3 Running 0 11m
pod/traefik-d869575c8-wfnfx 1/1 Running 0 11m
pod/svclb-traefik-nkfms 3/3 Running 0 5m39s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 13m
service/traefik LoadBalancer 10.43.213.124 192.168.1.3,192.168.1.6 80:32651/TCP,443:32650/TCP,8080:30917/TCP 11m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.extensions/coredns 1/1 1 1 13m
deployment.extensions/traefik 1/1 1 1 11m
Enabling Kubeproxy
sudo k3s kubectl proxy
Starting to serve on 127.0.0.1:8001
Verifying if all Pods are successfully running
sudo kubectl get po,deploy,svc -n kube-system
NAME READY STATUS RESTARTS AGE
pod/coredns-66f496764-dgftj 1/1 Running 0 17m
pod/helm-install-traefik-skl9c 0/1 Completed 0 17m
pod/svclb-traefik-bxxpf 3/3 Running 0 16m
pod/traefik-d869575c8-wfnfx 1/1 Running 0 16m
pod/svclb-traefik-nkfms 3/3 Running 0 10m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.extensions/coredns 1/1 1 1 17m
deployment.extensions/traefik 1/1 1 1 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 17m
service/traefik LoadBalancer 10.43.213.124 192.168.1.3,192.168.1.6 80:32651/TCP,443:32650/TCP,8080:30917/TCP In my next blog post, I will showcase how GPU-enabled Kubernetes Pods can be deploy. Stay tuned !
