Join our Discord Server
Ajeet Raina Ajeet Singh Raina is a former Docker Captain, Community Leader and Arm Ambassador. He is a founder of Collabnix blogging site and has authored more than 570+ blogs on Docker, Kubernetes and Cloud-Native Technology. He runs a community Slack of 8900+ members and discord server close to 2200+ members. You can follow him on Twitter(@ajeetsraina).

Top 50 Kubernetes Interview Questions For You

31 min read

Table of Contents

Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads. This provides much of the simplicity of Platform as a Service (PaaS) with the flexibility of Infrastructure as a Service (IaaS), and enables portability across infrastructure provider.

Below are top 50 interview questions for candidates who want to prepare on Kubernetes Technology:

Kubernetes Fundamentals – Beginners Level

What is Kubernetes and why is it damn popular?

This is a basic interview question asked by the interviewer to check candidate’s knowledge around Kubernetes. The interviewer expects the candidate to be aware of why Kubernetes is HOT in the market and what problem does Kubernetes solve for all of us. One can begin the answer with –

Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery.

Kubernetes is a HUGE open source project with a lot of code and functionalities. The primary responsibility of Kubernetes is container orchestration. That means making sure that all the containers that execute various workloads are scheduled to run physical or virtual machines. The containers must be packed efficiently following the constraints of the deployment environment and the cluster configuration. In addition, Kubernetes must keep an eye on all running containers and replace dead, unresponsive, or otherwise unhealthy containers. 

Said that Kubernetes is rightly a platform for managing application containers across multiple hosts. It provides lots of management features for container-oriented applications, such as  rolling deployment,   resource, and volume management. Same as the nature of containers, it’s designed to run anywhere, so we’re able to run it on a bare metal, in our data center, on the public cloud, or even hybrid cloud. 

Kubernetes considers most of the operational needs for application containers. The Top 10 Reasons why Kubernetes is so popular are as follow:

  • Largest Open Source project in the world
  • Great Community Support
  • Robust Container deployment
  • Effective Persistent storage
  • Multi-Cloud Support(Hybrid Cloud)
  • Container health monitoring
  • Compute resource management
  • Auto-scaling Feature Support
  • Real-world Use cases Available
  • High availability by cluster federation

Why should I use Kubernetes?

This is a bit tricky question. The intention of the interviewer is trying to understand candidate’s understanding of why should one head towards Kubernetes adoption inside the company or on the cloud. 

With the advent of microservice architecture, users to individually scale key functions of an application and handle millions of customers. On top of this, technologies like Docker containers emerged in the enterprise, creating a consistent, portable, and easy way for users to quickly build these microservices. While Docker continued to thrive, managing these microservices & containers became a paramount requirement. All you need is a robust orchestration platform which can manage those containers which host your entire application. Kubernetes comes to a rescue.

Kubernetes is a robust orchestration platform which brings a number of features and which can be thought of as:

  • As a container platform
  • As a microservices platform
  • As a portable cloud platform and a lot more.

Kubernetes provides a container-centric management environment. It orchestrates computing, networking, and storage infrastructure on behalf of user workloads. This provides much of the simplicity of Platform as a Service (PaaS) with the flexibility of Infrastructure as a Service (IaaS), and enables portability across infrastructure providers. Below are the list of features which Kubernetes provides –

  • Service Discovery and load balancing: Kubernetes has a feature which assigns the containers with their own IP addresses and a unique DNS name, which can used to balance the load on them.
  • Planning: Placement of the containers on the node is a crucial feature on which makes the decision based on the resources it requires and other restrictions. 
  • Auto Scaling: Based on the CPU usage, vertical scaling of applications is automatically triggered using the command line.
  • Self Repair: This is an unique feature in the Kubernetes which will restart the container automatically when it fails. If the Node dies, then containers are replaced or re-planned on the other Nodes. You can stop the containers, if they don’t respond for the health checks.
  • Storage Orchestration: This feature of Kubernetes enables the user to mount the network storage system as a local file system.
  • Batch execution: Kubernetes manages both batch and CI workloads along with replacing containers that fail.
  • Deployments and Automatic Rollbacks: During the configuration changes for the application hosted on the Kubernetes, progressively monitors the health to ensure that it does not terminate all the instances at once, it makes an automatic rollback only in the case of failure.
  • Configuration Management and Secrets: All classifies information like keys and passwords are stored under module called Secrets in Kubernetes. These Secrets are used specially while configuring the application without having to reconstruct the image.

How Kubernetes is related to docker?

This is one of the most important question ever asked in an interview. Though we compare Docker Vs Kubernetes, it is an apple-to-orange comparison. Reason – They are both fundamentally different technologies but they work very well together, and both facilitate the management and deployment of containers in a distributed architecture.

Let me elaborate –

Docker started as a GITHUB project back in 2013(which is almost 5+ years from now). Slowly it grew massively with HUGE contributors across the world. Today it is a platform which is shipped as both – an open source as well as a commercial product. The orchestration is just a mere feature of Docker Enterprise Edition.

But if we really want to study how K8s is related to Docker, then the most preferred answer would be –

Docker CLI provides the mechanism for managing the life cycle of the containers. Where as the docker image defines the build time framework of runtime containers. CLI commands are there to start, stop, restart and perform lifecycle operations on these containers. Containers can be orchestrated and can be made to run on multiple hosts. The questions that need to be answered are how these containers are coordinated and scheduled? And how will the application running in these containers will communicate each other?

 Kubernetes is the answer. Today, Kubernetes mostly uses Docker to package, instantiate, and run containerized applications. Said that there are various another container runtime available but Docker is the most popular runtime binary used by Kubernetes.

Both Kubernetes and Docker build a comprehensive standard for managing the containerized applications intelligently along with providing powerful capabilities.Docker provides a platform for building running and distributing Docker containers. Docker brings up its own clustering tool which can be used for orchestration. But Kubernetes is a orchestration platform for Docker containers which is more extensive than the Docker clustering tool, and has capacity to scale to the production level. Kubernetes is a container orchestration system for Docker containers that is more extensive than Docker Swarm and is meant to coordinate clusters of nodes at scale in production in an efficient manner.  Kubernetes is a plug and play architecture for the container orchestration which provides features like high availability among the distributed nodes

How Kubernetes simplify the containerized application deployment process?

A application deployment requires , web tier , application tier and database tier . All these requirements will spawn multiple containers and these containers should communicate among each other . Kubernetes cluster will take care of the whole system and orchestrates the container needs . 

Let us look at a quick WordPress application example. WordPress application consists of frontend(WordPress running on PHP and Apache) and backend(MySQL). The below YAML file can help you specify everything you will need to bring WordPress Application in a single shot:

apiVersion: v1
kind: Service
metadata:
 name: wordpress
 labels:
   app: wordpress
spec:
 ports:
   - port: 80
 selector:
   app: wordpress
   tier: frontend
 type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: wp-pv-claim
 labels:
   app: wordpress
spec:
 storageClassName: manual
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: 2Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
 name: wordpress
 labels:
   app: wordpress
spec:
 storageClassName: manual
 selector:
   matchLabels:
     app: wordpress
     tier: frontend
 strategy:
   type: Recreate
 template:
   metadata:
     labels:
       app: wordpress
       tier: frontend
   spec:
     containers:
     - image: wordpress:4.8-apache
       name: wordpress
       env:
       - name: WORDPRESS_DB_HOST
         value: wordpress-mysql
       - name: WORDPRESS_DB_PASSWORD
         valueFrom:
           secretKeyRef:
             name: mysql-pass
             key: password
       ports:
       - containerPort: 80
         name: wordpress
       volumeMounts:
       - name: wordpress-persistent-storage
         mountPath: /var/www/html
     volumes:
     - name: wordpress-persistent-storage
       persistentVolumeClaim:
         claimName: wp-pv-claim


I assume that you have n-node Kubernetes cluster running in your infrastructure. All you need is to run the below command:

kubectl create -f wordpress-deployment.yaml

           That’s it. Browse to http://<IP>:80 port to open to see WordPress App up and running. Hence, we saw that how Kubernetes simplifies the application deployment.


How to Install Kubernetes?

Install below packages on all of your machines:

  • kubeadm: the command to bootstrap the cluster.
  • kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
  • kubectl: the command line util to talk to your cluster.

Note : kubeadm will not install or manage kubelet or kubectl for you, so you will need to ensure they match the version of the Kubernetes control panel you want kubeadm to install for you.

 If you do not, there is a risk of a version skew occurring that can lead to unexpected, buggy behavior. 

However, one minor version skew between the kubelet and the control plane is supported, but the kubelet version may never exceed the API server version. For example, kubelet running 1.7.0 should be fully compatible with a 1.8.0 API server, but not vice versa.

Below is the example for installing in Debian or Ubuntu flavours


# apt-get update && apt-get install -y apt-transport-https curl
# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
# apt-get update
# apt-get install -y kubelet kubeadm kubectl
# apt-mark hold kubelet kubeadm kubectl


Configure cgroup driver used by kubelet on Master Node

When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet and set it in the /var/lib/kubelet/kubeadm-flags.env file during runtime.

If you are using a different CRI, you have to modify the file /etc/default/kubelet with your cgroup-driver value, like so:

KUBELET_KUBEADM_EXTRA_ARGS=–cgroup-driver=<value>

This file will be used by kubeadm init and kubeadm join to source extra user-defined arguments for the kubelet.

Please mind, that you only have to do that if the cgroup driver of your CRI is not cgroupfs, because that is the default value in the kubelet already.

Restarting the kubelet is required:


# systemctl daemon-reload
# systemctl restart kubelet


Explain the Architecture Layers of Kubernetes?

Kubernetes can be visualized as a system built in layers, with each higher layer abstracting the complexity found in the lower levels.

Base Layer

At the base, Kubernetes makes a cluster which is a collection of hosts storage and networking resources that Kubernetes uses to run the various workloads that comprise the system . Cluster groups together a large fleet of machines into a single unit that can be consumed .

Mid Layer

The machines in the cluster are each given a role within the Kubernetes ecosystem. 

The MASTER is the control plane of Kubernetes having functionalities like Authorization and authentication , RESTful API entry point. Master comprises of components, such as an API server , scheduler, and controller manager. The master is responsible for the global, cluster-level scheduling of pods and handling of events. 

  • The other machines in the cluster are designated as nodes: 

Nodes are managed by a Kubernetes master. The nodes are worker bees of Kubernetes and do all the processing and hardwork . Each node has a container runtime , receives instructions from master node , creates / destroys container as per the workload and enroutes traffic appropriately 

Application Layer/ Final Layer

Kubernetes itself is a somewhat complicated distributed system which runs on API approach . 

  • To run an application , a plan is submitted in yaml or json
  • The master server runs the plan by examining the requirements and current state of the cluster
  • All user interact with the cluster by the help of API ecosystem implemented by control plane of the master server 
  • Next comes the scheduler and controller-manager components that keeps the cluster functioning correctly 
  • In the last its the workers that will take the pain and provides the output for the job 

List out the Master Server Components of Kubernetes?


Kubernetes is a combination of multiple parts working together to get the container job done and the most vital part of it is Master node . The node acts as brain to the cluster and manages the whole ecosystem . 

Master connects to etcd via HTTP or HTTPS to store the data and also connects flannel to access the container application.

Worker nodes engage with master via HTTP or HTTPS to get a command and report the status.

Overlay network makes connections of their container applications. All of it will be discussed below for more in-depth

Below are the mentioned components :

etcd

  • The heart of any Kubernetes cluster that implements a distributed key value store where all of the objects in a kubernetes cluster are persisted . 
  • It works on a algorithm which has replication techniques across servers to maintain the data stored in etcd . 
  • Optimistic concurrency is also used to compare-and-swap data across etcd server , when a user reads and update a value , the system checks that no other component in the system has updated the same value . This technique removes the locking mechanism that increases the server throughput .
  • Another technique known as watch protocol , which accounts for changes made in key value pair in etcd directory . Its improves efficiency to the client as it wait for the changes and then react to the change without continuous polling to the server .

kube-apiserver

As the name connects , its a server that provides an HTTP- or HTTPS-based RESTful API that is allowed to have direct access to the Kubernetes cluster .

  • Its a connector between all the kubernetes components and mediates all interactions between clients and the API objects stored in etcd .
  • Api server database is external to it , so it is a stateless server which is replicated 3 times to implement fault-tolerance
  • The APIs are exposed and managed by the server , the characteristics of those API requests must be described so that the client and server know how to communicate .
  • Define API pattern where the request is defined like api paths or groups.
  • Internal loops are responsible for background operations like CRD (Custom Resource Definitions) which inherently creates new paths for API requests

kube-controller-manager

The controller manager is a general service that has many responsibilities. 

  • Controller manager is a collection of control loops rolled up into one binary
  • Manages Kubernetes nodes
  • The control loops needed to implement the functionality like replica sets and deployments are run by Manager
  • Creates and updates the Kubernetes internal information
  • changes the current status to the desired status

kube-scheduler

  • It is a simple algorithm that defines the priority to dispatch and is responsible for scheduling pods into nodes .
  • is continuously scanning the API server (with watch protocol) for Pods which don’t have a nodeName and are eligible for scheduling
  • Node affinity provide a simple way to guarantee that a Pod lands on a particular node
  • Predicates is a concept that helps in making correct resource requirements for the pods
  • Data locality

List the Node Server Components of Kubernetes ?

In Kubernetes, servers that perform work by running containers are known as nodes. Execution of jobs and reporting the status back to the master are the primary tasks on Node server .

kubelet

The main process on Kubernetes node that performs major container operations . 

  •  The Kubelet is the node-daemon that communicates with Kubernetes master for all machines that are part of a Kubernetes cluster. 
  • It periodically access the controller to check and report the status of the cluster
  • It merges the available CPU, disk and memory for a node into the large Kubernetes cluster.
  • Communicates the state of containers back up to the api-server for control loops to observe the current state of the containers.

kube-proxy

  • The kube proxy implements load-balancer networking model on each node. 
  • It makes the Kubernetes services locally and can do TCP and UDP forwarding.
  • The kube-proxy programs the network on its node, so that network requests to the virtual IP address of a service, are in-fact routed to the endpoints which implement this service
  • It finds cluster IPs via environment variables or DNS.
  • Routes traffic from Pods on the machine to Pods, anywhere in the cluster

List out Kubernetes Objects and Workloads?

Kubernetes object model provides set of features to manage containers and interact with instances.

Here are the few listed different types of objects used to define workloads.

Pods : Pod is the basic unit in the Kubernetes Object Model. In Kubernetes containers are not attached to hosts directly, instead one or more containers are tightly coupled form an encapsulated object called pod.

Replication Controllers and Replication Sets : know as replication of pods. These are created from pod templates and can be horizontally scaled by controllers known as replication controllers and replication sets.

Deployments : Deployments are one of the most common workloads to directly create and manage. Deployments use replication sets as a building block, adding flexible life cycle management functionality to the mix.

Stateful Sets : Stateful sets are specialized pod controllers that offer ordering and uniqueness guarantees. For instance, stateful sets are often associated with data-oriented applications, like databases, which need access to the same volumes even if rescheduled to a new node.

Daemon Sets : Daemon sets are another specialized form of pod controller that run a copy of a pod on each node in the cluster . This is most often useful when deploying pods that help perform maintenance and provide services for the nodes themselves.

Jobs and Cron Jobs : Kubernetes uses a workload called jobs to provide a more task-based workflow where the running containers are expected to exit successfully after some time once they have completed their work. Jobs are useful if you need to perform one-off or batch processing instead of running a continuous service.

Explain the Replication Controllers and Replication Sets in Kubernetes ?

ReplicationController makes sure that a specified number of pod replicas are running at any point of time. Specifically ReplicationController ensures that a pod or set of pods are homogeneous in nature and are always up and running.

ReplicationController always maintains desired number of pods, if the number exceeds then it will terminate extra pods, or if the number decreases extra pods will be created. The pods maintained by a ReplicationController are automatically replaced if they fail, are deleted, or are terminated. Replication controllers can also perform rolling updates to roll over a set of pods to a new version one by one, minimizing the impact on application availability.

Replication sets are an iteration on the replication controller design with greater flexibility in how the controller identifies the pods it is meant to manage. Replication sets are much more advanced than ReplicationController as they have greater replica selection capabilities, but they dont have the rolling updates capabilities.

Creating a replication controller

To create replication controllers, use the subcommand run after kubectl. 


//  kubectl run <REPLICATION CONTROLLER NAME> --images=<IMAGE NAME> [OPTIONAL_FLAGS]
# kubectl run my-first-rc --image=nginx
CONTROLLER    CONTAINER(S)   IMAGE(S)   SELECTOR      REPLICAS
my-first-rc   my-first-rc    nginx      run=my-first-rc   1


Above command is creating a replication controller by image nginx .The name, my-first-rc, must be unique in all replication controllers.

Without specified number of replicas, the system will only build one pod as its default value

Explain Pods in Kubernetes Context

Pod is a single or bunch of containers that is controlled as a single application 

  • Containers inside the Pod operate closely together and share a common life cycle, but has to be scheduled on the same node. 
  • Pods are managed as a unit and share common environment wrt volume and IP address space.
  • Every Pods consists of master container that satisfies of balancing the workload among the other containers that facilitate to orchestrate other related tasks.
  • For example, a pod may have one container running the primary application server and a helper container pulling down files to the shared file system when changes are detected in an external repository. 
  • Users are recommended not to manage pods themselves, because they might miss few features specifically needed in applications. 
  • Users are advised to operate with the objects that use pod templates as base components and add additional functionality to them.

Use Secrets in Pods

To use Secrets inside Pods, choose to expose pods in environment variables or mount the Secrets as volumes.

In terms of accessing Secrets inside a Pod, add env section inside the container spec

// using access-token Secret inside a Pod# cat 2-7-2_env.yamlapiVersion: v1kind: Podmetadata:   name: secret-pod-envspec:  containers:  – name: ubuntu    image: ubuntu    command: [“/bin/sh”, “-c”, “while : ;do echo $ACCESS_TOKEN; sleep 10; done”]    env:        – name: ACCESS_TOKEN          valueFrom:            secretKeyRef:              name: access-token              key: 2-7-1_access-token// create a pod# kubectl create -f 2-7-2_env.yamlpod “secret-pod-env” created

Below example, expose 2-7-1_access-token key in access-token Secret as ACCESS_TOKEN environment variable, and print it out through a while infinite loop .

// check stdout logs# kubectl logs -f secret-pod-env9S!g0U616456r

 Explain stateful sets

During regular deployment process, all the instances of a pod are identical,and these stateless applications can be easily scaled up and down. In a PetSet, each pod is unique and is been assigned with unique identifier that needs to be maintained. This is technique is generally used for more stateful applications.

Example creating Stateful Set

Use the following command to get to start the creation of this StatefulSet

$ kubectl create -f abc-statefulset.yaml

Use get subcommand to see stateful sets:

$ kubectl get statefulsets$ kubectl get pods

Get the volumes the set has created and claim for each pod :

$ kubectl get pv

 Explain Daemon sets

A DaemonSet is a set of pods that is run only once on a host. It’s used for host-layer features, for instance a network, host monitoring or storage plugin or other things which you would never want to run more than once on a host.

  1.  Explain Master

Master is the central control point that provides a unified view of the cluster. There is a single master node that control multiple minions.

Master servers work together to accept user requests, determine the best ways to schedule workload containers, authenticate clients and nodes, adjust cluster-wide networking, and manage scaling and health checking responsibilities

  1.  Explain Minions

A node is a worker machine in Kubernetes, previously known as a minion. A node may be a VM or physical machine, depending on the cluster. Each node contains the services necessary to run pods and is managed by the master components. The services on a node include the container runtime, kubelet and kube-proxy. 

  1.  ExplainVolumes and Persistent Volumes in kubernetes

A Kubernetes volume, on the other hand, the same as the Pod that encloses it. Consequently, a volume outlives any Containers that run within the Pod, and data is preserved across Container restarts. Of course, when a Pod ceases to exist, the volume will cease to exist, too. Perhaps more importantly than this, Kubernetes supports many types of volumes, and a Pod can use any number of them simultaneously.

The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this we introduce two new API resources:PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a storage in the cluster that has to be provisioned by an administrator and it is a cluster resource. PVs are volume plugins like Volumes, but have a life cycle independent of any individual pod that uses the PV. 

This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes 

 Explain Labels and Annotations in kubernetes

Label in Kubernetes is meaningful tag word that can be attached to Kubernetes objects to make them as a part of a group. 

Labels can be used for working on different instances for management or routing purposes. 

For example, the controller-based objects use labels to mark the pods that they should operate on. Micro Services use labels to understand the structure of backend pods they should route requests to.

Labels are key-value pairs. Each unit can have more than one label, but each unit can only have one entry for each key. Key is used as an identifier, but additionally can classify objects by other criteria based on  development stage, public accessibility, application version, etc activities.

Annotations attach arbitrary key-value information to an Kubernetes object. On the other hand labels should be used for meaningful information to match a pod with selection criteria, annotations contain less structured data. Annotations are a way of adding more metadata to an object that is not helpful for selection purposes.

What is a service role in kubernetes components

In Kubernetes, a service is a component that acts as a basic internal load balancer and ambassador for pods. A service groups together logical collections of pods that perform the same function to present them as a single entity.

  • This allows you to deploy a service that can keep track of and route to all of the backend containers of a particular type. Internal consumers only need to know about the stable endpoint provided by the service. 
  • A service’s IP address remains stable regardless of changes to the pods it routes to. By deploying a service, you easily gain discoverability and can simplify your container designs.
  • Any time you need to provide access to one or more pods to another application or to external consumers, you should to configure a service. 

For instance, if you have a set of pods running web servers that should be accessible from the internet, a service will provide the necessary abstraction. Likewise, if your web servers need to store and retrieve data, you would want to configure an internal service to give them access to your database pods.

Explain Kubelet

Each node runs services to run containers and be managed from the master. In addition to Docker, Kubelet is another key service installed there. 

It reads container manifests as YAML files that describes a pod. 

Kubelet ensures that the containers defined in the pods are started and continue running.

What Is The GKE?

Google Container Engine (GKE) is a management and orchestration system for Docker container and container clusters that run within Google’s public cloud services. Google Container Engine is based on Kubernetes, Google’s open source container management system.

Kubernetes Networking – ( Intermediate Level )

21. What happens when a master fails? What happens when a worker fails?

Kubernetes is designed to be resilient to any individual node failure, master or worker. When a master fails the nodes of the cluster will keep operating, but there can be no changes including pod creation or service member changes until the master is available. When a worker fails, the master stops receiving messages from the worker. If the master does not receive status updates from the worker the node will be marked as NotReady. If a node is NotReady for 5 minutes, the master reschedules all pods that were running on the dead node to other available nodes.


22. How does DNS work in Kubernetes?

There is a DNS server called skydns which runs in a pod in the cluster, in the kube-system namespace. That DNS server reads from etcd and can serve up dns entries for Kubernetes services to all pods. You can reach any service with the name <service>.<namespace>.svc.cluster.local. The resolver automatically searches <namespace>.svc.cluster.local dns so that you should be able to call one service to another in the same namespace with just <service>.


23. How do you build a High Availability (HA) cluster?

The only stateful part of a Kubernetes cluster is the etcd. The master server runs the controller manager, scheduler, and the API server and can be run as replicas. The controller manager and scheduler in the master servers use a leader election system, so only one controller manager and scheduler is active for the cluster at any time. So an HA cluster generally consists of an etcd cluster of 3+ nodes and multiple master nodes.

Add nodes in a HA cluster in kubernetes

Once the masters are ready, nodes can be added into the system. The node should be finished with the prerequisite configuration as a worker node in the kubeadm cluster. 

Need to start kublet

$ sudo systemctl enable kubelet && sudo systemctl start kubelet

Run  the join command as below . However, please change the master IP to the load balancer one:

// join command $ sudo kubeadm join –token <CUSTOM_TOKEN> <LOAD_BALANCER_IP>:6443 –discovery-token-ca-cert-hash sha256:<HEX_STRING>

Then go to the first master or second master to check the nodes’ status:

// see the node is added$ kubectl get nodesNAME       STATUS    ROLES     AGE       VERSIONmaster01   Ready     master    4h        v1.10.2master02   Ready     master    3h        v1.10.2node01     Ready     <none>    22s       v1.10.2

24. How do I determine the status of a Deployment?

Use kubectl get deployment <deployment>. If the DESIRED, CURRENT, UP-TO-DATE are all equal, then the Deployment has completed.


25. How do I update all my pods if the image changed but the tag is the same?

Make sure your imagePullPolicy is set to Always(this is the default). That means when a pod is deleted, a new pod will ensure it has the current version of the image. Then refresh all your pods.

The simplest way to refresh all your pods is to just delete them and they will be recreated with the latest image. This immediately destroys all your pods which will cause a service outage. Do this with kubectl delete pod -l <name>=<value> where name and value are the label selectors your deployment uses.

A better way is to edit your deployment and modify the deployment pod spec to add or change any annotation. This will cause all your pods to be deleted and rescheduled, but this method will also obey your rollingUpdate strategy, meaning no downtime assuming your rollingUpdate strategy already behaves properly. Setting a timestamp or a version number is convenient, but any change to pod annotations will cause a rolling update. For a deployment named nginx, this can be done with:

PATCH='{“spec”:{“template”:{“metadata”:{“annotations”:{“timestamp”:”‘$(date)'”}}}}}’

kubectl patch deployment nginx -p “$PATCH”

It is considered bad practice to rely on the :latest docker image tag in your deployments, because using :latest there is no way to rollback or specify what version of your image to use. It’s better to update the deployment with an exact version of the image and use –record so that you can use kubectl rollout undo deployment <deployment> or other commands to manage rollouts.

26. How do I debug a Pending pod?

A Pending pod is one that cannot be scheduled onto a node. Doing a kubectl describe pod <pod> will usually tell you why. kubectl logs <pod> can also be helpful. There are several common reasons for pods stuck in Pending:

The pod is requesting more resources than are available, a pod has set a request for an amount of CPU or memory that is not available anywhere on any node. eg. requesting a 8 CPU cores when all your nodes only have 4 CPU cores. Doing a kubectl describe node <node> on each node will also show already requested resources. ** There are taints that prevent a pod from scheduling on your nodes. ** The nodes have been marked unschedulable with kubectl cordon ** There are no Ready nodes. kubectl get nodes will display the status of all nodes.

Example:

$ kubectl get po requests-pod-3NAME             READY     STATUS    RESTARTS   AGErequests-pod-3   0/1       Pending   0          4m

Examining why a pod is stuck at Pending with kubectl describe pod

$ kubectl describe po requests-pod-3Name:       requests-pod-3Namespace:  defaultNode:       /   
…Conditions:  Type           Status    PodScheduled   False …Events:… Warning  FailedScheduling    No nodes are available                                                        that match all of the          following predicates::           Insufficient cpu (1).

27. What is KUBE proxy?

The Kubernetes network proxy runs on each node. Service cluster ips and ports are currently found through Docker-links-compatible environment variables specifying ports opened by the service proxy. There is an optional addon that provides cluster DNS for these cluster IPs. 

28. How do I debug a Pending pod?

A Pending pod is one that cannot be scheduled onto a node. Doing a kubectl describe pod <pod> will usually tell you why. kubectl logs <pod> can also be helpful. There are several common reasons for pods stuck in Pending:

** The pod is requesting more resources than are available, a pod has set a request for an amount of CPU or memory that is not available anywhere on any node. eg. requesting a 8 CPU cores when all your nodes only have 4 CPU cores. Doing a kubectl describe node <node> on each node will also show already requested resources. ** There are taints that prevent a pod from scheduling on your nodes. ** The nodes have been marked unschedulable with kubectl cordon ** There are no Ready nodes. kubectl get nodes will display the status of all nodes.

29. How do I rollback the Deployment?

If you apply a change to a Deployment with the –record flag then Kubernetes stores the previous Deployment in its history. The kubectl rollout history deployment <deployment> command will show prior Deployments. The last Deployment can be restored with the kubectl rollout undo deployment <deployment> command. In progress Deployments can also be paused and resumed.

When a new version of a Deployment is applied, a new ReplicaSet object is created which is slowly scaled up while the old ReplicaSet is scaled down. You can look at each ReplicaSet that has been rolled out with kubectl get replicaset. Each ReplicaSet is named with the format -, so you can also do kubectl describe replicaset <replicaset>.

30. What is an Ingress Controller?

An Ingress Controller is a pod that can act as an inbound traffic handler. It is a HTTP reverse proxy that is implemented as a somewhat customizable nginx. Among the features are HTTP path and service based routing and SSL termination.

Kubernetes – Expert Level

31. How do I expose a service to a host outside the cluster?

There are two ways:

  1. Set the service type to NodePort. This makes every node in the cluster listen on the specified NodePort, then any node will forward traffic from that NodePort to a random pod in the service.
  2. Set the service type to Load Balancer. This provisions a NodePort as above, but then does an additional step to provision a load balancer in your cloud(AWS or GKE) automatically. In AWS it also modifies the Auto-Scaling Group of the cluster so all nodes of that ASG are added to the ELB.

32. How does a Load Balancer service work?

A Load Balancer by default is set up as a TCP Load Balancer with your cloud provider (AWS or GKE). There is no support in bare metal or OpenStack for Load Balancer types. The Kubernetes controller manager provisions a load balancer in your cloud and puts all of your Kubernetes nodes into the load balancer. Because each node is assumed to be running kube-proxy it should be listening on the appropriate NodePort and then it can forward incoming requests to a pod that is available for the service.

Because the LoadBalancer type is by default TCP, not HTTP many higher level features of a LoadBalancer are not available. For instance health checking from the LoadBalancer to the node is done with a TCP check. HTTP X-Forwarded-For information is not available, though it is possible to use proxy protocol in AWS.

33. Is it possible to force the pod to run on a specific node?

Kubernetes by default does attempt node anti-affinity, but it is not a hard requirement, it is best effort, but will schedule multiple pods on the same node if that is the only way.

You can constrain a pod to only be able to run on particular nodes or to prefer to run on particular nodes. There are several ways to do this, and they all use label selectors to make the selection. Generally such constraints are unnecessary, as the scheduler will automatically do a reasonable placement (e.g. spread your pods across nodes, not place the pod on a node with insufficient free resources, etc.) but there are some circumstances where you may want more control on a node where a pod lands, e.g. to ensure that a pod ends up on a machine with an SSD attached to it, or to co-locate pods from two different services that communicate a lot into the same availability zone.

34. How do I get all the pods on a node?

You can use the following command to get all the pods on a node in kubernetes Cluster –

$ kubectl get po –all-namespaces  -o jsonpath='{range .items[?(@.spec.nodeName ==”nodename”)]}{.metadata.name}{“\n”}{end}’

35.Can pods mount NFS volumes?

Yes, there’s an example here of both an NFS client and server running within pods in the cluster: 

Example:

Configuring NFS Server

Define NFS server pod and NFS service:

$ kubectl create -f nfs-server-pod.yaml
$ kubectl create -f nfs-server-service.yaml

The server exports /mnt/data directory, which contains dummy index.html. Wait until the pod is running!

Configuring NFS Client

See WEB server pod, which runs a simple web server serving data from the NFS. The pod assumes your DNS is set up and the NFS service is reachable as nfs-server.default.kube.local. Edit the yaml file to supply another name or directly its IP address (use kubectl get services to get it).

Finally, define the pod:

$ kubectl create -f web-pod.yaml
Now the pod serves index.html from the NFS server:

$ curl http://<the container IP address>/
Hello World!

36. Is it possible to route traffic from outside the Kubernetes cluster directly to pods?

Yes. But one major downside of that is that ClusterIPs are implemented as iptables rules on cluster clients, so you’d lose the ability to see Cluster IPs and service changes. Because the iptables are managed by kube-proxy you could do this by running a kube-proxy, which is similar to just joining the cluster. You could make all your services Headless(ClusterIP = None), this would give your external servers the ability to talk directly to services if they could use the kubernetes dns. Headless services don’t use ClusterIPs, but instead just create a DNS A record for all pods in the service. kube-dns is run inside the cluster as a ClusterIP, so there’s a chicken and egg problem with DNS you would have to deal with.

37. Why is my pod showing “pending” status?

Pending usually means that a pod cannot be scheduled, because of a resource limitation, most commonly the cluster can’t find a node which has the available CPU and memory requests to satisfy the scheduler. kubectl describe pod <podid> will show the reason why the pod can’t be scheduled. Pods can remain in the Pending state indefinitely until the resources are available or until you reduce the number of required replicas.

38.What monitoring and metrics tools do people use for Kubernetes?

Heapster is included and its metrics are how Kubernetes measures CPU and memory in order to use horizontal pod autoscaling (HPA). Heapster can be queried directly with its REST API. Prometheus is also more full featured and popular.

39. How can containers within a pod communicate with each other?

Containers within a pod share networking space and can reach other on localhost. For instance, if you have two containers within a pod, a MySQL container running on port 3306, and a PHP container running on port 80, the PHP container could access the MySQL one through localhost:3306.

40. How do I configure credentials to download images from a private docker registry?

Create a special secret in a your namespace that provides the registry and credentials to authenticate with. Then use that secret in the spec.imagePullSecrets field of your pod specification. Private registries may require keys to read images from them. Credentials can be provided in several ways:

  • Using Google Container Registry
    • Per-cluster
    • automatically configured on Google Compute Engine or Google Kubernetes Engine
    • all pods can read the project’s private registry
  • Using AWS EC2 Container Registry (ECR)
    • use IAM roles and policies to control access to ECR repositories
    • automatically refreshes ECR login credentials
  • Using Azure Container Registry (ACR)
  • Configuring Nodes to Authenticate to a Private Registry
    • all pods can read any configured private registries
    • requires node configuration by cluster administrator
  • Pre-pulling Images
    • all pods can use any images cached on a node
    • requires root access to all nodes to setup
  • Specifying ImagePullSecrets on a Pod
    • only pods which provide own keys can access the private registry.

41. Explain the concept of Taints and Tolerations

Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.

You add a taint to a node using kubectl taint. For example,

kubectl taint nodes node1 key=value:NoSchedule

places a taint on node node1. The taint has key key, value value, and taint effect NoSchedule. This means that no pod will be able to schedule onto node1 unless it has a matching toleration.

To remove the taint added by the command above, you can run:

kubectl taint nodes node1 key:NoSchedule-

You specify a toleration for a pod in the PodSpec. Both of the following tolerations “match” the taint created by the kubectl taint line above, and thus a pod with either toleration would be able to schedule onto node1:

tolerations:
– key: “key”
  operator: “Equal”
  value: “value”
  effect: “NoSchedule”

tolerations:
– key: “key”
  operator: “Exists”
  effect: “NoSchedule”

A toleration “matches” a taint if the keys are the same and the effects are the same, and:

  • the operator is Exists (in which case no value should be specified), or
  • the operator is Equal and the values are equal

Operator defaults to Equal if not specified.

42. Explain the role of Secrets in Kubernetes

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in an image; putting it in a Secret object allows for more control over how it is used, and reduces the risk of accidental exposure.

Objects of type secret are intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image. 

Users can create secrets, and the system also creates some secrets.

To use a secret, a pod needs to reference the secret. A secret can be used with a pod in two ways: as files in a volume mounted on one or more of its containers, or used by kubelet when pulling images for the pod.

43. How can we priortize pods and preempt them.

Pods can have priority. Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible.

In Kubernetes 1.9 and later, Priority also affects scheduling order of Pods and out-of-resource eviction ordering on the Node.

Pod priority and preemption are moved to beta since Kubernetes 1.11 and are enabled by default in this release and later.

In Kubernetes versions where Pod priority and preemption is still an alpha-level feature, you need to explicitly enable it. To use these features in the older versions of Kubernetes, follow the instructions in the documentation for your Kubernetes version, by going to the documentation archive version for your Kubernetes version.

A PriorityClass is a non-namespaced object that defines a mapping from a priority class name to the integer value of the priority. The name is specified in the name field of the PriorityClass object’s metadata. The value is specified in the required value field. The higher the value, the higher the priority.

A PriorityClass object can have any 32-bit integer value smaller than or equal to 1 billion. Larger numbers are reserved for critical system Pods that should not normally be preempted or evicted. A cluster admin should create one PriorityClass object for each such mapping that they want.

PriorityClass also has two optional fields: globalDefault and description. The globalDefault field indicates that the value of this PriorityClass should be used for Pods without a priorityClassName. Only one PriorityClass with globalDefault set to true can exist in the system. If there is no PriorityClass with globalDefault set, the priority of Pods with no priorityClassName is zero.

44. Explain the Kubernetes model for connecting containers

Now that you have a continuously running, replicated application you can expose it on a network. Before discussing the Kubernetes approach to networking, it is worthwhile to contrast it with the “normal” way networking works with Docker.

By default, Docker uses host-private networking, so containers can talk to other containers only if they are on the same machine. In order for Docker containers to communicate across nodes, there must be allocated ports on the machine’s own IP address, which are then forwarded or proxied to the containers. This obviously means that containers must either coordinate which ports they use very carefully or ports must be allocated dynamically.

Coordinating ports across multiple developers is very difficult to do at scale and exposes users to cluster-level issues outside of their control. Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. We give every pod its own cluster-private-IP address so you do not need to explicitly create links between pods or mapping container ports to host ports. This means that containers within a Pod can all reach each other’s ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document will elaborate on how you can run reliable services on such a networking model.

45. Explain the concept of Storage Classes

A StorageClass provides a way for administrators to describe the “classes” of storage they offer. Different classes might map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. Kubernetes itself is unopinionated about what classes represent. This concept is sometimes called “profiles” in other storage systems.

Each StorageClass contains the fields provisioner, parameters, and reclaimPolicy, which are used when a PersistentVolume belonging to the class needs to be dynamically provisioned.

46. What are the volumes per node limits for each Cloud providers

The Kubernetes scheduler has default limits on the number of volumes that can be attached to a node:


Cloud service

Maximum volumes per node
Amazon Elastic Block Store (EBS)39
Google Persistent Disk16
Microsoft Azure Disk Storage16

47. How many types of Container Hooks are available, and explain each one of them

The hooks enable Containers to be aware of events in their management lifecycle and run code implemented in a handler when the corresponding lifecycle hook is executed.

There are two hooks that are exposed to Containers:

PostStart

This hook executes immediately after a container is created. However, there is no guarantee that the hook will execute before the container ENTRYPOINT. No parameters are passed to the handler.

PreStop

This hook is called immediately before a container is terminated. It is blocking, meaning it is synchronous, so it must complete before the call to delete the container can be sent. No parameters are passed to the handler.

48. What is a Resource Quota and Why we need them.

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that project.

Resource quotas work like this:

  • Different teams work in different namespaces. Currently this is voluntary, but support for making this mandatory via ACLs is planned.
  • The administrator creates one or more ResourceQuotas for each namespace.
  • Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.
  • If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.
  • If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements. See the walkthrough for an example of how to avoid this problem.

49. What is a Pod Security Policy and How to enable one?

A Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields.

Pod security policy control is implemented as an optional (but recommended) admission controller. PodSecurityPolicies are enforced by enabling the admission controller, but doing so without authorizing any policies will prevent any pods from being created in the cluster.

Since the pod security policy API (policy/v1beta1/podsecuritypolicy) is enabled independently of the admission controller, for existing clusters it is recommended that policies are added and authorized before enabling the admission controller.

50. Explain the Service Catalog along with its Architecture

Service Catalog is an extension API that enables applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider.

It provides a way to list, provision, and bind with external Managed Services from Service Brokers without needing detailed knowledge about how those services are created or managed.

Using Service Catalog, a cluster operator can browse the list of managed services offered by a service broker, provision an instance of a managed service, and bind with it to make it available to an application in the Kubernetes cluster.

Service Catalog uses the Open service broker API to communicate with service brokers, acting as an intermediary for the Kubernetes API Server to negotiate the initial provisioning and retrieve the credentials necessary for the application to use a managed service.

It is implemented as an extension API server and a controller, using etcd for storage. It also uses the aggregation layer available in Kubernetes 1.7+ to present its API.

Have Queries? Join https://launchpass.com/collabnix

Ajeet Raina Ajeet Singh Raina is a former Docker Captain, Community Leader and Arm Ambassador. He is a founder of Collabnix blogging site and has authored more than 570+ blogs on Docker, Kubernetes and Cloud-Native Technology. He runs a community Slack of 8900+ members and discord server close to 2200+ members. You can follow him on Twitter(@ajeetsraina).
Join our Discord Server
Table of Contents
Index