Ajeet Raina Docker Captain, ARM Innovator & Docker Bangalore Community Leader.

Kubernetes Networking 101

8 min read

Thanks to Collabnix community member Divyajeet Singh   for contributing this content for KubeLabs – The #1 Kubernetes Resources for all Levels. Do you have anything exciting to share with Collabnix community? Do visit Collabnix community Slack and we might feature you in Collabnix website.

Kubernetes is a technology that helps you get the most out of your hardware. Containers are deployed on several nodes, making sure that every CPU cycle, every byte of memory, and every block of storage is not wasted. However, this is no easy task. Networking in Kubernetes is a central part of Kubernetes, but it can be challenging to understand exactly how it is expected to work. There are 4 distinct networking problems to address:

• Highly-coupled container-to-container communications: this is solved by pods and localhost communications.
• Pod-to-Service communications: this is covered by services.
• External-to-Service communications: this is covered by services.
• Pod-to-Pod communications: this is the primary focus of this lab.

Kubernetes Networking Rules

Kubernetes is highly modular and open-source project. Several components were left to the community to develop. In particular, implementing a cluster-networking solution must conform to a set of high-level rules. They can be summarized as follows:

  • Pods scheduled on the same node must be able to communicate with other pods without using NAT (Network Address Translation).
  • All system daemons (background processes, for example, Kubelet) running on a particular node can communicate with the pods running on the same node.
  • Pods that use the host network must be able to contact all other pods on all other nodes without using NAT. Notice that the host network is only supported on Linux hosts.

Cluster Networking

So, as you can see Kubernetes eliminates the need for NAT or link containers. In Kubernetes every Pod gets its own IP address. This means you do not need to explicitly create links between Pods and you almost never need to deal with mapping container ports to host ports. This creates a clean, backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.

There are a number of networking models that adhere to the above rules. In this article, we’ll select some of them for discussion. But, before listing the different network plugin examples, let’s have a quick overview of some important Kubernetes networking terms.

What Is An Overlay Network?

In general, we can define networks as underlay and overlay types:

Underlay network

Underlay network is closer to the physical layer or you can say that, the network without SDN capabilities. It includes switches, routers, VLANs and so on. It is the basis on which overlay networks are built. It tends to be less scalable due to technical limitations. However, since it’s closer to the actual hardware, it is slightly faster than an overlay.

Overlay network

Overlay network refers to the virtual network layer (SDN). In this type, you’ll hear terms like veth (virtual eth or virtual network interface), and VxLAN. It is designed to be highly scalable than the underlying network. For example, while VLANs in the underlying network support only 4096 identifiers, VxLAN can reach up to 16 million ones.

Kubernetes supports both networking models, so you can base your model of choice on other factors than whether or not the cluster can handle it.

What is a Container Network Interface (CNI)?

A CNI is simply a link between the container runtime (like Docker or rkt) and the network plugin. The network plugin is nothing but the executable that handles the actual connection of the container to or from the network, according to a set of rules defined by the CNI. So, to put it simply, a CNI is a set of rules and Go libraries that aid in container/network-plugin integration.

All of the CNIs can be deployed by simply running a pod or a daemonset that launches and manages their daemons. Let’s have a look now at the most well-known Kubernetes networking solutions

AWS VPC CNI for Kubernetes

The AWS VPC CNI offers integrated AWS Virtual Private Cloud (VPC) networking for Kubernetes clusters. This CNI plugin offers high throughput and availability, low latency, and minimal network jitter. Additionally, users can apply existing AWS VPC networking and security best practices for building Kubernetes clusters. This includes the ability to use VPC flow logs, VPC routing policies, and security groups for network traffic isolation.

Using this CNI plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network. The CNI allocates AWS Elastic Networking Interfaces (ENIs) to each Kubernetes node and using the secondary IP range from each ENI for pods on the node. The CNI includes controls for pre-allocation of ENIs and IP addresses for fast pod startup times and enables large clusters of up to 2,000 nodes.

Additionally, the CNI can be run alongside Calico for network policy enforcement. The AWS VPC CNI project is open source with documentation on GitHub.

Azure CNI for Kubernetes

Azure CNI is an open source plugin that integrates Kubernetes Pods with an Azure Virtual Network (also known as VNet) providing network performance at par with VMs. Pods can connect to peered VNet and to on-premises over Express Route or site-to-site VPN and are also directly reachable from these networks. Pods can access Azure services, such as storage and SQL, that are protected by Service Endpoints or Private Link. You can use VNet security policies and routing to filter Pod traffic. The plugin assigns VNet IPs to Pods by utilizing a pool of secondary IPs pre-configured on the Network Interface of a Kubernetes node.

Azure CNI is available natively in the Azure Kubernetes Service (AKS).


Calico is a scalable and secure networking plugin. It can be used to manage and secure network policies not only for Kubernetes, but also for containers, virtual machines, and even bare metal servers. Calico works on Layer 3 of the network stack. It works by implementing a vRouter (as opposed to a vSwitch) on each node. Since it is working on L3, it can easily use the Linux kernel’s native forwarding functionality. The Felix agent is responsible for programming L3 Forwarding Information base with the IP addresses of the pods scheduled on the node where it is running.

Calico uses vRouters to allow pods to connect to each other across different nodes using the physical network (underlay). It does not use overlay, tunneling or VRF tables. Also, it does not require NAT since each pod can be assigned a public IP address that is accessible from anywhere as long as the security policy permits it.

Deployment differs based on the type of environment or the cloud provider where you’ll be hosting your cluster. This document contains all the supported Calico deployment methods.


Cilium uses layers 3, 4 (network), and layer 7 (application) to function. It brings a solution that is not only aware of the packets that pass through, but also the application and protocol (for example, HTTP) that those packets are using. Having such a level of inspection allows Cilium to control and enforce network and application security policies. Be aware, though that for this plugin to work, you must be using a Linux kernel that is equal to or higher than 4.8. That’s because Cilium uses a new kernel feature Berkeley Packet Filter (BPF), which can replace iptables.

Cilium runs a daemon called cillium-agent on each node. It compiles the BPF filters and transfers them to the kernel for further processing.

Weave Net from WeaveWorks

Weave Net is an easy-to-use, resilient, and fast-growing network plugin that can be used for more than just container networking. When installed, Weave Net creates a virtual router on each host (called peer). Those routers start communicating with each other to establish protocol handshake and, later, learn the network topology. The plugin also creates a bridge interface on each host. All pods get attached to this interface, and they are assigned IP addresses and netmasks. Within the same node, Weave Net uses the kernel to move packets from one pod to another. This protocol is called the fast data path. When the packet is destined to a pod on another host, the plugin uses the sleeve protocol, in which UDP is used to contact the router on the destination host to transfer packets. Subsequently, those packets are captured by the kernel and passed to the target pod.

One way to install Weave Net on a Kubernetes cluster is to apply a daemonset which will automatically install the necessary containers for running the plugin on each node. Once up and running, all pods will use this network for their communication. The peers are self-configuring, so you can add more nodes to the cluster and they’ll use the same network without further configuration from your side.


Flannel is a networking plugin created by CoreOS. It implements cluster networking in Kubernetes by creating an overlay network. It starts a daemon called flanneld on each node. This daemon runs under a pod whose name starts with kube-flannel-ds-*. When assigning IP addresses, Flannel allocates a small subset of IPs of each host (by default, 10.244.X.0/24). This subset is brought from a larger, preconfigured address space. This subset is used to assign an IP address of each pod on the node.

Flannel uses Kubernetes API server or the cluster’s etcd database directly to store information about the assigned subnets, network configuration, and the host IP address.

Packet forwarding among hosts is done through several protocols like UDP and VXLAN.

LAB – Weave Net Implementation�

Open https://labs.play-withk8s.com and login with your Github / Docker account and launch at least 2 instances simultaneously. Once you have the instances created run the follow kubeadm command to create a kubernetes cluster:

 kubeadm init --apiserver-advertise-address $(hostname -i)

Command completion may take some time, you will have following output once the command gets completed.

kubeadm join --token l6m3z8.qa8xj647tip4on1y \
    --discovery-token-ca-cert-hash sha256:10f8323fcd8c9f7f4921a8440776638494c939d11b8c6ba1aab5b41d4b65d040

Note:- Token and Certificate hash will vary.

You can use this output to join worker nodes to your kubernetes cluster, but make sure you have port 6443 opened for the VM in your Infra. Now lets check the CNI, our kubernetes is using at this moment.

[node1 ~]$ kubectl get po -A
NAMESPACE     NAME                            READY   STATUS    RESTARTS   AGE
kube-system   coredns-6dcc67dcbc-jhppl        1/1     Running   0          2m43s
kube-system   coredns-6dcc67dcbc-nbg7m        1/1     Running   0          2m43s
kube-system   etcd-node1                      1/1     Running   0          93s
kube-system   kube-apiserver-node1            1/1     Running   0          107s
kube-system   kube-controller-manager-node1   1/1     Running   0          92s
kube-system   kube-proxy-kn8kn                1/1     Running   0          2m43s
kube-system   kube-scheduler-node1            1/1     Running   0          89s

Currently we cannot see any CNI in the cluster. Now lets plugin weavnet CNI in our cluster. To do it run the following command:-

[node1 ~]$  kubectl apply -n kube-system -f \
>     "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 |tr -d '\n')"
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.apps/weave-net created
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-6dcc67dcbc-jhppl   0/1     Running   0          37s
kube-system   coredns-6dcc67dcbc-nbg7m   1/1     Running   0          37s
kube-system   kube-proxy-kn8kn           1/1     Running   0          37s
kube-system   weave-net-qlwp6            2/2     Running   0          33s

Now we can see the weave net CNI in our cluster. To install a different CNI plugin (Calico) run the following commands:

[node1 ~]$  kubectl delete -n kube-system -f     "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 |tr -d '\n')"
serviceaccount "weave-net" deleted
clusterrole.rbac.authorization.k8s.io "weave-net" deleted
clusterrolebinding.rbac.authorization.k8s.io "weave-net" deleted
role.rbac.authorization.k8s.io "weave-net" deleted
rolebinding.rbac.authorization.k8s.io "weave-net" deleted
daemonset.apps "weave-net" deleted
[node1 ~]$ kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
[node1 ~]$ kubectl get po -A
NAMESPACE    NAME                                       READY  STATUS   RESTARTS  AGE
kube-system  calico-kube-controllers-6ff88bf6d4-tgtzb   1/1    Running  0         2m45s
kube-system  calico-node-24h85                          1/1    Running  0         2m43s
kube-system  coredns-846jhw23g9-9af73                   1/1    Running  0         4m5s
kube-system  coredns-846jhw23g9-hmswk                   1/1    Running  0         4m5s
kube-system  etcd-jbaker-1                              1/1    Running  0         6m22s
kube-system  kube-apiserver-jbaker-1                    1/1    Running  0         6m12s
kube-system  kube-controller-manager-jbaker-1           1/1    Running  0         6m16s
kube-system  kube-proxy-8fzp2                           1/1    Running  0         5m16s
kube-system  kube-scheduler-jbaker-1                    1/1    Running  0         5m41s

Don’t forget to clean up.

Have Queries? Join https://launchpass.com/collabnix

Ajeet Raina Docker Captain, ARM Innovator & Docker Bangalore Community Leader.

55 Replies to “Kubernetes Networking 101”

  1. hey there and thank you for your information – I’ve definitely picked up something new from right here. I did however expertise several technical points using this site, since I experienced to reload the website lots of times previous to I could get it to load correctly. I had been wondering if your hosting is OK? Not that I am complaining, but slow loading instances times will often affect your placement in google and can damage your high quality score if ads and marketing with Adwords. Well I’m adding this RSS to my e-mail and could look out for much more of your respective intriguing content. Make sure you update this again very soon..

  2. I have been surfing online greater than 3 hours lately, yet I by no means discovered any fascinating article like yours. It¦s lovely price enough for me. Personally, if all webmasters and bloggers made just right content as you did, the internet will likely be a lot more useful than ever before.

  3. Hello there! I could have sworn I’ve been to this site before but after reading through some of the post I realized it’s new to me. Anyhow, I’m definitely glad I found it and I’ll be bookmarking and checking back frequently!

  4. This is really interesting, You are a very skilled blogger. I’ve joined your feed and look forward to seeking more of your fantastic post. Also, I’ve shared your site in my social networks!

  5. Hello very cool web site!! Man .. Excellent .. Amazing .. I’ll bookmark your blog and take the feeds also…I’m happy to seek out numerous useful info right here within the publish, we want develop extra techniques on this regard, thank you for sharing. . . . . .

  6. I believe this is among the so much vital info for me. And i am glad reading your article. However want to commentary on few common things, The site style is wonderful, the articles is in point of fact nice : D. Good task, cheers

  7. It’s the best time to make some plans for the future and it is time to be happy. I’ve read this post and if I could I wish to suggest you some interesting things or tips. Maybe you could write next articles referring to this article. I want to read even more things about it!

  8. Excellent read, I just passed this onto a colleague who was doing a little research on that. And he actually bought me lunch as I found it for him smile Therefore let me rephrase that: Thank you for lunch!

  9. Just desire to say your article is as surprising. The clearness on your post is just spectacular and that i can think you’re a professional on this subject. Well along with your permission let me to take hold of your feed to stay up to date with drawing close post. Thank you one million and please keep up the enjoyable work.

  10. naturally like your website but you have to check the spelling on quite a few of your posts. Many of them are rife with spelling issues and I find it very bothersome to tell the truth nevertheless I’ll certainly come back again.

  11. Whats up very nice blog!! Man .. Beautiful .. Superb .. I will bookmark your website and take the feeds also…I’m glad to find so many useful information here within the post, we’d like work out extra strategies in this regard, thank you for sharing. . . . . .

  12. I loved as much as you will receive carried out right here. The sketch is attractive, your authored subject matter stylish. nonetheless, you command get got an shakiness over that you wish be delivering the following. unwell unquestionably come more formerly again since exactly the same nearly very often inside case you shield this increase.

  13. I have been browsing online more than 3 hours lately, but I never found any fascinating article like yours. It is beautiful value sufficient for me. In my opinion, if all webmasters and bloggers made good content material as you did, the internet can be a lot more helpful than ever before.

  14. I have recently started a site, the info you offer on this site has helped me greatly. Thanks for all of your time & work. “My dear and old country, here we are once again together faced with a heavy trial.” by Charles De Gaulle.

  15. Terrific post however , I was wanting to know if you could write a litte more on this subject? I’d be very thankful if you could elaborate a little bit more. Kudos!

  16. I have not checked in here for a while as I thought it was getting boring, but the last several posts are good quality so I guess I will add you back to my everyday bloglist. You deserve it my friend 🙂

  17. I discovered your blog site on google and check a few of your early posts. Continue to keep up the very good operate. I just additional up your RSS feed to my MSN News Reader. Seeking forward to reading more from you later on!…

  18. Hey! Do you know if they make any plugins to protect against hackers? I’m kinda paranoid about losing everything I’ve worked hard on. Any tips?

  19. Aw, this was a very nice post. In concept I want to put in writing like this moreover – taking time and precise effort to make a very good article… but what can I say… I procrastinate alot and on no account seem to get one thing done.

  20. Definitely believe that which you said. Your favorite reason appeared to be on the net the easiest thing to be aware of. I say to you, I definitely get irked while people consider worries that they plainly don’t know about. You managed to hit the nail upon the top and defined out the whole thing without having side effect , people can take a signal. Will likely be back to get more. Thanks

  21. Thanks for sharing superb informations. Your website is so cool. I’m impressed by the details that you have on this website. It reveals how nicely you understand this subject. Bookmarked this website page, will come back for more articles. You, my friend, ROCK! I found just the information I already searched everywhere and simply could not come across. What a great site.

  22. This is the precise weblog for anybody who desires to find out about this topic. You realize so much its virtually exhausting to argue with you (not that I really would need…HaHa). You positively put a new spin on a subject thats been written about for years. Great stuff, just nice!

  23. Wow! This can be one particular of the most useful blogs We’ve ever arrive across on this subject. Actually Wonderful. I’m also an expert in this topic so I can understand your hard work.

  24. Undeniably believe that that you stated. Your favourite justification seemed to be at the web the simplest thing to understand of. I say to you, I definitely get annoyed whilst other folks consider worries that they plainly don’t realize about. You managed to hit the nail upon the top as neatly as outlined out the whole thing with no need side-effects , other people could take a signal. Will probably be back to get more. Thanks

  25. certainly like your web-site but you need to check the spelling on quite a few of your posts. Several of them are rife with spelling problems and I find it very bothersome to tell the truth nevertheless I will surely come back again.

  26. Excellent beat ! I would like to apprentice at the same time as you amend your website, how can i subscribe for a weblog website? The account aided me a appropriate deal. I had been a little bit acquainted of this your broadcast provided vivid transparent concept

  27. Howdy just wanted to give you a quick heads up and let you know a few of the images aren’t loading correctly. I’m not sure why but I think its a linking issue. I’ve tried it in two different browsers and both show the same outcome.

  28. Thank you for sharing superb informations. Your website is so cool. I am impressed by the details that you have on this site. It reveals how nicely you perceive this subject. Bookmarked this web page, will come back for extra articles. You, my friend, ROCK! I found just the info I already searched everywhere and simply couldn’t come across. What a perfect site.

  29. This is very interesting, You’re a very skilled blogger. I’ve joined your rss feed and look forward to seeking more of your magnificent post. Also, I’ve shared your web site in my social networks!

  30. You can definitely see your skills within the paintings you write. The sector hopes for even more passionate writers like you who aren’t afraid to say how they believe. At all times follow your heart. “History is the version of past events that people have decided to agree upon.” by Napoleon.

Leave a Reply

Your email address will not be published.

© Copyright Collabnix Inc

Built for Collabnix Community, by Community