Docker 17.06 Swarm Mode: Now with built-in MacVLAN & Node-Local Networks support

ajeetraina
Estimated Reading Time: 5 minutes

Docker 17.06.0-ce-RC5 got announced 5 days back and is available for testing. It brings numerous new features & enablements under this new upcoming release. Few of my favourites includes support for Secrets on Windows,  allows specifying a secret location within the container, adds --format option to docker system df command, adds support for placement preference to docker stack deploy, adds monitored resource type metadata for GCP logging driver and adding build & engine info prometheus metrics to list a few. But one of the notable and most awaited feature include support of swarm-mode services with node-local networks such as macvlan, ipvlan, bridge and host.

Under the new upcoming 17.06 release, Docker provides support for local scope networks in Swarm. This includes any local scope network driver. Some examples of these are bridge, host, and macvlan though any local scope network driver, built-in or plug-in, will work with Swarm. Previously only swarm scope networks like overlay were supported. This is a great news for all Docker Networking enthusiasts.

A Brief Intro to MacVLAN:

Picture1

 

macvlan

In case you’re new , the MACVLAN driver provides direct access between containers and the physical network. It also allows containers to receive routable IP addresses that are on the subnet of the physical network.

MACVLAN offers a number of unique features and capabilities. It has positive performance implications by virtue of having a very simple and lightweight architecture. It’s use cases includes very low latency applications and networking design that requires containers be on the same subnet as and using IPs as the external host network.The macvlan driver uses the concept of a parent interface. This interface can be a physical interface such as eth0, a sub-interface for 802.1q VLAN tagging like eth0.10 (.10representing VLAN 10), or even a bonded host adaptor which bundles two Ethernet interfaces into a single logical interface.

To test-drive MacVLAN under Swarm Mode, I will leverage the existing 3 node Swarm Mode clusters on my VMware ESXi system. I have tested it on bare metal system and VirtualBox and it works equally great.  

[Updated: 9/27/2017 – I have added docker-stack.yml at the end of this guide to show you how to build services out of docker-compose.yml file. DO NOT FORGET TO CHECK IT OUT]

Installing Docker 17.06 on all the Nodes:

curl -fsSL https://test.docker.com > install-docker.sh
sh install-docker.sh

 

Verifying the latest Docker version:

Screen Shot 2017-06-26 at 12.51.18 AM

 

Setting up 2 Node Swarm Mode Cluster:

 

 

Attention VirtualBox Users: – In case you are using VirtualBox,  the MACVLAN driver requires the network and interfaces to be in promiscuous mode. 

A local network config is created on each host. The config holds host-specific information, such as the subnet allocated for this host’s containers. --ip-range is used to specify a pool of IP addresses that is a subset of IPs from the subnet. This is one method of IPAM to guarantee unique IP allocations.

Manager:

manager1==>sudo docker network create --config-only --subnet 100.98.26.0/24 -o parent=ens160.60 --ip-range 100.98.26.100/24 collabnet

 

Worker-1:

worker1==>sudo docker network create --config-only --subnet 100.98.26.0/24 -o parent=ens160.60 --ip-range 100.98.26.100/24 collabnet

 

 

Instantiating the macvlan network globally

Manager:

manager1==> $sudo docker network create -d macvlan --scope swarm --config-from collabnet swarm-macvlan

 

Deploying a service to the swarm-macvlan network:

Let us go ahead and deploy WordPress application. We will be creating 2 services – wordpressapp and wordpressdb1 and attach it to “swarm-macvlan” network as shown below:

Creating Backend Service:

docker service create --replicas 1 --name wordpressdb1 --network swarm-macvlan --env MYSQL_ROOT_PASSWORD=collab123 --env MYSQL_DATABASE=wordpress mysql

Let us verify if MacVLAN network scope holds this container:

 

Creating Frontend Service

Next, it’s time to create wordpress application i.e. wordpressapp

docker service create --env WORDPRESS_DB_HOST=wordpressdb1 --env WORDPRESS_DB_PASSWORD=collab123 --network swarm-macvlan --replicas 4 --name wordpressapp --publish 80:80/tcp wordpress:latest

Verify if both the services are up and running:

 

Verifying if all the containers on the master node picks up desired IP address from the subnet:

 

Docker Compose File showcasing MacVLAN Configuration

Ensure that you run the below commands to setup MacVLAN configuration for your services before you execute the above docker stack deploy CLI:

root@ubuntu-1610:~# docker network create --config-only --subnet 100.98.26.0/24 --gateway 100.98.26.1 -o parent=ens160.60 --ip-range 100.98.26.120/24 collabnet
da2912d762cbf5f5ea412e6e4d69352a3285f720e23740529af9e533c7168729
 
root@ubuntu-1610:~#docker network create -d macvlan --scope swarm --config-from collabnet swarm-macvlan
jp76lts6hbbheqlbbhggumujd

 

Verify that the containers inspection shows the correct information:

root@ubuntu-1610:~/docker101/play-with-docker/wordpress/example1# docker network inspect swarm-macvlan
[
{
“Name”: “swarm-macvlan”,
“Id”: “jp76lts6hbbheqlbbhggumujd”,
“Created”: “2017-09-27T02:12:00.827562388-04:00”,
“Scope”: “swarm”,
“Driver”: “macvlan”,
“EnableIPv6”: false,
“IPAM”: {
“Driver”: “default”,
“Options”: null,
“Config”: [
{
“Subnet”: “100.98.26.0/24”,
“IPRange”: “100.98.26.120/24”,
“Gateway”: “100.98.26.1”
}
]
},
“Internal”: false,
“Attachable”: false,
“Ingress”: false,
“ConfigFrom”: {
“Network”: “collabnet”
},
“ConfigOnly”: false,
“Containers”: {
“3c3f1ec48225ef18e8879f3ebea37c2d0c1b139df131b87adf05dc4d0f4d8e3f”: {
“Name”: “myapp2_wordpress.1.nd2m62alxmpo2lyn079x0w9yv”,
“EndpointID”: “a15e96456870590588b3a2764da02b7f69a4e63c061dda2798abb7edfc5e5060”,
“MacAddress”: “02:42:64:62:1a:02”,
“IPv4Address”: “100.98.26.2/24”,
“IPv6Address”: “”
},
“d47d9ebc94b1aa378e73bb58b32707643eb7f1fff836ab0d290c8b4f024cee73”: {
“Name”: “myapp2_db.1.cxz3y1cg1m6urdwo1ixc4zin7”,
“EndpointID”: “201163c233fe385aa9bd8b84c9d6a263b18e42893176271c585df4772b0a2f8b”,
“MacAddress”: “02:42:64:62:1a:03”,
“IPv4Address”: “100.98.26.3/24”,
“IPv6Address”: “”
}
},
“Options”: {
“parent”: “ens160”
},
“Labels”: {},
“Peers”: [
{
“Name”: “ubuntu-1610-1633ea48e392”,
“IP”: “100.98.26.60”
}
]
}
]

Docker Stack Deploy CLI:

docker stack deploy -c docker-stack.yml myapp2
Ignoring unsupported options: restart
Creating service myapp2_db
Creating service myapp2_wordpress

Verifying if the services are up and running:

root@ubuntu-1610:~/# docker stack ls
NAME SERVICES
myapp2 2
root@ubuntu-1610:~/# docker stack ps myapp2
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
nd2m62alxmpo myapp2_wordpress.1 wordpress:latest ubuntu-1610 Running Running 15 minutes ago
cxz3y1cg1m6u myapp2_db.1 mysql:5.7 ubuntu-1610 Running Running 15 minutes ago

Looking for Docker Compose file for Single Node?

 

Cool..I am going to leverage this for my Apache JMeter Setup so that I can push loads from different IPs using Docker containers.

Did you find this blog helpful?  Feel free to share your experience. Get in touch @ajeetsraina. 

If you are looking out for contribution/discussion, join me at Docker Community Slack Channel.

Know more what’s new upcoming under Docker 17.06 CE release by clicking on this link.

Clap

  • EXCELLENT Post.thanks for share..more wait.

  • Great post. I am confronting a couple of these difficulties.

  • prakash

    Hello,

    I tried your steps described above for macvlan setup in our lab, my db service is not starting. here is the service ps output. Any pointers to troubleshoot

    [root@docker-vm-1 ~]# docker service ps wordpressdb1
    ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
    6c7hefd2prw9 wordpressdb1.1 mysql:latest docker-vm-2.padc.local Ready Rejected 4 seconds ago “invalid subinterface vlan nam…”
    y2dh61x29xm6 _ wordpressdb1.1 mysql:latest docker-vm-2.padc.local Shutdown Rejected 9 seconds ago “invalid subinterface vlan nam…”
    l64i82v2x5av _ wordpressdb1.1 mysql:latest docker-vm-1.padc.local Shutdown Rejected 14 seconds ago “invalid subinterface vlan nam…”
    dmde1xsqp6na _ wordpressdb1.1 mysql:latest docker-vm-2.padc.local Shutdown Rejected 19 seconds ago “invalid subinterface vlan nam…”
    x95v97kzcxyt _ wordpressdb1.1 mysql:latest docker-vm-1.padc.local Shutdown Rejected 24 seconds ago “invalid subinterface vlan nam…”

    • Ajeet Singh Raina

      Can you let me know which Docker version are you running?

      • prakash

        thank you for responding. let me recheck on the parent interface name

        • Ajeet Singh Raina

          Approved.

          • prakash

            thanks for pointing out the parent interface name. after using the right interface name, service is deployed and each container got an IP. But my understanding of macvlan was, each container will be in isolated network – what I mean by that is – containers shouldn’t communicate with each other. for example, if I ping container 2 IP from container 1, it shouldn’t reach.

            My requirement is – I would like to build each container in an isolated network. we have a lot of customers, we want each customer to be isolated with its own network and container. Any pointers to design this.

          • Ajeet Singh Raina

            Macvlan & host network support in swarm-mode essentially are the easiest way to bring a traditional/legacy application into the container world.essentially macvlan networks will make it easy for the operators to make a container work like a “VM” with its dedicated IP-Address and direct access to the containers from their on-prem data-center networks.

            If you are looking out for the way to put each service in isolation, then all you need is isolated subnet IPs. Think of how you do that in terms of Virtual machines, the same way you do it here. So, if I want to create 2 application stack – WordPress with MYSQL as backend and another – WordPress with postgresql, you will need to put them in separate subnet.

          • prakash

            thanks. I tried below solution too
            https://docs.docker.com/engine/userguide/networking/get-started-macvlan/#dual-stack-ipv4-ipv6-macvlan-bridge-mode

            It works but there is no swarm ( service/scheduler ) way of deploying the containers. This solution has portability issue.

          • Ajeet Singh Raina

            Looking at your comments earlier, you don’t need macvlan but a general Swarm mode overlay networks. Try it out and it should work.

            See the attached diagram https://uploads.disquscdn.com/images/828a78272be820b0411e0981b70a6588c525d65484eb84350d5dda4b9fd4abd9.jpg

          • Ajeet Singh Raina

            So there are two overlay network and one can’t be accessible from another unless you allow them to.

          • Ajeet Singh Raina

            Thanks for the comments.i will go through it when I am in front of the laptop.

  • reportados

    Hi,

    I’m not too experienced with networking, but I want to use a private network interface (eth1) for my service to communicate over. DigitalOcean (as well as other cloud providers) allows for private networking, which enables another net interface, eth1 (eth0 for public). My swarm nodes are deployed in various regions, so I must use the public network interface (eth0) to communicate for the service, which is done via an overlay network, but for the same service in the same region, I wanted to create another overlay network which is for communication in the same region. The problem is that there is no way to specify a network interface/ip to use for the overlay network. I’ve looked into several different network drivers, and it seems like macvlan does what I want.

    I tried `docker network create –config-only –subnet 10.132.0.0/16 –gateway 10.132.0.1 -o parent=eth1 –ip-range 10.132.150.30/24 myregion` for several nodes, and on the manager node, I do
    `docker network create -d macvlan –scope swarm –config-from myregion region-net`

    However, I cannot ping any of my container ips from inside the container, or ping any of the hosts running the containers from inside the container. Also, it appears that all the containers get the same ip address, 10.132.150.0. My manager private ip address is 10.132.27.71. I chose 150 to be part of the ip address for the containers because from my understanding, as long as it’s within the same subnet, it can be discovered and routed. I do not have control over the private network’s gateway, which is apparently 10.132.0.1 according to digitalocean, so it seems like I cannot do any routing.

    Did I misconfigure something? Am I misunderstanding how macvlan works? Most of the examples I’ve seen run the macvlan driver on a single host with multiple containers, but your example seems to run on multiple hosts with a container (potentially more containers as well). Also, does macvlan support the DNS service name lookup/load balancing? Say, I have a couple of replicas for a service named, db. And from inside a container running in the same network, I ping db, will it automatically go through the routing mesh, choosing any of the nodes that is running the db service like the overlay driver?

    Thanks.

  • bb

    I wrote up a small tutorial on running swarm + macvlan with consul + autoscaling https://gist.github.com/killcity/d95e231ea98af4432f200e6d7ceb4961 if anyone is interested.

    • Ajeet Singh Raina

      Cool..

  • Gabriel Sousa

    how can containers ( from different host ) communication which other

    • Ajeet Singh Raina

      Isn’t this blog talking about the same?? Do let me know if you have specific queries?

    • Ajeet Singh Raina

      You will need to set it up on either bare metal system or VirtualBox. Tested it on Virtualbox and it works fine.

  • Christian Reiter

    Hello!

    First of all, thank you very much for this great tutorial.

    To form a JBoss (Jgroups) cluster in Docker I need multicast between containers of a Docker Swarm and therefore I need to add one MACVLAN interface to the containers running JBoss. So far so good, works great. But:

    The containers which have the additional MACVLAN interface are no longer connected to the outside world, even if I add additional other overlay networks to them.

    Does anybody know how I can have one MACVLAN interface plus additional access to the outside world (e.g. Internet) via the standard overlay networks from Docker Swarm?