Ajeet Raina Docker Captain, ARM Innovator & Docker Bangalore Community Leader.

Demystifying Service Discovery under Docker Engine 1.12.0

4 min read

Prior to Docker 1.12 release, setting up Swarm cluster needed some sort of service discovery backend. There are multiple discovery backends available like hosted discovery service, using a static file describing the cluster, etcd, consul, zookeeper or using static list of IP address.

pic-intro

Thanks to Docker 1.12 Swarm Mode, we don’t have to depend upon these external tools and complex configurations. Docker Engine 1.12 runs it’s own internal DNS service to route services by name.Swarm manager nodes assign each service in the swarm a unique DNS name and load balances running containers. You can query every container running in the swarm through a DNS server embedded in the swarm.

How does it help?

When you create a service and provide a name for it, you can use just that name as a target hostname, and it’s going to be automatically resolved to the proper container IP of the service. In short, within the swarm, containers can simply reference other services via their names and the built-in DNS will be used to find the appropriate IP and port automatically. It is important to note that if the service has multiple replicas, the requests would be round-robin load-balanced. This would still work if you didn’t forward any ports when you created your docker services.

Pic10

Embedded DNS is not a new concept. It was first included under Docker 1.10 release. Please note that DNS lookup for containers connected to user-defined networks works differently compared to the containers connected to default bridge network. As of Docker 1.10, the docker daemon implements an embedded DNS server which provides built-in service discovery for any container created with a valid name or net-alias or aliased by link. Moreover,container name configured using --name is used to discover a container within an user-defined docker network. The embedded DNS server maintains the mapping between the container name and its IP address (on the network the container is connected to).

How does Embedded DNS resolve unqualified names?

Pic22

 

With Docker 1.12 release, a new API called “service” is being included which clearly talks about the functionality of service discovery.  It is important to note that Service discovery is scoped within the network. What it really means is –  If you have redis application and web client as two separate services , you combine into single application and put them into same network.If you try build your application in such a way that you are trying to reach to redis through name “redis”,it will always resolve to name “redis”. Reason – both of these services are part of the same network. You don’t need to be inside the application trying to resolve this service using FQDN. Reason – FQDN name is not going to be portable which in turn, makes your application non-portable.

Internally, there is a listener opened inside the container itself. If we try to enter into the container which is providing a service discovery and look at /etc/resolv.conf, we will find that the nameserver entry holds something really different like 127.0.0.11.This is nothing but a loopback address. So, whenever resolver tried to resolve, it will resolve to 127.0.0.11 and this request is rightly trapped.

Pic-12

Once this request is trapped, it is sent to particular random UDP / TCP port currently being listened under the docker daemon. Consequently, the socket is to be created inside the namespace. When DNS server and daemon gets the request, it knows that this is coming from which specific network, hence gets aware of  the context of from where it is coming from.Once it knows the context, it can generate the appropriate DNS response.

To demonstrate Service Discovery  under Docker 1.12, I have upgraded Docker 1.12.rc5 to 1.12.0 GA version. The swarm cluster look like:

Pico01

I have created a network called “collabnet” for the new services as shown below:

Pic-2

Let’s create a service called “wordpressdb” under collabnet network :

pico-mysql

You can list the running tasks(containers) and the node on which these containers are running on:

Pic-4

Let’s create another service called “wordpressapp” under the same network:

pico-app

Now, we can list out the number of services running on our swarm cluster as shown below.

pico-2

I have scaled out the number of wordpressapp and wordpressdb just for demonstration purpose.

Let’s consider my master node where I have two of the containers running as shown below:

Pico-1

I can reach out one service(wordpressapp) from another service(wordpressapp) through just service-name as shown below:

pico-last

Also, I can reach out to particular container by its name from other container running different service but on the same network. As shown below, I can reach out to wordpressapp.3.6f8bthp container via wordpressdb.7.e62jl57qqu running wordpressdb.

pico-tasktoo

The below picture depicts the Service Discovery in a nutshell:

Pic23

Every service has Virtual IP(VIP) associated which can be derived as shown below:

pic-list

As shown above, each service has an IP address and this IP address maps to multiple container IP address associated with that service. It is important to note that service IP associated with a service does not change even though containers associated with the service dies/ restarts.

Few important points to remember:

  • VIP based services use Linux IPVS load balancing to route to the backend containers. This works only for TCP/UDP protocols. When you use DNS-RR mode services don’t have a VIP allocated. Instead service names resolves to one of the backend container IPs randomly.
  • Ping not working for VIP is as designed. Technically, IPVS is a TCP/UDP load-balancer, while ping uses ICMP and hence IPVS is not going to load-balance the ping request.
  • For VIP based services the reason ping works on the local node is because the VIP is added a 2nd IP address on the overlay network interface
  • You can any of the tools like  dig, nslookup or wget -O- <service name> to demonstrate the service discovery functionality

Below picture depicts that the network is the scope of service discoverability which means that when you have a service running on one network , it is scoped to that network and won’t be able to reach out to different service running on different network(unless it is part of that network).

SD

Let’s dig little further introducing Load-balancing aspect too. To see what is basically enabling the load-balancing functionality, we can go into sandbox of each containers and see how it has been resolved.

Let’s pick up the two containers running on the master node. We can see the sandbox running through the following command:

pico-namespace

Under /var/run/docker/netns, you will find various namespaces. The namespaces marked with x-{id} represents network namespace managed by the overlay network driver for its operation (such as creating a bridge, terminating vxlan tunnel, etc…). They don’t represent the container network namespace. Since it is managed by the driver, it is not recommended to manipulate anything within this namespace. But if you are curious on the deep dive, then you can use the “nsenter” tool to understand more about this internal namespace.

We can enter into sandbox through the nsenter utility:

pico-mangle

In case you faced an error stating “nsenter: reassociate to namespace ‘ns/net’ failed: Invalid argument”, I suggest to look at this workaround.

10.0.3.4 service IP is marked 0x108 using iptables OUTPUT chain. ipvs uses this marking and load balances it to containers 10.0.3.5 and 10.0.3.6 as shown below:

ipvs

Here are key takeaways from this entire post:

Pic34

In my next blog post, I am going to deep dive into Load-Balancing aspect of Swarm Mode. Thanks for reading.

Have Queries? Join https://launchpass.com/collabnix

Ajeet Raina Docker Captain, ARM Innovator & Docker Bangalore Community Leader.

56 Replies to “Demystifying Service Discovery under Docker Engine 1.12.0”

    1. Whenever any node running the database is lost, all the containers gets rescheduled to other available nodes. in case any container running in the node fails, the engine starts the new containers automatically with the new ID.

  1. Hi would you mind letting me know which web host you’re using?
    I’ve loaded your blog in 3 completely different internet browsers and I must say this blog loads a lot quicker then most.
    Can you suggest a good internet hosting provider at
    a reasonable price? Thanks, I appreciate it! http://yahoo.co.uk

  2. Great article. One question that maybe you can help me with.

    You’ve mentioned :
    VIP based services use Linux IPVS load balancing to route to the backend containers. This works only for TCP/UDP protocols. When you use DNS-RR mode services don’t have a VIP allocated. Instead service names resolves to one of the backend container IPs randomly.

    Is there any smooth way how to achieve DNS based load balancing for services running in the swarm cluster without any 3rd party tooling?

    If you check my playground:
    https://github.com/nohaapav/docker-swarm
    I’m running docker swarm (3managers/3workers) with 2 services A and B. Each is scaled to 3 tasks.

    Test results:

    Test round-robin (calling service A only) works as charm:
    I’m 71d297bcdbdf
    I’m 8b18f5ede3e6
    I’m 15e4e5b64372
    I’m 71d297bcdbdf
    I’m 8b18f5ede3e6
    I’m 15e4e5b64372

    Test DNS service discovery(calling service A redirecting to B):
    Redirecting to serviceB. Response: I’m 019b0bdbce52
    Redirecting to serviceB. Response: I’m 019b0bdbce52
    Redirecting to serviceB. Response: I’m 019b0bdbce52
    Redirecting to serviceB. Response: I’m 019b0bdbce52
    Redirecting to serviceB. Response: I’m 019b0bdbce52
    Redirecting to serviceB. Response: I’m 019b0bdbce52

    Basically even if service A is loadbalanced fine when hit all its tasks are pointing to the same service B task when redirected.

  3. This article is very attractive to people just like me.
    It’s not just thought-provoking, it draws you in right from the start.
    This is well-written articles. The views listed here are also encouraging to me.
    Thanks.

  4. Wow! This blog looks just like my old one!

    It’s on a entirely different subject but it has pretty much the same
    layout and design. Great choice of colors!

  5. Thank you for such a informative article. Really appreciate your efforts. keep writing and help us grow together ..

  6. Nice post. I was checking continuously this blog and I’m impressed! Extremely useful info specially the last part 🙂 I care for such info a lot. I was seeking this particular information for a long time. Thank you and good luck.

  7. I needed to write you this little bit of word to finally thank you so much yet again considering the fantastic tips you’ve discussed in this case. It is simply extremely generous with you to grant easily all a few individuals could possibly have supplied as an e book in order to make some cash for their own end, especially considering the fact that you might have tried it if you desired. These pointers likewise served like the great way to fully grasp other people online have similar keenness the same as my personal own to understand a good deal more with reference to this issue. I am certain there are millions of more pleasurable sessions up front for individuals who discover your blog.

  8. You actually make it appear really easy together with your presentation however I in finding this matter to be actually one thing that I think I might by no means understand. It seems too complex and very extensive for me. I’m having a look ahead to your next submit, I¦ll attempt to get the cling of it!

  9. Hiya, I am really glad I’ve found this info. Today bloggers publish just about gossips and internet and this is actually irritating. A good web site with exciting content, this is what I need. Thanks for keeping this website, I will be visiting it. Do you do newsletters? Can not find it.

  10. Undeniably consider that which you said. Your favourite justification seemed to be at the web the easiest thing to consider of. I say to you, I certainly get annoyed at the same time as other people think about concerns that they just do not realize about. You controlled to hit the nail upon the highest and defined out the whole thing with no need side effect , people can take a signal. Will likely be again to get more. Thanks

  11. I¦ve been exploring for a bit for any high-quality articles or weblog posts on this sort of space . Exploring in Yahoo I ultimately stumbled upon this web site. Reading this info So i¦m satisfied to express that I’ve a very just right uncanny feeling I found out just what I needed. I so much indisputably will make sure to don¦t overlook this website and give it a look regularly.

  12. excellent post, very informative. I wonder why the other specialists of this sector do not notice this. You must continue your writing. I am sure, you have a huge readers’ base already!

  13. Thank you for the sensible critique. Me and my neighbor were just preparing to do a little research on this. We got a grab a book from our area library but I think I learned more from this post. I’m very glad to see such fantastic information being shared freely out there.

  14. Thanx for the effort, keep up the good work Great work, I am going to start a small Blog Engine course work using your site I hope you enjoy blogging with the popular BlogEngine.net.Thethoughts you express are really awesome. Hope you will right some more posts.

  15. Great ?V I should certainly pronounce, impressed with your website. I had no trouble navigating through all the tabs and related info ended up being truly easy to do to access. I recently found what I hoped for before you know it at all. Quite unusual. Is likely to appreciate it for those who add forums or anything, web site theme . a tones way for your client to communicate. Excellent task..

  16. naturally like your web site but you have to check the spelling on quite a few of your posts. Several of them are rife with spelling issues and I find it very bothersome to tell the truth nevertheless I’ll surely come back again.

  17. Hi, Neat post. There is an issue along with your web site in internet explorer, might check this?K IE still is the marketplace chief and a huge component to people will miss your excellent writing because of this problem.

  18. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point. You clearly know what youre talking about, why waste your intelligence on just posting videos to your blog when you could be giving us something enlightening to read?

  19. Thanks a bunch for sharing this with all of us you actually know what you’re talking about! Bookmarked. Please also visit my site =). We could have a link exchange arrangement between us!

  20. Youre so cool! I dont suppose Ive read something like this before. So good to find somebody with some unique ideas on this subject. realy thank you for starting this up. this website is something that’s wanted on the net, somebody with a little originality. useful job for bringing one thing new to the internet!

  21. I loved as much as you will receive carried out right here. The sketch is tasteful, your authored material stylish. nonetheless, you command get bought an shakiness over that you wish be delivering the following. unwell unquestionably come further formerly again as exactly the same nearly very often inside case you shield this hike.

  22. You could certainly see your enthusiasm in the work you write. The world hopes for even more passionate writers such as you who are not afraid to say how they believe. All the time go after your heart.

  23. I like what you guys are up also. Such clever work and reporting! Keep up the superb works guys I’ve incorporated you guys to my blogroll. I think it’ll improve the value of my site 🙂

Leave a Reply

Your email address will not be published.