Kong: Large DNS delay for APIs on internal docker networks

Created on 30 Nov 2017 · 7Comments · Source: Kong/kong

Summary

If you run kong with docker-compose and try to add a container inside the same docker network as an API, every first request to that API will take 20s (+- a few ms). This happens only if you try to resolve the container using its docker DNS alias (in docker-compose the name of the service by default).

If the client is added with its internal docker IP address, it works normally.

Subsequent request to the same API within a short time period also work normally.

Steps To Reproduce

Clone https://github.com/lodrantl/kong-dns-timeout
Follow instuctions in the README

Additional Details & Logs

Kong version (0.11.1 and 0.11.2)
Ubuntu 17.10
Docker version 17.11.0-ce
docker-compose version 1.16.1

Example log of such a request:

{  
   "api":{  
      "created_at":1512002942599,
      "strip_uri":true,
      "id":"1e23c1fa-02d6-4bff-bc37-301e55291f13",
      "name":"test",
      "http_if_terminated":false,
      "https_only":false,
      "upstream_url":"http:\/\/nginx",
      "uris":[  
         "\/test"
      ],
      "preserve_host":false,
      "upstream_connect_timeout":60000,
      "upstream_read_timeout":60000,
      "upstream_send_timeout":60000,
      "retries":5
   },
   "request":{  
      "querystring":{  

      },
      "size":"82",
      "uri":"\/test",
      "request_uri":"http:\/\/localhost:8000\/test",
      "method":"GET",
      "headers":{  
         "host":"localhost:8000",
         "accept":"*\/*",
         "user-agent":"curl\/7.55.1"
      }
   },
   "client_ip":"172.26.0.1",
   "latencies":{  
      "request":20005,
      "kong":20003,
      "proxy":0
   },
   "response":{  
      "headers":{  
         "content-type":"text\/html; charset=UTF-8",
         "date":"Thu, 30 Nov 2017 01:05:50 GMT",
         "connection":"close",
         "via":"kong\/0.11.2",
         "content-length":"612",
         "x-kong-proxy-latency":"20004",
         "server":"nginx\/1.13.7",
         "x-kong-upstream-latency":"0",
         "etag":"\"5a1437f4-264\"",
         "accept-ranges":"bytes",
         "last-modified":"Tue, 21 Nov 2017 14:28:04 GMT"
      },
      "status":200,
      "size":"940"
   },
   "tries":[  
      {  
         "balancer_latency":0,
         "port":80,
         "ip":"172.26.0.2"
      }
   ],
   "started_at":1512003930601
}

Source

lodrantl

Most helpful comment

Thanks for your help. I raised an issue with moby. https://github.com/moby/moby/issues/35801

lodrantl on 14 Dec 2017

👍2

All 7 comments

I identified the source of the problem.

Kong will first try to resolve the given address as a SRV record, but this takes around 20s to timeout inside a docker container.

Reproduce:

docker create network test_net
docker run -it --network=test_net ubuntu
apt-get update && apt-get install dnsutils
dig testservice -t SRV # this will take a long time
dig testservice.anydomain -t SRV # this is instant

lodrantl on 30 Nov 2017

@lodrantl Maybe you could prevent Kong's resolver from trying SRV queries in the configuration:

dns_order = LAST,A,CNAME

thibaultcha on 1 Dec 2017

👍1

That works. I was also thinking, that lua-resty-dns-client could check the format of the qname and only try to resolve it as SRV if it matches _service._proto.name..

lodrantl on 1 Dec 2017

Good to know! It might be worth opening a ticket for that in lua-resty-dns-client.

hishamhm on 8 Dec 2017

^ FWIW this is also why the resolution order is configurable in Kong. We want to run on a wide variety of platforms, which may have relatively subtle different semantics and designs. @thibaultcha's suggestion is on point 👍