Kong: IPv6 support?

[user419]

Summary

Started with some issues on 0.9.x resolving upstream host names.
After updating to 0.10.1 I hit the 'Error: Invalid configuration, no dns servers found' error.
Reading the doc it looks like only IPv4 is supported.
Also tried to explicitly configuring an (IPv6) resolver but got an error suggesting IPv4.

Is IPv6 supported at all by Kong?

Additional Details & Logs

• Kong version ($ kong version)
  0.10.1
• Kong debug-level startup logs ($ kong start --vv)
  2017/04/02 11:44:38 [verbose] Kong: 0.10.1
  2017/04/02 11:44:38 [debug] ngx_lua: 10007
  2017/04/02 11:44:38 [debug] nginx: 1011002
  2017/04/02 11:44:38 [debug] Lua: LuaJIT 2.1.0-beta2
  2017/04/02 11:44:38 [verbose] reading config file at /etc/kong/kong.conf
  Error:
  /usr/local/share/lua/5.1/kong/cmd/start.lua:10: dns_resolver must be a comma separated list in the form of IPv4 or IPv4:port, got '2001:4860:4860::8888'
  stack traceback:
  [C]: in function 'assert'
  /usr/local/share/lua/5.1/kong/cmd/start.lua:10: in function 'cmd_exec'
  /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function
  [C]: in function 'xpcall'
  /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function
  /usr/local/bin/kong:5: in function 'file_gen'
  init_worker_by_lua:38: in function
  [C]: in function 'pcall'
  init_worker_by_lua:45: in function
• Kong error logs (<KONG_PREFIX>/logs/error.log)
• Kong configuration (registered APIs/Plugins & configuration file)
• Operating System
  CentOS Linux release 7.3.1611 (Core)

[bungle]

Many places it isn't but I just released lua-resty-mediador that I hope we can use in many places to replace lua-resty-iputils, but for whole IPv6 support, we need to do a bit more research.

[pamiel]

+1 for the full support of IPv6...

[rkujawa]

Guys, not having proper IPv6 support can be a problem blocking deployment of Kong. In two weeks or so I'll be evaluating Kong to see whether it can be utilised in my organisation. Due to cost issues with public IPv4 addresses and desire to simplify network management, some of our server environments are IPv6 only now. I guess I am not the only one going this way 😉. Just saying...

[bungle]

@rkujawa,

I made a quick review to codebase, and it seems these are the few places where IPv6 is not supported right now:

• dns_resolver (@Tieske knows about the complexity of getting this to work?)
• cluster_listen (@thibaultcha, do you remember the reason why we only accept IPv4 here?)
• ip-restriction plugin (this we can easily fix just be replacing lua-resty-iputils with lua-resty-mediador)

I think that everything else will just happily work with both IPv4 and IPv6.

[Tieske]

@bungle The resolver should be IPv6 already, what's missing is configuring it. The config properties only allow IPv4 entries.

I think that everything else will just happily work with both IPv4 and IPv6.

My concern here is effort required for testing with IPv6...

[rkujawa]

@Tieske

I suppose you are referring to the fact that IPv6 is completely broken on Travis, while you are using Travis to run tests? It's perma-disabled even on loopback interfaces. That's an issue I ran into myself. Unfortunately, Travis devs are not even willing to talk about it (all the issues touching IPv6 are closed and locked immediately, like that solves the problem, duh).

[rant]
On a side note, IPv6 is now supported in Amazon EC2 virtual private cloud instances. Besides, it is offered by many minor cloud and VPS providers (Brightbox, Mythic Beasts, OVH etc.). So it's not like it's some super bleeding edge technology. At @osec-pl we even have native IPv6+IPv4 dual stack network in the whole office.
[/rant]

[Tieske]

@rkujawa indeed that is one part, the other being that all core functionality of Kong must be tested against both IP families, so lots of extra test cases.

[thibaultcha]

@rkujawa indeed that is one part, the other being that all core functionality of Kong must be tested against both IP families, so lots of extra test cases.

While such a test environment would eventually be nice, I'm not sure we absolutely need it given that the moving parts implicating DNS resolution are tested separately already (and might sometimes be used atomically from each other as well).

Assuming:

1. OpenResty is compiled with ipv6 support and such support is tested in cosockets
2. resty-cli allows resolving with ipv6
3. The Nginx DNS resolver does not contain the ipv6=off option
4. Serf supports ipv6 values
5. The Kong DNS resolver/balancer accepts both ipv4 and ipv6 and is tested as such
6. The Kong properties accept both ipv4/ipv6 and are tested as such

1. 
   
   1. and 3. are already done. I think that 4 and 5. are as well, with a lower level of confidence. What would be missing here would be 6.
   
   

What should probably be done as well to support cases where Nginx was compiled with ipv6 but those ar enot supported on the network in use, is adding a property in kong.conf to disable ipv6 altogether (since recent version of Nginx do not support the --without-ipv6 flag anymore), the same way the Nginx resolver does. (At least, for as long as we need our custom Lua-land resolver and Serf).

Anything missing in there implicating DNS resolutions?

[Tieske]

correct 5. does already have IPv6

[filterfish]

In case you are after some more data points our internal network is ipv6 only. I'm just deploying a testing release of a new product and it didn't occur to me that Kong wouldn't support ipv6. I know I should have checked but it is 2017 ;)

[denis-fortin]

Hello,

Is there any news on the ipv6 support ?

Thanks :pray: !

[tofiksa]

we have the same problems with the balancer. Many of our upstream api servers are ipv6 only, so the dns_resolver is not capable of resolving AAAA for ipv6 only addresses.

[wjdnsdhydl]

I just built kong (v1.4) on k8s(v1.11), but I also encountered problems with DNS resolution. The specific error message is as follows:

2018/11/12 09:23:05 [error] 55#0: *80510 [lua] balancer.lua:806: execute(): [dns] dns client error: 101 empty record received. Tried: (short)org:(na) - cache-miss
org.etu-staging.svc.cluster.local.:33 - cache-miss/scheduled/querying/1:org.etu-staging.svc.cluster.local removed/33:org.etu-staging.svc.cluster.local removed/dns client error: 101 empty record received
org.svc.cluster.local.:33 - cache-miss/scheduled/querying/dns server error: 3 name error
org.cluster.local.:33 - cache-miss/scheduled/querying/dns server error: 3 name error
org.localdomain:33 - cache-miss/scheduled/querying/dns server error: 3 name error
org.pek3.qingcloud.com:33 - cache-miss/scheduled/querying/dns server error: 3 name error
org:33 - cache-miss/scheduled/querying/dns client error: 101 empty record received
org.etu-staging.svc.cluster.local.:1 - cache-miss/scheduled/querying/1:org.etu-staging.svc.cluster.local removed/dns client error: 101 empty record received
org.svc.cluster.local.:1 - cache-miss/scheduled/querying/dns server error: 3 name error
org.cluster.local.:1 - cache-miss/scheduled/querying/dns server error: 3 name error
org.localdomain:1 - cache-miss/scheduled/querying/dns server error: 3 name error
org.pek3.qingcloud.com:1 - cache-miss/scheduled/querying/dns server error: 3 name error
org:1 - cache-miss/scheduled/querying/dns client error: 101 empty record received
org.etu-staging.svc.cluster.local.:5 - cache-miss/scheduled/querying/dns client error: 101 empty record received
org.svc.cluster.local.:5 - cache-miss/scheduled/querying/dns server error: 3 name error
org.cluster.local.:5 - cache-miss/scheduled/querying/dns server error: 3 name error
org.localdomain:5 - cache-miss/scheduled/querying/dns server error: 3 name error
org.pek3.qingcloud.com:5 - cache-miss/scheduled/querying/dns server error: 3 name error
org:5 - cache-miss/scheduled/querying/dns client error: 101 empty record received
, client: 10.10.0.0, server: kong, request: "GET /org/api/v1/user HTTP/1.0", host: "api-test.etutech.com"
10.10.0.0 - - [12/Nov/2018:09:23:05 +0000] "GET /org/api/v1/user HTTP/1.0" 503 37 "-" "PostmanRuntime/7.3.0"
10.10.3.118 - - [12/Nov/2018:09:27:21 +0000] "GET /routes? HTTP/1.1" 200 370 "-" "-"
10.10.3.118 - - [12/Nov/2018:09:27:21 +0000] "GET /services/fb1514f2-19ce-4ebd-842b-8e013dac8580 HTTP/1.1" 200 227 "-" "-"

Is this error also a support issue for IPv6? If this is the case, I would like to ask what is the progress of IPv6 support now.

Thanks !

[Tieske]

@wq7 from the logs it shows that it tries to query for 33, 1, 5 type records, respectively SRV, A, CNAME. Apparently it cannot resolve the name, since it gets a "3 name error" response from the dns server.

Please note: kong contacted the dns server, submitted the query, got a response. So every thing (from Kong perspective) works. It's just that the name server apparently doesn't have a record for those names.

If your enviornment is IPv6 only, then it makes sense, since Kong doesn't support it.

[wjdnsdhydl]

@Tieske
Thank you for your reply.

Today I entered the docker container and used the dig command to get the following result:

root@kong-rc-78db95869-2g64z:/# dig org.etu-staging.svc.cluster.local. SRV

; <<>> DiG 9.9.5-9+deb8u16-Debian <<>> org.etu-staging.svc.cluster.local. SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62683
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;org.etu-staging.svc.cluster.local. IN  SRV

;; ANSWER SECTION:
org.etu-staging.svc.cluster.local. 5 IN SRV 0 100 80 org.etu-staging.svc.cluster.local.

;; ADDITIONAL SECTION:
org.etu-staging.svc.cluster.local. 5 IN A   192.168.24.168

;; Query time: 6 msec
;; SERVER: 192.168.24.2#53(192.168.24.2)
;; WHEN: Tue Nov 13 17:55:14 CST 2018
;; MSG SIZE  rcvd: 197

root@kong-rc-78db95869-2g64z:/# dig org.etu-staging.svc.cluster.local. A

; <<>> DiG 9.9.5-9+deb8u16-Debian <<>> org.etu-staging.svc.cluster.local. A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8336
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;org.etu-staging.svc.cluster.local. IN  A

;; ANSWER SECTION:
org.etu-staging.svc.cluster.local. 5 IN A   192.168.24.168

;; Query time: 1 msec
;; SERVER: 192.168.24.2#53(192.168.24.2)
;; WHEN: Tue Nov 13 17:55:28 CST 2018
;; MSG SIZE  rcvd: 111

root@kong-rc-78db95869-2g64z:/# dig org.etu-staging.svc.cluster.local. CNAME

; <<>> DiG 9.9.5-9+deb8u16-Debian <<>> org.etu-staging.svc.cluster.local. CNAME
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11533
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;org.etu-staging.svc.cluster.local. IN  CNAME

;; AUTHORITY SECTION:
cluster.local.      30  IN  SOA ns.dns.cluster.local. hostmaster.cluster.local. 1542098768 7200 1800 86400 30

;; Query time: 1 msec
;; SERVER: 192.168.24.2#53(192.168.24.2)
;; WHEN: Tue Nov 13 17:55:30 CST 2018
;; MSG SIZE  rcvd: 155

Find the SRV and A type addresses of the org.etu-staging.svc.cluster.local domain name that can be found.

I deployed kong in the k8s cluster. Is the problem I encountered related to the DNS of k8s?

[wjdnsdhydl]

@Tieske
I just tried to modify the kong's service and changed the host from org to org.etu-staging.svc.cluster.local. The problem is solved and the API can be accessed normally.

Previous service:

{  
   "next":null,
   "data":[  
      {  
         "host":"org",
         "created_at":1542011318,
         "connect_timeout":60000,
         "id":"fb1514f2-19ce-4ebd-842b-8e013dac8580",
         "protocol":"http",
         "name":"org",
         "read_timeout":60000,
         "port":80,
         "path":null,
         "updated_at":1542101743,
         "retries":2,
         "write_timeout":60000
      }
   ]
}

Current service:

{  
   "next":null,
   "data":[  
      {  
         "host":"org.etu-staging.svc.cluster.local",
         "created_at":1542011318,
         "connect_timeout":60000,
         "id":"fb1514f2-19ce-4ebd-842b-8e013dac8580",
         "protocol":"http",
         "name":"org",
         "read_timeout":60000,
         "port":80,
         "path":null,
         "updated_at":1542101743,
         "retries":2,
         "write_timeout":60000
      }
   ]
}

[Tieske]

@wq7 have you tried the hostname without the trailing dot?

[wjdnsdhydl]

@Tieske

root@kong-rc-78db95869-2g64z:/# dig org

; <<>> DiG 9.9.5-9+deb8u16-Debian <<>> org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28222
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;org.               IN  A

;; AUTHORITY SECTION:
org.            30  IN  SOA a0.org.afilias-nst.info. noc.afilias-nst.info. 2013230583 1800 900 604800 86400

;; Query time: 386 msec
;; SERVER: 192.168.24.2#53(192.168.24.2)
;; WHEN: Tue Nov 13 20:28:22 CST 2018
;; MSG SIZE  rcvd: 114

The above is the result of dig org.

Short name in k8s may not find DNS, it should be just an alias name.

Here are the results of ping:

root@kong-rc-78db95869-2g64z:/# ping org
PING org.etu-staging.svc.cluster.local (192.168.24.168): 56 data bytes

[rkujawa]

The DNS server will never answer a query for "org" name, this is by design. Use fully qualified host names when querying DNS. The fact that ping command works in your example is a pure coincidence, due to the fact that local resolver is configured to append a suffix (probably by configuration in /etc/resolv.conf)

This issue has nothing to do with IPv6.

[wjdnsdhydl]

@rkujawa

Yes, now I figured it out, my judgment was wrong before, I can use it after modifying the host of kong's service.

Thanks ！

[bungle]

As we now mostly support ipv6 with most notable exception being ip-restriction plugin, I will close this.