Kong: Cannot access Kong admin port 8001 after kong started

[freemanxx]

Summary

When kong started or reloaded, the admin port 8001 has no response in some minutes. This problem doesn't happen on CentOS 6.8, only happens on CentOS 7.3.1611

Additional Details & Logs

• Kong version (0.10.0)
• Kong debug-level startup logs ($ kong start --vv)
• Kong error logs (<KONG_PREFIX>/logs/error.log)
  error.txt
• Kong configuration (registered APIs/Plugins & configuration file)
  kong.txt
• Operating System
  CentOS Linux release 7.3.1611 (Core)

[Tieske]

have you tried explicitly setting admin_listen in the config file or the KONG_ADMIN_LISTEN environment variable?

[freemanxx]

YES, I explicitly set admin_listen as 0.0.0.0:8001 in /etc/kong/kong.conf.

[Tieske]

I meant set it to a the ip address of the adapter your listening on, eg 192.168.23.45:8001. This because using 0.0.0.0:8001 requires some system logic to find the correct ips, if that fails, you cannot connect.

[freemanxx]

I have tried to set it to 127.0.0.1:8001 or the lan ip 10.0.0.30:8001, it doesn't work. It seems not the ip problem. Because when I tried to access some invalid apis, eg 10.0.0.30:8001/wrongApi, it returns { "message": "Not found"}. That shows we can connect to port 8001. It seems kong cannot connect to postgresql server from the error log. But if this is true, how can it be started! After kong started about 3 minutes or a longer time, it works correctly. It's really a strange problem! Here is a part of my error log:

2017/03/17 16:50:01 [error] 10093#0: *5 [lua] hooks.lua:164: member_join(): [postgres error] timeout, client: 10.0.0.30, server: kong_admin, request: "POST /cluster/events/ HTTP/1.1", host: "10.0.0.30:8001"
2017/03/17 16:50:01 [notice] 10093#0: signal 17 (SIGCHLD) received
2017/03/17 16:50:01 [info] 10093#0: waitpid() failed (10: No child processes)
2017/03/17 16:50:02 [error] 10093#0: *13 [lua] responses.lua:101: [postgres error] timeout, client: 10.0.0.15, server: kong_admin, request: "GET / HTTP/1.1", host: "10.0.0.30:8001"
2017/03/17 16:50:02 [info] 10093#0: *13 client 10.0.0.15 closed keepalive connection
2017/03/17 16:50:03 [error] 10093#0: *114 [lua] postgres.lua:158: [postgres] could not cleanup TTLs: no route to host, context: ngx.timer
2017/03/17 16:50:30 [debug] 10094#0: *163 [lua] cluster.lua:20: log(): [cluster] sending keepalive event to datastore
2017/03/17 16:50:33 [error] 10094#0: *50 [lua] cluster.lua:20: log(): [cluster] [postgres error] timeout, context: ngx.timer
2017/03/17 16:51:00 [error] 10094#0: *113 [lua] postgres.lua:158: [postgres] could not cleanup TTLs: timeout, context: ngx.timer
2017/03/17 16:51:00 [debug] 10094#0: *199 [lua] cluster.lua:20: log(): [cluster] sending keepalive event to datastore
2017/03/17 16:51:02 [error] 10094#0: *9 [lua] hooks.lua:164: member_join(): [postgres error] timeout, context: ngx.timer
2017/03/17 16:51:02 [notice] 10094#0: signal 17 (SIGCHLD) received
2017/03/17 16:51:02 [info] 10094#0: waitpid() failed (10: No child processes)
2017/03/17 16:51:03 [error] 10094#0: *15 [lua] cluster.lua:20: log(): [cluster] [postgres error] timeout, context: ngx.timer
2017/03/17 16:51:06 [error] 10094#0: *21 [lua] cluster.lua:20: log(): [cluster] [postgres error] timeout, context: ngx.timer
2017/03/17 16:51:06 [error] 10093#0: *25 [lua] responses.lua:101: handler(): timeout, client: 10.0.0.30, server: kong_admin, request: "POST /cluster/events/ HTTP/1.1", host: "10.0.0.30:8001"
2017/03/17 16:51:06 [info] 10093#0: *25 client 10.0.0.30 closed keepalive connection

[Tieske]

have you tried a fully qualified domain name for postgres?

[freemanxx]

@Tieske I have tried to set postgres to localhost and it works, thank you very much. Buf if postgresql server works on the other machine, what should I do? I try to add a item of postgres' ip and its domain name in /etc/hosts, but that doesn't work.

[Tieske]

The dns resolver will resolve names in the /etc/hosts file and fully qualified names. See #2201

[freemanxx]

Problem solved! I reset the value of nameserver in /etc/resolv.conf and everything is fine. Now, no matter what I set to pg_host (a domain name or a valid ip), it works well.