Kong: serf defunct in docker

[gavinzhou]

I run kong in docker
get this messages

root       115   111  0 Mar10 ?        00:01:24 nginx: worker process
root       153     1  0 Mar10 ?        00:00:00 [serf] 
root       178     1  0 Mar10 ?        00:00:00 [serf] 
root       193     1  0 Mar10 ?        00:00:00 [serf] 
root       208     1  0 Mar10 ?        00:00:00 [serf] 
root       233     1  0 Mar10 ?        00:00:00 [serf] 
root       248     1  0 Mar10 ?        00:00:00 [serf] 
root       263     1  0 Mar10 ?        00:00:00 [serf] 

[subnetmarco]

There should only be one instances of serf running, can you check the log at {working_directory}/serf.log ?

[gavinzhou]

I run docker kong in the one instance

[root@658ca45ad17b /]# more /usr/local/kong/serf.log
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
         Node name: '658ca45ad17b_0.0.0.0:7946'
         Bind addr: '0.0.0.0:7946'
          RPC addr: '127.0.0.1:7373'
         Encrypted: false
          Snapshot: false
           Profile: wan

==> Log data will now stream in as it occurs:

thank you

[subnetmarco]

What happens if you run serf members?

[gavinzhou]

I not set anything in serf

[bketelsen]

I am seeing the same error
I'll attach to the container next time they build up (I can get several thousand zombie serf processes in a day) and run the serf members command. Any other debugging you'd like me to do while I'm in there?

[subnetmarco]

@bketelsen what version of Kong are you using? I will try to replicate. After how long does this usually happen?

[hoffin]

Having the same issue.
Followed these instructions
https://getkong.org/install/docker/
Cassandra running on a different box with updated cassandra address in config.
Runs fine other than the constant stream of serf defunct processes.

Kong version 0.7.0.
Docker 1.10.3, build 20f81dd

root      2725  0.0  0.0      0     0 ?        Z    00:19   0:00 [serf] <defunct>
root      2765  0.0  0.0      0     0 ?        Z    00:20   0:00 [serf] <defunct>
root      2783  0.0  0.0      0     0 ?        Z    00:20   0:00 [serf] <defunct>
root      2798  0.0  0.0      0     0 ?        Z    00:21   0:00 [serf] <defunct>
root      2811  0.0  0.0      0     0 ?        Z    00:21   0:00 [serf] <defunct>
root      2833  0.0  0.0      0     0 ?        Z    00:22   0:00 [serf] <defunct>
root      2866  0.0  0.0      0     0 ?        Z    00:22   0:00 [serf] <defunct>

[gavinzhou]

I used alpine build kong 0.7.0

It working
https://github.com/gavinzhou/alpine-s6-kong

hub-docker
https://hub.docker.com/r/orangesys/alpine-s6-kong/

very small size

docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
orangesys/alpine-s6-kong       0.7.0-0             b451b4c39a60        About an hour ago   63.59 MB

[subnetmarco]

I was thinking to use Alpine in our official Docker image to reduce the size anyways - I will take inspiration from your repos @gavinzhou.

[gavinzhou]

thank you
alpine is good

[codebien]

Kong: 0.7.0
Docker: 1.9.1
AWS ECS cluster

We are experiencing the same issue. A lot of [serf] <defunct>. We also view a high increase of the CPUs usage but we are not sure it depends on this problem.

[marcelmfs]

+1 here, hundreds of serf defunct, potential to reach thousands and thrash the server.

kong 0.7.0, docker 1.10.

[djodjo3]

Hi,

Don't know if that helps, but same thing here, with docker 1.10.2 and kong 0.7.0.

Thanks in advance,

[marcelmfs]

This is probably due to dockerized kong not properly handling defunct processes with something like https://github.com/krallin/tini

[jfirehammer-credibly]

Thanks, I forked it to https://github.com/jfirehammer-credibly/docker-kong and added tini, I'll let you know how it goes.

[subnetmarco]

In Kong 0.8.0 I switched to Serf 64bit (as opposed to 32bit which was also causing some other issues) - Can you try with the new version and see if the problem persists?

[gavinzhou]

it worked!!
thanks you

[marcelmfs]

I can also confirm that serf is not becoming defunct anymore inside docker containers.

[subnetmarco]

Great - I am closing this issue. Please re-open if it happens again.

[laura-herrera]

Hi there,
I am having the same issue and I am using Kong 0.8.3

This is the content of serf.log
==> Starting Serf agent...
==> Starting Serf agent RPC...
==> Serf agent running!
Node name: 'kong-yt93m_0.0.0.0:7946_e4b0ec565b374609a9bd228a2eef9ee2'
Bind addr: '0.0.0.0:7946'
RPC addr: '127.0.0.1:7373'
Encrypted: false
Snapshot: false
Profile: wan
==> Log data will now stream in as it occurs:
2016/07/21 17:02:39 [ERR] memberlist: Received invalid msgType (71) from=[::1]:48355

And running serf members:
[root@kong-yt93m kong]# serf members
kong-yt93m_0.0.0.0:7946_e4b0ec565b374609a9bd228a2eef9ee2 10.244.5.13:7946 alive

any ideas?

[gavinzhou]

I used "mashape/kong" docker image,
get same err in in the k8s. but it working in the docker
and can't create jwt.
@laura-herrera @thefosk

[root@kong-2981305527-m3ny0 /]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 01:00 ?        00:00:00 /usr/local/bin/luajit -e package
root        73     0  0 01:01 ?        00:00:00 /bin/bash
root        95     1  0 01:01 ?        00:00:00 [dnsmasq] <defunct>
root        96     1  0 01:01 ?        00:00:00 dnsmasq -p 8053 --pid-file=/usr/
root       114     1  0 01:01 ?        00:00:00 serf agent -profile=wan -rpc-add
root       171     1  0 01:01 ?        00:00:00 [nc] <defunct>
root       172     1  0 01:01 ?        00:00:00 sh -c /bin/bash /tmp/lua_2yDewd
root       173   172  0 01:01 ?        00:00:00 /bin/bash /tmp/lua_2yDewd
root       174   173  0 01:01 ?        00:00:00 nginx: master process /usr/local
root       176   174  0 01:01 ?        00:00:00 nginx: worker process
root       214     1  0 01:01 ?        00:00:00 [serf] <defunct>
root       247     1  0 01:02 ?        00:00:00 [serf] <defunct>
root       265     1  0 01:02 ?        00:00:00 [serf] <defunct>
root       356     1  0 01:03 ?        00:00:00 [serf] <defunct>
root       375     1  0 01:03 ?        00:00:00 [serf] <defunct>
root       390     1  0 01:04 ?        00:00:00 [serf] <defunct>
root       403     1  0 01:04 ?        00:00:00 [serf] <defunct>
root       418     1  0 01:04 ?        00:00:00 [serf] <defunct>
root       431     1  0 01:05 ?        00:00:00 [serf] <defunct>
root       444     1  0 01:05 ?        00:00:00 [serf] <defunct>
root       457     1  0 01:06 ?        00:00:00 [serf] <defunct>
root       470     1  0 01:06 ?        00:00:00 [serf] <defunct>
root       490     1  0 01:06 ?        00:00:00 [nc] <defunct>
root       495     1  0 01:06 ?        00:00:00 [serf] <defunct>
root       508     1  0 01:06 ?        00:00:00 [serf] <defunct>
root       522     1  0 01:07 ?        00:00:00 [serf] <defunct>
root       535     1  0 01:07 ?        00:00:00 [serf] <defunct>
root       546    73  0 01:07 ?        00:00:00 ps -ef

curl http://127.0.0.1:8001/consumers/influx0000/jwt
<html>
<head><title>415 Unsupported Media Type</title></head>
<body bgcolor="white">
<center><h1>415 Unsupported Media Type</h1></center>
<hr><center>openresty/1.9.7.5</center>
</body>
</html>

[gavinzhou]

I found this

https://github.com/docker/docker/issues/24284

[gavinzhou]

@laura-herrera
@thefosk

I fix zombie issue by using dumb-init
I create kong docker with alpine

https://github.com/gavinzhou/alpine-kong

[laura-herrera]

Thanks @gavinzhou, yes adding tini fixed the problem for me too.

[486]

Can we reopen this? I see the same issue with kong:0.8.3 as described above:

   31 ?        Z      0:00 [nc] <defunct>
   66 ?        Zs     0:00 [dnsmasq] <defunct>
   67 ?        S      0:00 dnsmasq -p 8053 --pid-file=/usr/local/kong/dnsmasq.pid -N -o --listen-address=127.0.0.1
   87 ?        Sl     0:00 serf agent -profile=wan -rpc-addr=127.0.0.1:7373 -event-handler=member-join,member-leave,member-failed,member-update,memb
  158 ?        Z      0:00 [nc] <defunct>

Instead of having everyone build his own kong image, I'd happily do a PR but we need a consensus on what should be in there.

[subnetmarco]

I will give it a try and build the Kong image on top of https://github.com/phusion/baseimage-docker which should fix the PID 1 problem (or https://github.com/Yelp/dumb-init as somebody else has suggested).

[486]

@thefosk Cool - while we are at it, what do think about "fixing" some other nagging points about the image?

• docker-entrypoint.sh writes "database: $DATABASE" to kong.yml on every start of the container
• kong dies when the db is not ready on startup

Both issues can be solved with dockerize. So the init process (from baseimage-docker or dumb-init) would start dockerize which can wait for the db and create a clean kong.yml from a template (dockerize does not reap zombie processes so we still need a "good" init). If you'd like to go that route, I could do a PR.

[StephanZaat]

An official fix would be great. Our servers are crashing due to a lack of free PIDs from too many defunct serf processes.

[orphaner]

Hello,
Here is a fix proposal using tini : https://github.com/Mashape/docker-kong/pull/48

[subnetmarco]

In the 0.9 I have tried an implementation with dumb-init that _should_ fix this problem - do you confirm?

If it works I will retro-port the patch into 0.8.3 too.

[gavinzhou]

@thefosk
I run https://github.com/Mashape/docker-kong in my STAGE
it working .
thank your patch

[subnetmarco]

Perfect, I have also applied the patch to 0.8.3. If you pull again mashape/kong:0.8.3 it should work with the previous version too.

[StephanZaat]

Yes, works great here on 0.8.3. also! Thanks!