Cp-docker-images: [cp-zookeeper] Include health check script for Kubernetes etc.

Created on 11 Oct 2017 · 2Comments · Source: confluentinc/cp-docker-images

When deploying with Kubernetes, a means of checking the health of the Zookeeper container is advisable. This can be achieved (to an extent) with a TCP socket probe, but this approach will result in a lot of log warning spam in Zookeeper:

[2017-10-11 13:53:10,276] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:745)

The official Kubernetes project has solved this with a simple shell script that invokes Zookeeper's RUOK command:
https://github.com/kubernetes/contrib/blob/master/statefulsets/zookeeper/zkOk.sh

A simple implementation for cp-zookeeper would be:

#!/bin/bash
OK=$(echo ruok | nc 127.0.0.1 $ZOOKEEPER_CLIENT_PORT)
if [ "$OK" == "imok" ]; then
    exit 0
else
    exit 1
fi

Could we please have this script, or something like it, included in https://github.com/confluentinc/cp-docker-images/tree/3.3.x/debian/zookeeper/include/etc/confluent/docker in the next release?

Source

jtv8

👍4

Most helpful comment

@zerogjoe you can do the following in your deployment manifest (container spec section) without external scripts (assuming you can template zookeeper template port)

         livenessProbe:
            exec:
              command:    ['/bin/bash', '-c', 'echo "ruok" | nc -w 2 -q 2 localhost 2181 | grep imok']
            initialDelaySeconds: 15
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command: ['/bin/bash', '-c', 'echo "ruok" | nc -w 2 -q 2 localhost 2181 | grep imok']

gAmUssA on 8 Jan 2018

👍8 ❤6

All 2 comments

@zerogjoe you can do the following in your deployment manifest (container spec section) without external scripts (assuming you can template zookeeper template port)

         livenessProbe:
            exec:
              command:    ['/bin/bash', '-c', 'echo "ruok" | nc -w 2 -q 2 localhost 2181 | grep imok']
            initialDelaySeconds: 15
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command: ['/bin/bash', '-c', 'echo "ruok" | nc -w 2 -q 2 localhost 2181 | grep imok']

gAmUssA on 8 Jan 2018

👍8 ❤6

Doing what @gAmUssA suggests in a docker-compose setup, I'm seeing warnings and low performance in the clients.