Patroni: Liveness probe on k8s sts?

[bappr]

Hey guys,

Any reason why you are not using Liveness/Readiness probes in your k8s statefulset (in this repo and in your helm chart)?
Can I use one? Wondering if it's gonna impact something else.

Thanks a lot,

Ben

[CyberDem0n]

Hi,

there is no big value in readiness probe. Patroni will not set role label until postgres is ready to accept connections.

With liveness probe it is a very interesting question. From one side we can use it as a sort of watchdog in order to protect from the case when Patroni crashed or become unresponsive, but from another side killing the stateful application is not always something that you want.

From the Patroni side it will require some little changes in order to implement the liveness probe. One should implement do_GET_healthz() method which always responds with http status code 200 in the RestApiHandler class.

[bappr]

Ok thanks for the info. ;)
I'll leave it opened until it can be implemented.

[anikin-aa]

May be it is worth to use something like this in liveness probe:

state=$(curl -XGET --silent http://$(hostname -i):8008/ | jq -r '.state | ascii_downcase'); 
if [ "$state" != "running" ];
then
    exit 1;
fi

[CyberDem0n]

Nope, liveness probe should never take into account state of the postgres.

[mreise]

I am looking for liveness and readiness probe, too. Could be an approach to determine a simple liveness probe on TCP 8008 (patroni rest port)? At readiness probe I would check if state is running and label role is set as already mentioned above. It is tested with version 1.5.6...

[CyberDem0n]

I would recommend you to read about liveness probes: https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html

[bappr]

@CyberDem0n i see no reason to close this issue. This post you shared does not mention anything dangerous, just how to use probes.

When I opened the issue, postgres sometimes was down and the pod was still running. Is it resolved?

[CyberDem0n]

postgres sometimes was down and the pod was still running

This is absolutely normal workflow:

• Patroni is could do pg_basebackup or something similar in order to (re)initialize the replica.
• Patroni could do pg_rewind, therefore postgres will not be running
• postgres might fail to start due to different issues

Killing the pod in all this cases will not improve situation, it will just make it harder to investigate what is going on.

Liveness probe is a big footgun, I don't want to open a pandora box by implementing it.