Charts: [bitnami/postgresql-ha] HA does not survive worker node restart

[ashokspeelyaal]

Which chart:
postgresql-ha-1.4.4

Describe the bug
I have an ubuntu worker node where both the pods( 0 and 1) were running.
The node got restarted and see both the pods status remain CrashLoopBackOff.
I do not see any issue in pod logs.

From the logs pod-0:
repmgr 04:56:39.99 INFO ==> Preparing repmgr configuration...
repmgr 04:56:40.02 INFO ==> Initializing Repmgr...
postgresql 04:56:40.03 INFO ==> Initializing PostgreSQL database...
postgresql 04:56:40.12 INFO ==> Custom configuration /opt/bitnami/postgresql/conf/postgresql.conf detected
postgresql 04:56:40.13 INFO ==> Custom configuration /opt/bitnami/postgresql/conf/pg_hba.conf detected
postgresql 04:56:40.16 INFO ==> Deploying PostgreSQL with persisted data...
postgresql 04:56:40.19 INFO ==> Stopping PostgreSQL...
postgresql-repmgr 04:56:40.20 INFO ==> * PostgreSQL with Replication Manager setup finished! *

postgresql 04:56:40.35 INFO ==> Starting PostgreSQL in background...
From the logs pod-1:
postgresql-repmgr 04:54:00.19 INFO ==> * Starting PostgreSQL with Replication Manager setup *
repmgr 04:54:00.25 INFO ==> Validating settings in REPMGR_* env vars...
postgresql 04:54:00.25 INFO ==> Validating settings in POSTGRESQL_* env vars..
repmgr 04:54:00.26 INFO ==> Querying all partner nodes for common upstream node...
repmgr 04:54:00.29 INFO ==> Preparing PostgreSQL configuration...
postgresql 04:54:00.30 INFO ==> postgresql.conf file not detected. Generating it...
repmgr 04:54:00.36 INFO ==> Preparing repmgr configuration...
repmgr 04:54:00.38 INFO ==> Initializing Repmgr...
repmgr 04:54:00.38 INFO ==> Waiting for primary node...
postgresql 04:55:10.65 INFO ==> Stopping PostgreSQL...

To Reproduce
Steps to reproduce the behavior:

1. create with default configurations
2. restart one of the worker nodes(with primary running)
3. all other pods are coming back to running stage
4. Except, postgresql pods go to status CrashLoopBackOff

Expected behavior
The HA should survive the workder node restarts(for multiple reasons: scale up server, patch, unplanned restart by a cloud instance issue etc., )

Version of Helm and Kubernetes:
Helm: 3.1.0
Kubernetes: 1.17

• Output of helm version:

version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.8"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Additional context
I am not sure what is the ideal replica count or any other configuration to tackle this issue.
basically, the HA never comes back if primary fails. Hence in my current setup the HA purpose is not met.
Am I missing something to configure?

[dani8art]

Hi @ashokspeelyaal thank you for reporting the issue, could it be related to #2033?

[ashokspeelyaal]

Hello @dani8art , yes I assume so, the primary pod (0) was running on that node. I can try to reproduce in few days and report back if it is a duplicate issue of #2033

[dani8art]

Ok, we will appreciate it if you can confirm it. Thank you.

[ashokspeelyaal]

@dani8art , Yes I can confirm. The primary pod (0) was running on the node which was restarted and this caused the issue.

[ahilsend]

I am having a similar issue after a failure in the HA pods, one node cannot come back up.

Now both agree that pod 0 is the master, but pod 1 is in a CrashLoopBackOff:

The logs only show this:

repmgr 10:45:01.32 INFO  ==> Auto-detected primary node: 'nextcloud-postgresql-ha-postgresql-0.nextcloud-postgresql-ha-postgresql-headless.nextcloud.svc.cluster.local:5432'
repmgr 10:45:01.33 INFO  ==> Preparing PostgreSQL configuration...
postgresql 10:45:01.40 INFO  ==> postgresql.conf file not detected. Generating it...
repmgr 10:45:01.69 INFO  ==> Preparing repmgr configuration...
repmgr 10:45:01.70 INFO  ==> Initializing Repmgr...
repmgr 10:45:01.71 INFO  ==> Waiting for primary node...
repmgr 10:45:01.74 INFO  ==> Cloning data from primary node...
postgresql 10:45:21.82 INFO  ==> Stopping PostgreSQL...

I have already disabled the probes, so the pod can finish cloning without being killed, but that doesn't help. I guess the postgresql process itself might be dying.

Any suggestions on how to debug this further?

[dani8art]

Hi @ahilsend, have you try postgresqlImage.debug=true ? as you can see on #2033, we are trying to find a better way of shooting down or graceful exit for the pods.

[ahilsend]

Hi @ahilsend, have you try postgresqlImage.debug=true ? as you can see on #2033, we are trying to find a better way of shooting down or graceful exit for the pods.

Ohh, I missed that configuration. Thx! With the debug log I could see the issue was actually with DNS, and not with postgres. Restart of CoreDNS resolved my issue. Now I'll have to check what was up with coredns, thx! :blush:

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[dani8art]

Hi,
Could you confirm that this issue was solved with the PR above? Thanks!

[ashokspeelyaal]

Hello @dani8art , I will install with this chart version, verify and confirm as soon as possible.

[dani8art]

Cool! let us know if you have any problems! thanks!

[ashokspeelyaal]

Hello @dani8art, I can confirm it works now. It tried creating a new release and tried restarting nodes. When the nodes are up again the all the pods are back to status running after a while. Thanks for this.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.