Harbor: Updating Helm deployment from v1.8.3 to v1.9.0 causes DB pod fail to start

[eroji]

I was running on v1.8.3 previously and after updating to v1.9.0, the DB pod fails to start with the following error.

FATAL:  data directory "/var/lib/postgresql/data" has group or world access
DETAIL:  Permissions should be u=rwx (0700).

I verified that on a deployment I still have that is running on v1.8.3 that the /var/lib/postgresql/data directory is in fact 0700. Not entirely sure why v1.9.0 thinks it differs. As a workaround, I edited the deployment for the busybox sidecar with the following. This allowed the pod to launch successfully.

- args:
  - -c
  - chown -R 999:999 /var/lib/postgresql/data; chmod 0700 /var/lib/postgresql/data

[reasonerjt]

@eroji in future when you see issue in chart deployment please open issue on repo:
https://github.com/goharbor/harbor-helm

[ywk253100]

@eroji What is the volume provider you are using? I cannot reproduce the issue in my own environment. Could you dig into what happened in this case? Maybe you can add some logs in the init container to help figure out the root cause.

[eroji]

This is with Rook Ceph RBD storage. Filesystem is XFS. I was using the previous chart version 1.12 (I believe) which I overrode the image tags from 'dev' to the v1.8.3 versions. It deployed without any issue and I can console into the DB pod and see that the /var/lib/postgresql/data folder is 0700. However, when I did a 'helm upgrade' the DB pod failed with the above message. I had 2 instances of v1.8.3 running in two different clusters replicating with each other. The second deployment also encountered the same exact problem.

[jbartlet]

Same issue, but happens while running 1.9.1. After a short time of running healthy, ~ 1 hour, database fails with FATAL: data directory "/var/lib/postgresql/data" has group or world access DETAIL: Permissions should be u=rwx (0700). Using Rook-Ceph RBD / XFS as well. Something must be triggering db pod to restart and that's where the permissions issue pops up.

[Altes]

Same problem here with the 1.9.4. I'm using rook-ceph RBD storage with XFS filesystem. I tried to edit the deployment from the Rancher ui but it says that its forbidden to edit a statefulset. Any idea on what i can do?

[jbartlet]

Follow the solution given by @eroji but in Rancher, you have to edit the YAML for the statefulset. Add the args under the initContainer line for the busybox image.

[Altes]

Done that and it works, thanks! but do i lose the changes if i update the helm template?

[eroji]

@Altes I'm assuming you are using Helm 2. If you manually make changes outside of Helm install/upgrade using variable overrides, then it should retain this change. Actually this is an unexpected behavior that I had raised an issue with Helm about awhile back.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.