Charts: [PostgreSQL] salves cannot connect to master due to password authentication failed

[alesanmed]

Which chart:
Chart: PostgreSQL
Version: 8.10.11

Describe the bug
I installed the chart using the production values. The slaveCount is set to 2, the user is the default repl_user and the password is empty because I'm setting the existingSecret key. The secret I'm using contains postgresql-replication-password. When I install the chart, the master is running but both slaves are showing the error in the logs:

pg_basebackup: could not connect to server: FATAL:  password authentication failed for user "repl_user"

And the master, of course:

2020-07-01 22:26:38.420 GMT [10561] FATAL:  password authentication failed for user "repl_user"
2020-07-01 22:26:38.420 GMT [10561] DETAIL:  Password does not match for user "repl_user".
    Connection matched pg_hba.conf line 6: "host      replication     all             0.0.0.0/0               md5"

I connected to one of the slaves and the repl_user password is correctly injected in an environment variable. I also connected to the master DB and the role is correctly created.

To Reproduce
Steps to reproduce the behavior:

1. Copy the values-production.yaml
2. Change the postgresqlUsername to a different one (don't know if this can affect)
3. Set the postgresql-postgres-password and postgresql-replication-password in a secret
4. Install the chart

Expected behavior
The slaves starts and connects successfully to the master using the repl_user credentials

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.6", GitCommit:"d32e40e20d167e103faf894261614c5b45c44198", GitTreeState:"clean", BuildDate:"2020-05-20T13:08:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[miguelaeh]

Hi @alesanmed ,
Did you deploy the release previously? Try to delete the previous PVC before installing it again. The database password is stored in the PVCs that are not deleted with the helm delete command, so the database could be configured with a previous password.
IF you don't want to delete the PVCs you can just install the Chart with a different name.

[plmercereau]

Thanks @miguelaeh, I had the same problem and it solved it.
If I understood well what happened, the password is not updated when changed through a helm command.
Is there another way to prevent this?

[miguelaeh]

I am afraid no. We are currently working to find the proper solution since this is a common issue. The problem here is that our containers are configured only the first time they are deployed and the configuration is stored into the PV. To change the passwords you will need to change it manually by executing the proper psql command inside the pod.

[plmercereau]

Thanks for this - the update of the config through helm will be definitely of great help!
In the meantime I now know this pitfall and its workaround.
Good luck in finding the solution!

[alesanmed]

Thanks for the help. I'll try the solution and check if it worked. If so, I'll close the issue.

[alesanmed]

It worked! Thanks! Closing the issue.