Charts: [stable/postgresql] Failed initializing postgresql

[Glorf]

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:
helm: {SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
kubernetes: {Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart: [stable/postgresql], version 2.2.0

What happened: I was using versions 0.x previously. When I updated to latest 2.2.0, I got constant crash-loop-backoff with information in pod's log:
EACCES: permission denied, open '/.nami/registry.json'
My configuration is default, except I set custom postgresql[User|Password|Database], and custom persistence (1Gi, ReadWriteOnce).
But problem exists even without setting these

What you expected to happen: Expected postgresql to deploy and work smoothly

How to reproduce it (as minimally and precisely as possible): Literally even 'helm install stable/postgresql' makes it crash

Anything else we need to know: I've found similar problem on your redis issues tracker, but didn't find it related to my use case.

[marcoceppi]

To add to this, the update made by the new maintainers have effectively rendered this chart broken.

Welcome to the Bitnami postgresql container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
nami INFO Initializing postgresql
postgre INFO ==> No injected postgresql.conf file found. Creating default postgresql.conf file...
postgre INFO ==> No injected pg_hba.conf file found. Creating default pg_hba.conf file...
postgre INFO ==> Deploying PostgreSQL from scratch...
Error executing 'postInstallation': EACCES: permission denied, mkdir '/bitnami/postgresql/data'

Of course, this switch to a bitnami image is the crux of the issue. which leads to this: https://github.com/bitnami/bitnami-docker-mariadb/issues/136 and the ultimate solution needing an initContainer.

Furthermore, because everything is hard-coded to bitnami's image, it's not as simple as just changing

image:
  repository: postgres

since the persistent data is /bitnami/postgresql - the official postgresql image supports moving the data dir, but bitnami does not.

Please, new bitnami backed maintainers, either fix this chart or make it compatible with non-bitnami images.

[carrodher]

Hi @Glorf thanks for the report!

What happened: I was using versions 0.x previously. When I updated to latest 2.2.0, I got constant crash-loop-backoff with information in pod's log:

EACCES: permission denied, open '/.nami/registry.json'

My configuration is default, except I set custom postgresql[User|Password|Database], and custom persistence (1Gi, ReadWriteOnce).

The image stores data and configurations at the /bitnami/postgresql path. Persistence is managed using PVC (Persistent Volume Claims). PVCs are used to keep the data across deployments, this is known to work in GCE, AWS, and minikube. To configure the PVC or to disable persistence you can use the following parameters:

persistence.enabled      | Enable persistence using PVC              | true
persistence.storageClass | PVC Storage Class for PostgreSQL volume   | nil
persistence.accessMode   | PVC Access Mode for PostgreSQL volume     | ReadWriteOnce
persistence.size         | PVC Storage Request for PostgreSQL volume | 8Gi
persistence.annotations  | Annotations for the PVC                   | {}

Taking a look at the error, it seems an issue related to the volume permissions. Can you share more information about the way you're using the persistence (volume permissions, owner, etc)?

But problem exists even without setting these

How to reproduce it (as minimally and precisely as possible): Literally even helm install stable/postgresql makes it crash

Running a simple helm install stable/postgresql doesn't work for you? I mean, because it is a major version with a ton of heavy changes (new image, new app version, move from deployment to statefulset, etc) it could be expected that during the update of existing deployments some conflicts/issues might appear, but for sure an installation from scratch should work without issues.

I am trying to reproduce the issue but I can't, I am using similar environment:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", BuildDate:"2018-09-28T15:18:13Z", GoVersion:"go1.11", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Platform:"linux/amd64"}

```bash
$ minikube version
minikube version: v0.28.0

```bash
$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Running helm install stable/postgresql on the above environment, this is the result:

$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
brown-hyena-postgresql-0   1/1     Running   0          10m

$ kubectl logs brown-hyena-postgresql-0 -f

Welcome to the Bitnami postgresql container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues

nami    INFO  Initializing postgresql
postgre INFO  ==> No injected postgresql.conf file found. Creating default postgresql.conf file...
postgre INFO  ==> No injected pg_hba.conf file found. Creating default pg_hba.conf file...
postgre INFO  ==> Deploying PostgreSQL from scratch...
postgre INFO  ==> Creating postgres user with unrestricted access...
postgre INFO  ==> Configuring PostgreSQL...
postgre INFO  ==> Configuring replication parameters...
postgre INFO  ==> Configuring permissions for config files...
postgre INFO
postgre INFO  ########################################################################
postgre INFO   Installation parameters for postgresql:
postgre INFO     Root User: postgres
postgre INFO     Password: **********
postgre INFO   (Passwords are not shown for security reasons)
postgre INFO  ########################################################################
postgre INFO
nami    INFO  postgresql successfully initialized
INFO  ==> Starting postgresql...
2018-10-25 23:31:25.282 GMT [53] LOG:  received fast shutdown request
2018-10-25 23:31:25.285 GMT [53] LOG:  aborting any active transactions
2018-10-25 23:31:25.286 GMT [53] LOG:  worker process: logical replication launcher (PID 61) exited with exit code 1
2018-10-25 23:31:25.286 GMT [56] LOG:  shutting down
2018-10-25 23:31:25.309 GMT [53] LOG:  database system is shut down
2018-10-25 23:31:27.075 GMT [97] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2018-10-25 23:31:27.075 GMT [97] LOG:  listening on IPv6 address "::", port 5432
2018-10-25 23:31:27.079 GMT [97] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-10-25 23:31:27.099 GMT [99] LOG:  database system was shut down at 2018-10-25 23:31:25 GMT
2018-10-25 23:31:27.103 GMT [97] LOG:  database system is ready to accept connections

[Glorf]

Hi, @carrodher thanks for response
My PV provider is NFS configured as follows:
/mnt/exports/nfsprovroot *(rw,sync,crossmnt,no_subtree_check,no_root_squash)

I'm not sure what other things related to the PV you might want to know (I'm not the one who set it up), but all the other charts are working without problems on this configuration, and this new postgresql is the only one to brake

As you assumed, that's not db update problem - fresh installation also fails, so I think it's somehow related to the way files are stored by the pod.

[Glorf]

It's worth saying, that even switch to second PV provider (rook block storage) don't fix the problem - and error stays the same

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[marcoceppi]

It's not stale if the maintainers ignore it.

[tompizmor]

I would say the issue is that PostgreSQL is running as a non-root user and the NFS provisioner does not support the securityContext kubernetes feature.

To be sure, I would try to deploy the postgres chart with the following options so it can be run as root:

helm repo update
helm install stable/postgresql --name postgresql --set securityContext.fsGroup=0,securityContext.runAsUser=0

[Glorf]

Hi @tompizmor thanks for reply
I confirm that setting given securityContext fixes this issue and postgresql is running properly, while when this option is not set, there is CrashLoopBackoff.
Given this information, can the chart be fixed or it's up to my storageclass provisioner? @carrodher ?

[tompizmor]

@Glorf I've been taking a look at the chart and I think the default configuration should also work for you since there is an initContainer changing the permissions of the volume.

Can you let me know which version of the PostgreSQL chart are you trying to install?

[Glorf]

I try to deploy literally latest chart - postgresql-3.1.3 - still permissionDenied and CrashLoopBackoff
Last chart that was working for us without changes in securityContext is postgresql-1.0.0
Postgresql-3.1.3 is deploying and working well when security context fix is applied

[giorgiosironi]

I don't think the current chart can work with the official postgres image because the environment variables are prefixed with POSTGRESQL_* rather than POSTGRES_.

[juan131]

Hi @giorgiosironi

What do you mean with "official" image? Do you mean the image maintained by Docker, Inc?

As you mentioned the chart is adapted to the PostgreSQL image maintained by Bitnami. Is there any feature missing in the current image? Please open an issue/PR at https://github.com/bitnami/bitnami-docker-postgresql if you consider so. Any contribution/suggestion to improve the image is very welcomed.

[giorgiosironi]

Yes, I see https://hub.docker.com/_/postgres has a Docker Official Image label.

Essentially the environment variables names are different hence the chart cannot be used with the Docker Official Image; but this is just what's visible, things like folders and users may be even more different. Hence I personally gave up trying to use the postgres image and am sticking with the Bitnami one.

[juan131]

The "official" label is sth Docker adds to its own images. However, they don't offer any mechanism so other people/companies can certified their images by auditing the images (at least AFAIK).

That's the reason why I have my concerns about calling them "officials". The company that acts as "authority" that provides the "official" label and the company that develops it is the same one.

I know is just a label, but I'd rather called them "Trusted by Docker" the same way you can call the Bitnami one, "Trusted by Bitnami".

[marcoceppi]

Official or not, people - myself included - who come to get the stable postgres chart expect to be using the PostgreSQL docker image in library. NGINX maintains their library image, Ubuntu theirs, and several others maintainer their images. I understand Postgres as an entity is a bit more nebulous but I personally don't want bitnami's opinions on Postgres - I want to use library/postgres. As such I've been forced to move to 1.0 of the chart because the init container just does not work with the storage I use in my Kubernetes (rook.io as default storage class) - I've been recommending users stay away from any versions beyond 1.0.

Bitnami's Postgres image is honestly in the wrong - if the Bitnami maintainers wanted to do the right thing they'd either add backwards compatibility to their image with the environment variables expected by the upstream docker library postgres or make it so the chart did the translation so I can use a "vendorless" docker image.

[juan131]

I understand your point. @marcoceppi. There are many PostgreSQL images (no matter who's the company/developer maintaining them) and you're free to use the one you consider more reliable.

Bitnami's Postgres image is honestly in the wrong - if the Bitnami maintainers wanted to do the right thing they'd either add backwards compatibility to their image with the environment variables expected by the upstream docker library postgres or make it so the chart did the translation so I can use a "vendorless" docker image.

Bitnami adapted its image to support every missing feature that was available in the docker library postgres image once became the chart maintainer. However, as you mentioned, env. variables use a different format.
If there's something that can be done to improve the image, everyone's free to open an issue and it'll be taken into account on future versions:

https://github.com/bitnami/bitnami-docker-postgresql

You can find the reasons why the maintenance by Bitnami proposal was accepted in the PR below:

https://github.com/helm/charts/pull/8004

On the other hand, with Helm moving to distributed repositories soon, there won't be incubator/stable charts anymore. You'll be free to use the chart you desire from the charts repository you like.

[arianitu]

I'm so confused, how can you switch to bitnami/postgresql because of missing features and break backwards compatibility for everyone involved.

Also, what kind of features are we talking about here? If you're talking about replication, I'm pretty sure the chart could have setup replication on its own with the stable image (which is what I'm going to have to manually write now.)

Instead, people using stable/postgresql don't get replication and have to entirely move away from the chart.

I also find the switch a little bit bizarre, what makes you think any of us on stable/postgresql want to switch to bitnami/postgresql, an image that works entirely different?

Please develop an internal policy for charts and changing the underlying containers/functionality since this has ruined the trust I had in helm/charts. It's just not realistic to use any of these in production.

[carrodher]

I'm so confused, how can you switch to bitnami/postgresql because of missing features and break backwards compatibility for everyone involved.

Also, what kind of features are we talking about here? If you're talking about replication, I'm pretty sure the chart could have setup replication on its own with the stable image (which is what I'm going to have to manually write now.)

Instead, people using stable/postgresql don't get replication and have to entirely move away from the chart.

I also find the switch a little bit bizarre, what makes you think any of us on stable/postgresql want to switch to bitnami/postgresql, an image that works entirely different?

Please develop an internal policy for charts and changing the underlying containers/functionality since this has ruined the trust I had in helm/charts. It's just not realistic to use any of these in production.

Hi, I just provide an answer in the issue that you have opened (https://github.com/helm/charts/issues/10451), to avoid duplicate conversations in different places (I have also seen a similar comment in a pull request https://github.com/helm/charts/pull/8004#issuecomment-452119963), do you think it is good to continue the discussion in a single thread (https://github.com/helm/charts/issues/10451)?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[docktermj]

This issue has been marked as closed, but is there a fix? I've been wrestling with it at https://github.com/bitnami/charts/issues/1210 and can't seem to find an answer for using stable/postgresql nor bitnami/postgresql.

[juan131]

Hi @docktermj

Thanks for reporting the issue!!

I noticed you created a new issue (https://github.com/helm/charts/issues/14390) to report the same problem with non-root permissions.

I'd rather continue the discussion in that issue to avoid confusion and prevent duplication.

[Glorf]

Hi @docktermj
I've tested that in my case the latest version of chart is working properly. So apparently, my bug was fixed somewhere in chart development. However this behavior seems to be related to multiple problems