Postgres-operator: Check for existence of service account on cluster creation

[aixeshunter]

Issue

Hi, I created a namespace named testdb:

[root@master1 ~]# kubectl create ns testdb
namespace/testdb created

And then, I created a postgresql resource:

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: acid-aixeshunter
spec:
  teamId: "ACID"
  volume:
    size: 1Gi
  numberOfInstances: 2
  dockerImage: registry.opensource.zalan.do/acid/spilo-cdp-10:1.5-p35
  users:
    aixeshunter:
    - superuser
    - createdb
  enableMasterLoadBalancer: true
  enableReplicaLoadBalancer: true
  #databases: name->owner
  databases:
    testdb: aixeshunter
  postgresql:
    version: "10"
  resources:
    requests:
      cpu: 10m
      memory: 100Mi
    limits:
      cpu: 500m
      memory: 500Mi

The statefuleset were always pending, and I found the logs:

kubectl describe sts -n testdb acid-aixeshunter

Events:
  Type     Reason            Age               From                    Message
  ----     ------            ----              ----                    -------
  Normal   SuccessfulCreate  1m                statefulset-controller  create Claim pgdata-acid-aixeshunter-0 Pod acid-aixeshunter-0 in StatefulSet acid-aixeshunter success
  Warning  FailedCreate      4s (x16 over 1m)  statefulset-controller  create Pod acid-aixeshunter-0 in StatefulSet acid-aixeshunter failed error: pods "acid-aixeshunter-0" is forbidden: error looking up service account testdb/zalando-postgres-operator: serviceaccount "zalando-postgres-operator" not found

Question

Do I need to create a service account named zalando-postgres-operator by file operator-service-account-rbac.yaml in namespace testdb manually？

[erthalion]

Yes, every new namespace should have a service account, that you need to create manually, see the documentation:

The watched namespace also needs to have a (possibly different) service account in the case database pods need to talk to the Kubernetes API (e.g. when using Kubernetes-native configuration of Patroni).

[aixeshunter]

@erthalion Hi, I found a confused situation.
I created a new namespace named testdb and created a postgresql cluster in it:

[root@master1 ~]# kubectl create ns testdb
namespace/testdb created

[root@master1 ~]# kubectl get serviceaccount -n testdb
NAME      SECRETS   AGE
default   1         17s

[root@master1 ~]# kubectl apply -f pgs-sample.yaml -n testdb
postgresql.acid.zalan.do/acid-aixeshunter created

[root@master1 ~]# kubectl get serviceaccount -n testdb
NAME      SECRETS   AGE
default   1         1m

[root@master1 ~]# kubectl get sts -n testdb
NAME               DESIRED   CURRENT   AGE
acid-aixeshunter   1         0         1m

And there was a known error in statefulset"

kubectl describe sts -n testdb acid-aixeshunter

Events:
  Type     Reason            Age                From                    Message
  ----     ------            ----               ----                    -------
  Normal   SuccessfulCreate  1m                 statefulset-controller  create Claim pgdata-acid-aixeshunter-0 Pod acid-aixeshunter-0 in StatefulSet acid-aixeshunter success
  Warning  FailedCreate      18s (x16 over 1m)  statefulset-controller  create Pod acid-aixeshunter-0 in StatefulSet acid-aixeshunter failed error: pods "acid-aixeshunter-0" is forbidden: error looking up service account testdb/zalando-postgres-operator: serviceaccount "zalando-postgres-operator" not found

In operator log was:

time="2018-12-29T01:31:40Z" level=info msg="\"ADD\" event has been queued" cluster-name=testdb/acid-aixeshunter pkg=controller worker=0
time="2018-12-29T01:31:40Z" level=info msg="creation of the cluster started" cluster-name=testdb/acid-aixeshunter pkg=controller worker=0
time="2018-12-29T01:31:40Z" level=info msg="endpoint \"testdb/acid-aixeshunter\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=info msg="master service \"testdb/acid-aixeshunter\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=info msg="replica service \"testdb/acid-aixeshunter-repl\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=debug msg="resolved a conflict of role \"aixeshunter\" between manifest role and infrastructure role to infrastructure role" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=debug msg="team API is disabled, returning empty list of members" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=info msg="users have been initialized" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=debug msg="created new secret \"testdb/aixeshunter.acid-aixeshunter.credentials\", uid: \"7e547bb4-0b09-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:41Z" level=debug msg="created new secret \"testdb/postgres.acid-aixeshunter.credentials\", uid: \"7e91bae9-0b09-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:42Z" level=debug msg="created new secret \"testdb/standby.acid-aixeshunter.credentials\", uid: \"7eafb3b2-0b09-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:42Z" level=info msg="secrets have been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:42Z" level=info msg="pod disruption budget \"testdb/postgres-acid-aixeshunter-pdb\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:42Z" level=debug msg="created new statefulset \"testdb/acid-aixeshunter\", uid: \"7ec78a65-0b09-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:42Z" level=info msg="statefulset \"testdb/acid-aixeshunter\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:31:42Z" level=info msg="waiting for the cluster being ready" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0

But, when I deleted the operator pod and did the above action again, the operator pod log was:

time="2018-12-29T01:36:33Z" level=info msg="\"ADD\" event has been queued" cluster-name=testdb/acid-aixeshunter pkg=controller worker=0
time="2018-12-29T01:36:33Z" level=info msg="creating pod service account in the namespace testdb" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="successfully deployed the pod service account zalando-postgres-operator to the testdb namespace" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="Creating the role binding zalando-postgres-operator in the namespace testdb" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="successfully deployed the role binding for the pod service account \"zalando-postgres-operator\" to the \"testdb\" namespace" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="creation of the cluster started" cluster-name=testdb/acid-aixeshunter pkg=controller worker=0
time="2018-12-29T01:36:34Z" level=info msg="endpoint \"testdb/acid-aixeshunter\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:34Z" level=info msg="master service \"testdb/acid-aixeshunter\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=info msg="replica service \"testdb/acid-aixeshunter-repl\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=debug msg="resolved a conflict of role \"aixeshunter\" between manifest role and infrastructure role to infrastructure role" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=debug msg="team API is disabled, returning empty list of members" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=info msg="users have been initialized" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=debug msg="created new secret \"testdb/aixeshunter.acid-aixeshunter.credentials\", uid: \"2d8b1a9f-0b0a-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=debug msg="created new secret \"testdb/postgres.acid-aixeshunter.credentials\", uid: \"2d905302-0b0a-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=debug msg="created new secret \"testdb/standby.acid-aixeshunter.credentials\", uid: \"2da58d3f-0b0a-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=info msg="secrets have been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:35Z" level=info msg="pod disruption budget \"testdb/postgres-acid-aixeshunter-pdb\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:36Z" level=debug msg="created new statefulset \"testdb/acid-aixeshunter\", uid: \"2dd27304-0b0a-11e9-8a98-52540052d19a\"" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:36Z" level=info msg="statefulset \"testdb/acid-aixeshunter\" has been successfully created" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:36Z" level=info msg="waiting for the cluster being ready" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0
time="2018-12-29T01:36:39Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=testdb/acid-aixeshunter pkg=cluster worker=0

that confused me was:

time="2018-12-29T01:36:33Z" level=info msg="creating pod service account in the namespace testdb" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="successfully deployed the pod service account zalando-postgres-operator to the testdb namespace" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="Creating the role binding zalando-postgres-operator in the namespace testdb" pkg=controller
time="2018-12-29T01:36:33Z" level=info msg="successfully deployed the role binding for the pod service account \"zalando-postgres-operator\" to the \"testdb\" namespace" pkg=controller

[root@master1 ~]# kubectl get serviceaccount -n testdb
NAME                        SECRETS   AGE
default                     1         35s
zalando-postgres-operator   1         4s

the operator would create serviceaccount automatically, why? Is there need a condition?
I found the related code, but I still not very clear.

[Jan-M]

The operator does create the service account the first time a cluster is created in a new namespace.

If you delete the namespace and start from scratch it fails, it will not retry in a namespace as long as the operator is running (this state it keeps in memory).

[aixeshunter]

The operator does create the service account the first time a cluster is created in a new namespace.

If you delete the namespace and start from scratch it fails, it will not retry in a namespace as long as the operator is running (this state it keeps in memory).

Why not the operator detect service account in namespace before cluster start?
I think the deletion of the namespace is a more frequent operation, service account should always be created when it not exist.

[Jan-M]

I would agree now, that we should always check for the existence of the service account on cluster create.

[erthalion]

Should be fixed in #685