Postgres-operator: Cluster dont start

[mobilesfinks]

this is my second attempt to create a cluster postgres through this operator.
In the first case, i acted in accordance with the instructions in the Readme and deploy it in the minicube. Everything worked out with minikube.

Now I deploy in a separate namespace on the working cluster k8s.
The operator unwrapped, finally. But the cluster does not up.

first problem: operator want to use ConfigMap and infrastructure-roles only from Default Namespace
second problem: The cluster does not start

# kubectl -n inner get postgresql
NAME         AGE
pg-cluster   21m

cluster definition:

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: pg-cluster
spec:
  teamId: "PG"
  volume:
    size: 1Gi
    storageclass: rbd
  numberOfInstances: 2
  users:
    # database owner
    root:
    - superuser
    - createdb
    cachet:
    - createdb
    sentry:

    # role for application foo
    # foo_user:

  #databases: name->owner
  databases:
    test: root
    cachet: cachet
    sentry: sentry
  postgresql:
    version: "10"

in operator logs:

# kubectl -n inner logs postgres-operator-56ccbf8d9c-tj8hk
time="2018-02-26T16:26:45Z" level=info msg="customResourceDefinition \"postgresqls.acid.zalan.do\" is already registered" pkg=controller
time="2018-02-26T16:26:48Z" level=info msg="started working in background" pkg=controller
time="2018-02-26T16:26:48Z" level=info msg="listening on :8080" pkg=apiserver
time="2018-02-26T16:26:48Z" level=debug msg="new node has been added: \"/kuber03.lan.local\" ()" pkg=controller
time="2018-02-26T16:26:48Z" level=debug msg="new node has been added: \"/kuber05.lan.local\" ()" pkg=controller
time="2018-02-26T16:26:48Z" level=debug msg="new node has been added: \"/kuber06.lan.local\" ()" pkg=controller
time="2018-02-26T16:26:48Z" level=debug msg="new node has been added: \"/kuber00.lan.local\" ()" pkg=controller
time="2018-02-26T16:26:48Z" level=info msg="no clusters running" pkg=controller
time="2018-02-26T16:31:49Z" level=info msg="no clusters running" pkg=controller
time="2018-02-26T16:40:44Z" level=info msg="no clusters running" pkg=controller
time="2018-02-26T16:46:48Z" level=info msg="no clusters running" pkg=controller
time="2018-02-26T16:51:49Z" level=info msg="no clusters running" pkg=controlle

I still do not understand where else I can look to understand the cluster deploy

[Jan-M]

First question would be are you running the most recent master version? We recently improved the way the multi namespace support works to make it more clear and consistent. And that should force the operator to look for its config map and infrastructure roles in the namespace it runs in. And at the same time it will listen to CRDs in all namespaces and create clusters and secrets there.

[mobilesfinks]

Yes, i use registry.opensource.zalan.do/acid/postgres-operator:c17aabb most recently in current master

[Jan-M]

Second you can make your users definition simpler, no flags needed anymore unless you really want the user to be able to create more DBs. But we normally don't give out that power neither superuser.

In that case it comes down to:

users:
  root: []
  cachet: []
  sentry: []

Bump the image version to: 0f392c2 the version u use does work with default but not more, it was hard coded in the manifest back then to make sure it worked.

[mobilesfinks]

change image version help
now operator can view cluster definition in namespace

time="2018-02-26T19:38:30Z" level=debug msg="syncing services" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-26T19:38:30Z" level=debug msg="syncing master service" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-26T19:38:30Z" level=info msg="could not find the cluster's master endpoint" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-26T19:38:30Z" level=info msg="created missing master endpoint \"inner/pg-cluster\"" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-26T19:38:30Z" level=warning msg="error while syncing cluster state: could not sync services: could not sync master service: could not get master service: services \"pg-cluster\" is forbidden: User \"system:serviceaccount:inner:operator\" cannot get services in the namespace \"inner\"" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-26T19:38:30Z" level=error msg="could not sync cluster: could not sync services: could not sync master service: could not get master service: services \"pg-cluster\" is forbidden: User \"system:serviceaccount:inner:operator\" cannot get services in the namespace \"inner\"" cluster-name=inner/pg-cluster pkg=controller worker=0
time="2018-02-26T19:38:30Z" level=debug msg="cluster already exists" cluster-name=inner/pg-cluster pkg=controller worker=0

Already good. It is understood that it is necessary to understand with the rbac, but I will adjust it tomorrow. Thanks for the help.

[mobilesfinks]

Now problem in rbac with services and statefulsets

time="2018-02-27T08:05:02Z" level=error msg="could not sync cluster: could not sync statefulsets: could not get statefulset: statefulsets.apps \"pg-cluster\" is forbidden: User \"system:serviceaccount:inner:operator\" cannot get statefuls
ets.apps in the namespace \"inner\"" cluster-name=inner/pg-cluster pkg=controller worker=0

In ClusterRole

- apiGroups:
  - ""
  resources:
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - patch
  - watch
  - list

after change verbs to "*" nothing changes.

Operator can view Services after i change rule like this

- apiGroups:
  - ""
  resources:
  - service
  - services
  verbs:
  - '*'

[alexeyklyukin]

@mobilesfinks
Please, try with the definitions from the commit:
https://github.com/zalando-incubator/postgres-operator/blob/d15682b7bddcee3060d2a7e5561ac6b5ac3c021f/manifests/operator-rbac.yaml

it changes the statefulset rules, adding list and defining them in the apps API group instead of an empty one.
We don't need watch, as the operator doesn't watch statefulsets.

[mobilesfinks]

With this permissions i get error

time="2018-02-27T08:42:24Z" level=info msg="\"ADD\" event has been queued" cluster-name=inner/pg-cluster pkg=controller worker=0
time="2018-02-27T08:42:24Z" level=info msg="creation of the cluster started" cluster-name=inner/pg-cluster pkg=controller worker=0
time="2018-02-27T08:42:24Z" level=info msg="endpoint \"inner/pg-cluster\" has been successfully created" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T08:42:24Z" level=error msg="could not create cluster: could not create master service: services is forbidden: User \"system:serviceaccount:inner:operator\" cannot create services in the namespace \"inner\"" cluster-name=inner/pg-cluster pkg=controller worker=0

[mobilesfinks]

All seems work after change service to services in clusterrole definition and add verbs list and watch

- apiGroups:
  - ""
  resources:
  - services      <<< here
  verbs:
  - create
  - delete
  - get
  - patch
  - watch           <<< here
  - ist             <<< here

[alexeyklyukin]

@mobilesfinks good catch on the 'services'. However, I think list and watch are not necessary, the operator doesn't watch on services, neither it calls list on them. Which error did you get when 'list' and 'watch' were not there?

[mobilesfinks]

There are still problems

time="2018-02-27T09:50:11Z" level=debug msg="secret \"inner/standby.pg-cluster.credentials\" already exists, fetching its password" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:11Z" level=debug msg="syncing services" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:11Z" level=debug msg="syncing master service" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:11Z" level=debug msg="syncing replica service" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:11Z" level=debug msg="syncing statefulsets" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:11Z" level=debug msg="syncing roles" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:26Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:41Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:50:56Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:51:11Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:51:26Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:51:41Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:51:56Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:52:11Z" level=error msg="could not connect to PostgreSQL database: dial tcp 10.96.208.155:5432: i/o timeout" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:52:11Z" level=warning msg="error while syncing cluster state: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:52:11Z" level=warning msg="could not set \"SyncFailed\" status for the cluster: the server rejected our request due to an error in our request (patch postgresqls.acid.zalan.do pg-cluster)" cluster-name=inner/pg-cluster pkg=cluster worker=0
time="2018-02-27T09:52:11Z" level=error msg="could not sync cluster: could not sync roles: could not init db connection: could not init db connection: still failing after 8 retries" cluster-name=inner/pg-cluster pkg=controller worker=0

Error from pod logs
```
2018-02-27 08:49:02,083 WARNING: Can not get cluster from dcs
2018-02-27 08:49:02,095 ERROR: get_cluster
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 134, in _load_cluster
response = self.retry(self._api.list_namespaced_kind, self._namespace, label_selector=self._label_selector)
File "/usr/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 95, in retry
return self._retry.copy()(args, *kwargs)
File "/usr/lib/python3.6/site-packages/patroni/utils.py", line 259, in __call__
return func(args, *kwargs)
File "/usr/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 49, in wrapper
return getattr(self._api, func)(args, *kwargs)
File "/usr/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 12395, in list_namespaced_config_map
(data) = self.list_namespaced_config_map_with_http_info(namespace, **kwargs)
File "/usr/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 12497, in list_namespaced_config_map_with_http_info
collection_formats=collection_formats)
File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/usr/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/usr/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Tue, 27 Feb 2018 08:49:02 GMT', 'Content-Length': '266'})
HTTP response body: {"kind":"Status","apiVers^C
````

[alexeyklyukin]

Are you using the same serviceAccount for the operator configuration (service_account_name ) as for the operator deployment?

If you do, then there are 2 options you can fix the issues above:

• by adding list for configMaps to the service account you use (for both operator deployment and inside the operator configuration)
• by using a different service Account for the service_account_name with the RBAC defined at https://github.com/zalando/patroni/blob/master/kubernetes/patroni_k8s.yaml.

The reason is that the cluster the operator executes uses Patroni in order to do failovers, and Patroni has its own requirements for accessing kubernetes objects. Without it, Patroni is not able to promote any node to the primary, therefore, the SQL statements that creates users/databases in the database on the primary fail, as there is no primary.

[mobilesfinks]

For operator configuration i use serviceAccountName: operator
I did not understand the difference
what is the difference between an operator and its deployment
Deployment definition:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: postgres-operator
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: postgres-operator
    spec:
      serviceAccountName: operator
      containers:
      - name: postgres-operator
        image: registry.opensource.zalan.do/acid/postgres-operator:c17aabb
        imagePullPolicy: IfNotPresent
        env:
        # uncomment to overwrite a similar setting from operator configmap
        # if set to the empty string, watch the operator's own namespace
        # if set to the "*", listen to all namespaces
        - name: WATCHED_NAMESPACE
          value: 'inner'
          # valueFrom:
          #   fieldRef:
          #     fieldPath: metadata.namespace
        - name: CONFIG_MAP_NAME
          value: "postgres-operator"

[alexeyklyukin]

The operator is configured using the postgres-operator configmap (specified in the CONFIG_MAP_NAME). Inside that configmap, one can specify the service account that the pods created by the operator will run with. That service account requires different permissions from the operator itself.

[mobilesfinks]

Yes, in configmap use same service account
service_account_name: operator

[mobilesfinks]

fix permissions for configmaps
Now both pods have in log:

2018-02-27 11:13:19,436 INFO: waiting for leader to bootstrap
2018-02-27 11:13:28,468 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:13:28,471 INFO: waiting for leader to bootstrap
2018-02-27 11:13:38,508 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:13:38,511 INFO: waiting for leader to bootstrap
2018-02-27 11:13:48,540 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:13:48,542 INFO: waiting for leader to bootstrap
2018-02-27 11:13:58,573 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:13:58,576 INFO: waiting for leader to bootstrap

and nothing happens anymore

[alexeyklyukin]

Strange, do you have a leftover configmap pg-cluster-config from your previous experiments? If you do, please, delete it, one of the pods would pick up the leader role and the other would follow as a replica.

[mobilesfinks]

Ok.
I delete cluster definition (also delete by hand service, endpoints, pdb)
Also delete configmaps for cluster (not for operator)
Apply pg-cluster definition
Stateful set was deployed, in pod logs i see next errors

 2018-02-27 11:48:10,905 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:48:10,910 INFO: failed to acquire initialize lock
2018-02-27 11:48:20,909 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:48:20,914 INFO: failed to acquire initialize lock
2018-02-27 11:48:34,222 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=3.3333333333333335)",)': /api/v1/namespaces/inner/pods?labelSelector=application%3Dspilo%2Cversion%3Dpg-cluster
2018-02-27 11:48:34,222 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='10.96.0.1', port=443): Read timed out. (read timeout=3.3333333333333335)",)': /api/v1/namespaces/inner/pods?labelSelector=application%3Dspilo%2Cversion%3Dpg-cluster
2018-02-27 11:48:35,286 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:48:35,296 INFO: failed to acquire initialize lock
2018-02-27 11:48:40,903 INFO: Lock owner: None; I am pg-cluster-0
2018-02-27 11:48:40,908 INFO: failed to acquire initialize lock

[alexeyklyukin]

Seems like Patroni reporting it cannot access the master, however, the operator managed to create all necessary objects via the calls to the master, so it looks like another permission issue.

You can either to create a new postgres cluster from scratch instead of reusing the same name, or to assign a cluster-admin to the service account used by the pods to pinpoint the issue.

[mobilesfinks]

OK, i Delete all resources associated with operator and cluster from every where (inner and default)
Then i add configmaps to inner ns. Set serviceaccounts with admin privileges and deploy operator and pg-cluster
cluster leader pg-cluster-0 bootstrap correct, i think

init db and other logs
...
2018-02-27 15:57:17,631 INFO: no action.  i am the leader with the lock
2018-02-27 15:57:18.435 31 LOG {ticks: 0, maint: 0, retry: 0}
2018-02-27 15:57:27,533 INFO: Lock owner: pg-cluster-0; I am pg-cluster-0
2018-02-27 15:57:27,624 INFO: no action.  i am the leader with the lock
2018-02-27 15:57:37,531 INFO: Lock owner: pg-cluster-0; I am pg-cluster-0
2018-02-27 15:57:37,579 INFO: no action.  i am the leader with the lock

but pg-cluster-1 cannot bootstrap from leader

2018-02-27 15:57:57,595 ERROR: failed to bootstrap from leader 'pg-cluster-0'
2018-02-27 15:57:57,595 INFO: Removing data directory: /home/postgres/pgdata/pgroot/data
2018-02-27 15:57:57,610 INFO: trying to bootstrap from leader 'pg-cluster-0'
2018-02-27 15:58:07,668 INFO: Lock owner: pg-cluster-0; I am pg-cluster-1
2018-02-27 15:58:07,669 ERROR: failed to bootstrap from leader 'pg-cluster-0'
2018-02-27 15:58:07,670 INFO: Removing data directory: /home/postgres/pgdata/pgroot/data
2018-02-27 15:58:07,683 INFO: trying to bootstrap from leader 'pg-cluster-0'

[FxKu]

Closing due to inactivity. Feel free to reopen, if the error still exists. Hope you got it working :)
If not, you might try the newest version.