Patroni: [GKE] Patroni cluster not starting over kubernetes

[Krishna1408]

Hello,

I use Google Kubernetes Engine (version v1.11) and trying to run my postgresql cluster over kubernetes. I created a fresh cluster with 3 Nodes. I am using the below helm charts as it is without any changes in values:
https://github.com/helm/charts/tree/master/incubator/patroni

Chart runs without any errors and also I see pods are coming. None of the pods are crashing. And I was not able to connect to postgresql using: psql -U admin -h kristest-patroni.default.svc.cluster.local postgres

I then tried to find the postgresql master and got was surprised to see that only the first pod shows up with a spilo-role with role as replica:

kubectl get pods -l spilo-role -L spilo-role
NAME                 READY     STATUS    RESTARTS   AGE       SPILO-ROLE
kristest-patroni-0   1/1       Running   0          14m       replica

I went into logs and saw that only patroni-0 was only pod accepting connections correctly all other pods are having logs as:

2019-02-03 20:37:01,913 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-02-03 20:37:01,913 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2019-02-03 20:37:01,913 INFO Included extra file "/etc/supervisor/conf.d/patroni.conf" during parsing
2019-02-03 20:37:01,913 INFO Included extra file "/etc/supervisor/conf.d/pgq.conf" during parsing
2019-02-03 20:37:01,923 INFO RPC interface 'supervisor' initialized
2019-02-03 20:37:01,923 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-02-03 20:37:01,923 INFO supervisord started with pid 1
2019-02-03 20:37:02,927 INFO spawned: 'cron' with pid 32
2019-02-03 20:37:02,930 INFO spawned: 'patroni' with pid 33
2019-02-03 20:37:02,933 INFO spawned: 'pgq' with pid 34
2019-02-03 20:37:03,554 INFO: Lock owner: None; I am kristest-patroni-2
2019-02-03 20:37:03,563 INFO: waiting for leader to bootstrap
2019-02-03 20:37:04,565 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-03 20:37:04,565 INFO success: patroni entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-03 20:37:04,565 INFO success: pgq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-02-03 20:37:13,554 INFO: Lock owner: None; I am kristest-patroni-2
2019-02-03 20:37:13,564 INFO: waiting for leader to bootstrap
2019-02-03 20:37:23,551 INFO: Lock owner: None; I am kristest-patroni-2
2019-02-03 20:37:23,561 INFO: waiting for leader to bootstrap
2019-02-03 20:37:33,542 INFO: Lock owner: None; I am kristest-patroni-2

I searched kubernetes issues for INFO: waiting for leader to bootstrap but didn't find any solution which works for me. Not sure if I missed something or there are some pre-steps I need to do. I have also tried by providing my usercluster-admin role.

Can someone please help me find a solution ? Would be really grateful for the support.

[laevilgenius]

I had a similar issue. Worked it around by setting the release name to patroni.

[Krishna1408]

Thanks a lot @laevilgenius. What I will need to do if I want to run multiple clusters over helm ? And can I dump data from existing cluster outside kubernetes to patroni one ?

[laevilgenius]

I haven't had time to debug this issue properly yet. Maybe open a similar issue in helm repo?

[noony]

I have the same problem, it seems that we have a viable cluster if the release name is not patroni. Even if we delete kubernetes endpoint.

[sarneaud]

I think it might be the endpoint used for locking.

When k8s endpoints are enabled, Patroni creates an endpoint named $CLUSTER_NAME-config, which Helm doesn't know about and can't manage, so I had a stale one from a previous deployment. I'm not using that Helm chart, but changing the release name would cause the new deployment to use a different endpoint.

This got my cluster to boot:

kubectl get endpoints
kubectl delete endpoints postgres-config

(Change the endpoint name to match your deployment, of course.)

[Krishna1408]

Thanks for the info @sarneaud, Pretty sure others will find it useful. In my case, It was a fresh cluster and patroni helm chart was the first thing I deployed.

[noony]

Hi,
If I'm deploying with helm with a name different than patroni (ie: toto) cluster stay stuck with one replica :

helm install --name toto -f values.yaml incubator/patroni with default values (except persistentVolume.enabled: false)

```Every 2,0s: kubectl get pods -l spilo-role -L spilo-role PAR-TColomb.local: Thu Feb 7 18:24:34 2019

NAME READY STATUS RESTARTS AGE SPILO-ROLE
toto-patroni-0 1/1 Running 0 1m replica

Log of toto-patroni-0 : 

2019-02-07 17:23:42,093 INFO: Could not take out TTL lock
2019-02-07 17:23:42,110 INFO: initialized a new cluster
2019-02-07 17:23:52,104 INFO: Could not take out TTL lock
2019-02-07 17:23:53,107 INFO reaped unknown pid 69
2019-02-07 17:23:53,107 INFO reaped unknown pid 71
2019-02-07 17:23:53,159 INFO: demoted self after trying and failing to obtain lock
2019-02-07 17:23:53,160 INFO: closed patroni connection to the postgresql cluster
2019-02-07 17:23:53,199 INFO: postmaster pid=181
/var/run/postgresql:5432 - no response
2019-02-07 17:23:53 UTC [181]: [1-1] 5c5c69a9.b5 0 LOG: listening on IPv4 address "0.0.0.0", port 5432
2019-02-07 17:23:53 UTC [181]: [2-1] 5c5c69a9.b5 0 LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-02-07 17:23:53 UTC [181]: [3-1] 5c5c69a9.b5 0 LOG: redirecting log output to logging collector process
2019-02-07 17:23:53 UTC [181]: [4-1] 5c5c69a9.b5 0 HINT: Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2019-02-07 17:23:54,240 INFO: establishing a new patroni connection to the postgres cluster
2019-02-07 17:23:54,268 INFO: Could not take out TTL lock
2019-02-07 17:23:54,291 INFO: following new leader after trying and failing to obtain lock
2019-02-07 17:24:04,270 INFO: Could not take out TTL lock
2019-02-07 17:24:04,293 INFO: following new leader after trying and failing to obtain lock
2019-02-07 17:24:14,277 INFO: Could not take out TTL lock
2019-02-07 17:24:14,298 INFO: following new leader after trying and failing to obtain lock
2019-02-07 17:24:24,260 INFO: Could not take out TTL lock

endpoints contain this : 

kubectl get endpoints -o yaml
apiVersion: v1
items:

• apiVersion: v1
  kind: Endpoints
  metadata:
  creationTimestamp: "2019-02-07T17:51:14Z"
  labels:
  app: patroni
  chart: patroni-0.11.0
  heritage: Tiller
  release: toto
  name: toto-patroni
  namespace: toto
  resourceVersion: "9173309"
  selfLink: /api/v1/namespaces/toto/endpoints/toto-patroni
  uid: f65777bf-2b00-11e9-a03c-0ad6bb79e722
• apiVersion: v1
  kind: Endpoints
  metadata:
  annotations:
  config: '{"loop_wait":10,"maximum_lag_on_failover":33554432,"postgresql":{"parameters":{"archive_mode":"on","archive_timeout":"1800s","autovacuum_analyze_scale_factor":0.02,"autovacuum_max_workers":5,"autovacuum_vacuum_scale_factor":0.05,"checkpoint_completion_target":0.9,"hot_standby":"on","log_autovacuum_min_duration":0,"log_checkpoints":"on","log_connections":"on","log_disconnections":"on","log_line_prefix":"%t
  [%p]: [%l-1] %c %x %d %u %a %h ","log_lock_waits":"on","log_min_duration_statement":500,"log_statement":"ddl","log_temp_files":0,"max_connections":519,"max_replication_slots":5,"max_wal_senders":5,"tcp_keepalives_idle":900,"tcp_keepalives_interval":100,"track_functions":"all","wal_keep_segments":8,"wal_level":"hot_standby","wal_log_hints":"on"},"use_pg_rewind":true,"use_slots":true},"retry_timeout":10,"ttl":30}'
  initialize: "6655317624700997694"
  creationTimestamp: "2019-02-07T17:51:25Z"
  labels:
  app: toto-patroni
  release: toto
  name: toto-patroni-config
  namespace: toto
  resourceVersion: "9173407"
  selfLink: /api/v1/namespaces/toto/endpoints/toto-patroni-config
  uid: fd374f83-2b00-11e9-a03c-0ad6bb79e722
  kind: List
  metadata:
  resourceVersion: ""
  selfLink: ""
  ```

[noony]

I think this line in chart has to be app: {{ template "patroni.fullname" . }} to work correctly

[smeeklai]

I think this line in chart has to be app: {{ template "patroni.fullname" . }} to work correctly

Now Patroni could elect a master successfully but I got this error afterwards

2019-02-10 14:21:29,063 INFO: Lock owner: pg-test-patroni-0; I am pg-test-patroni-2
2019-02-10 14:21:29,079 INFO: trying to bootstrap from leader 'pg-test-patroni-0'
2019-02-10 14:21:29,079 ERROR: failed to bootstrap from leader 'pg-test-patroni-0'

[smeeklai]

I've confirmed falling back to use etcd worked

[noony]

By setting nameOverride and fullnameOverride with the same name of the release seems to fix all my problems. Maybe a better documentation on this could avoid problems.

[CyberDem0n]

Somebody should really figure out why the helm chart fails and either fix the helm chart or improve the documentation.

[thomasboni]

Hi, I've opened an issue with a PR : https://github.com/helm/charts/issues/12153

[thomasboni]

Is someone using the helm chart and concerned about this issue ? Nobody react in issue https://github.com/helm/charts/issues/12153 and PR https://github.com/helm/charts/pull/12154