Patroni: kubernetes example does not start

[davecramer]

All three nodes are 'waiting for leader to bootstrap'

[CyberDem0n]

Hi Dave,

I think you have some k8s objects left from the previous attempt, for example ComfigMap patronidemo-config

This ConfigMap contains information that patronidemo cluster was already initialized.

$ kubectl get cm patronidemo-config -o yaml will show you something like:

apiVersion: v1
kind: ConfigMaps
metadata:
  annotations:
    config: '{"postgresql":{"use_pg_rewind":true}}'
    initialize: "6535013044642451482"
  creationTimestamp: 2018-03-20T13:08:13Z
  labels:
    application: patroni
    cluster-name: patronidemo

If you see there metadata.annotations.initialize, that means cluster was already initialized and Patroni will not run initdb the second time.

Just remove it and it will start working: kubectl delete cm patronidemo-config

[davecramer]

@CyberDem0n thanks, that worked. Is there some way to output that in the logs ?

[jberkus]

@CyberDem0n

What if, after 6 cycles of "failure to bootstrap", we added a suggestion in the logs about the initialization key? Since users deleting clusters and forgetting to remove the initialization key is going to be more common than combination failures that the key protects against, in a lot of cases that's exactly what they want to know.

If that works for you, I'll submit a PR.

[davecramer]

Can we not put a reason as to why the we are not booting. IE existing
cluster ...

Dave Cramer

On 22 March 2018 at 18:35, Josh Berkus [e-mail] wrote:

@CyberDem0n https://github.com/cyberdem0n

What if, after 6 cycles of "failure to bootstrap", we added a suggestion
in the logs about the initialization key? Since users deleting clusters and
forgetting to remove the initialization key is going to be more common than
combination failures that the key protects against, in a lot of cases
that's exactly what they want to know.

If that works for you, I'll submit a PR.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/zalando/patroni/issues/641#issuecomment-375479864,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAYz9ow49p7NHBziuKJhQ21sFdarikdGks5thCc7gaJpZM4SyGJA
.

[jberkus]

@davecramer we can't, because the individual node simply doesn't know. It knows that it's waiting for the master to become available so it can bootstrap, which is what it will tell you.

For posterity, let me explain two hypothetical situations, which are identical from the individual node's point of view:

Situation 1: You intentionally delete all pods in order to delete the cluster.

In this situation, there's still an initialization key (on Kubernetes, in the form of a configmap) for the cluster by that name. This key doesn't get deleted when you do a kubectl delete -f patroni.yaml because it was created by the master in the original cluster, and thus isn't part of the yaml file. Operator implementations do delete the key, as a separate action.

This means that when the postgres node comes up, it knows that it doesn't have any data. It checks to see if the cluster has already been initialized, which means it should have data, and finds that yes, it has been. So now it knows that it's not the master (because no data) so it's going to wait for a master to show up, however long that takes.

This is the common "cluster will not start" situation. It's regrettable and provides a bad out-of-box experience but is also unavoidable because of Case 2.

2: Combination Failure

1. You have a 3-pod postgres cluster, postgres-0,-1,-2.
2. postgres-0 is currently the master.
3. Your kubernetes cluster goes down, taking all nodes offline.
4. Your kubernetes cluster comes back up, but the physical node housing postgres-0 does not, and the nodes housing postgres-1 and postgres-2 are slow to rejoin the cluster.
5. As a result, kubernetes schedules a replacement for postgres-0 before postgres-1 or postgres-2 have come back up.
6. postgres-0 sees that the master key has expired.
7. postgres-0 sees the initialization key, so it knows that there must be data somewhere, so it waits for a pod with data to show up.
8. postgres-1 reconnects to the cluster, obtains the master key, and promotes itself to master.
9. postgres-0 starts replication from postgres-1.

The critical part of this sequence is what happens at step 7. From the perspective of postgres-0, its situation is absolutely identical to Case 1. There's an initialization key, the master key is invalid, and it has no data. At this point, there's no way for postgres-0 to know, or to write to its log, which of the two cases you are in because it simply doesn't know.

This is why my proposal for making this situation better is to wait for 6 polling cycles, or so, and then emit a more generic error message, like:

"Still waiting for master to bootstrap. Did you recreate a cluster with the same name as a deleted cluster? If so, you may need to remove the initialization key at {configmap-location}. WARNING: removing the initialization key on a live cluster will erase all data."

[bradnicholson]

WARNING: removing the initialization key on a live cluster will erase all data.

@jberkus Is this specific to the k8s implementation? With etcd, it won't delete the PG data. It will attempt to initialize a new cluster with the exiting data.

[jberkus]

@bradnicholson no, it's not, and no, it won't.

The erasure is specifically if you're still at step 7 in the sequence above. That is, postgres-0 has no data and postgres-1 and postgres-2 haven't connected yet.

If you delete the initialization key at that stage, then postgres-0 will intialize its data directory and claim the master key. postgres-1 and postgres-2 will then come up, and discover that postgres-0 is the master, and attempt to restart replication from it. In a best-case scenario, they will simply fail to replicate and wait for administator intervention. However, between pg_rewind and however you have your health checks set up, what can happen is that postgres-1,2 can be failed out (or get corrupted) and restarted replicating from postgres-0's empty database.

So I suppose "... can lead to erasure of all of your data." would me a more accurate way to put it.

[ants]

I think we could do better error messages by storing extra metadata in the initialize key. When grabbing the initialize key for bootstrapping, instead of an empty value we could store a JSON object with a bunch of metadata like node name and current time. When registering a freshly bootstrapped (or existing) database in DCS, instead of just the sysid, we could also store the node name and current time.

Then we could have messages like:

• Waiting for node1 to finish cluster bootstrap (started at 2018-03-23 23:00:00+02). Remove initialize key manually if bootstrap process has failed.
• Cluster initialized at 2018-03-23 23:00:00+02 by node1, but no bootstrap source available. Waiting for a node with data to appear.

At the very least we could distinguish error messages between empty initialize key and a valid sysid.

Another option would be to have a ttl on initialize key too and have the bootstrap process keep updating it. Much simpler now that Alexander made bootstrap async. This would give automatic retry for failed bootstrap.

[davecramer]

@jberkus thanks for the detailed explanation. I think at the very least adding an FAQ to the README would be useful

[jberkus]

@ants adding metadata seems like a good idea to me. Particularly, this message option:

Cluster initialized at 2018-03-23 23:00:00+02 by node1, but no bootstrap source available. Waiting for a node with data to appear.

I am not in favor of having a TTL for the initialization key, because that could easily result in database deletion in the event of cluster-wide downtime.

[ants]

TTL would only be for the case when bootstrap is running. Once we have a valid sysid it would get installed permanently. The handover process from bootstrap to normal operation will still have some interesting race conditions, at a minimum I don't think we can do something useful about the master host being incinerated after bootstrap completed, but before first standby was able to clone it or a backup was done.

[jberkus]

@ants seems like in that case the TTL would only help us for clusters that fail to start, and need to be started again, then? Or am I not getting this?

[ants]

Right. It would help with Patroni failing during bootstrap, either due to system failure or administrator intervention. Not really a huge use case which is why I am unsure if it is even worth implementing. Based on my experience system failure is rare enough to be a real problem and administrator failure would be solved by better error messages.

[jberkus]

Well, I think it's worth doing for failure during bootstrap; consider deployment on a kubernetes cluster which is nearly out of PV space. In that case, you want auto-redeployment to "just work" when the storage is fixed. But it doesn't solve the more general case.

One can argue that the more general case is only solved by using an operator.

[lwsbox]

@bradnicholson no, it's not, and no, it won't.

The erasure is specifically if you're still at step 7 in the sequence above. That is, postgres-0 has no data and postgres-1 and postgres-2 haven't connected yet.

If you delete the initialization key at that stage, then postgres-0 will intialize its data directory and claim the master key. postgres-1 and postgres-2 will then come up, and discover that postgres-0 is the master, and attempt to restart replication from it. In a best-case scenario, they will simply fail to replicate and wait for administator intervention. However, between pg_rewind and however you have your health checks set up, what can happen is that postgres-1,2 can be failed out (or get corrupted) and restarted replicating from postgres-0's empty database.

So I suppose "... can lead to erasure of all of your data." would me a more accurate way to put it.

If postgres-0 had data before, then restart the kubernetes, the postgres-0 would always clean the data directory? If not , would postgres-0 become a leader again? @jberkus

By the way, with configs and three nodes:
synchronous_mode: true
synchronous_mode_strict: true

one standby would be in "rsync" status, when the primary is dead, how to find the standby with lastest data to promote?

thanks :)

[lwsbox]

I tried to del all pods ,then K8S would restart the pods automatically that would causes this issue.
Reboot cluster is a common scenario, such as power failure, then how to fix this issue to avoid data lost?

[CyberDem0n]

how to fix this issue to avoid data lost?

You have to use persistent volumes for that.

[lwsbox]

how to fix this issue to avoid data lost?

You have to use persistent volumes for that.

I forgot about it.... The default config is:

      volumes:
      - name: pgdata
        emptyDir: {}

thank you :) @CyberDem0n

[henrywangx]

Hi, so persistent volumes can be used?
I realy don't like the spilo, it's too big and complex.
We can easily lose control on spilo.
All we need is the patroni with PV of k8s

[CyberDem0n]

I general I don't see any limitations why PV could not be used.

[jberkus]

I'm goiing to answer @lwsbox's question and then close this issue since it's old and not actually a bug.

If postgres-0 had data before, then restart the kubernetes, the postgres-0 would always clean the data directory? If not , would postgres-0 become a leader again? @jberkus

Well, I'm positing above the situation where we've lost the physical note for postgres-0. In that case, there is no data to recover.

If postgres-0 has data, but is not the master, it would wait for the programmed TTL, and then participate in a leader election.

[henrywangx]

I general I don't see any limitations why PV could not be used.

@CyberDem0n

Hi, thanks for reminding me.
For kubernetes's example, I added a mount path for "/home/postgres/pgdata" with "/tmp/pgdata" and changeed the hostfile's permission by chown -R 999:999 /tmp/pgdata.
It worked.
But I don't know whether it's suitable for a production or not.
And is there more graceful way to do with persistent volume?