We recently had an issue with v0.11.0 that led to losing the entire database. I'm still trying to reproduce and better figure out what happened, but this is important enough I need to reach out now.
We have 1 primary with 2 replicas. We had an app server that has been caught consuming all of the connections to the primary. It connects as the postgres superuser. We patched our Postgres Resource to use a usermap for user.conf, setting a higher max_connections This caused the replicas and then the primary to be recreated. After they came up, the database was empty. It was after this that we then also found that the last backup job had been running for more than 13 days, uncompleted.
I haven't been able to reproduce the problem to definitely pin the blame on the app server and its misuse of the superuser account. It seems poor that the replicas would stay running if they weren't connected to the primary and it seems very poor that a replica without data and presumably unable to sync from the primary would become the primary in a leader election. and most poor that a replica that was primary would be ok with not being the primary after knowing it had out of date replicas.
Is there any log information or any other debug information that would be helpful to determine why we lost data?
brimstone
All 20 comments
Something similar has happened with me once, where I saw an empty replica becoming the primary during a rolling update. In our case we first increased the number of replicas in one update and then, before it was able to sync we did another update triggering a rolling update.
But it only happened once and with development data. We wasn't able to reproduce the bug. Adding here because seems to be the very same problem.
Since then we have never ran kubedb's postgres with replicas on production data.
lbguilherme
on 19 Jun 2019
I can also confirm that running Postgres with ReplicaCount > 1 leads to data loss.
We're also working on a reliable repro for this problem, however it happens for us quite often.
Possibly related XRefs: https://github.com/kubedb/project/issues/405 https://github.com/kubedb/project/issues/330 https://github.com/kubedb/project/issues/461
/cc @johannwagner @fin-ger
martin31821
on 19 Jun 2019
We are running into the same problem with two databases, where empty replicas are being elected leader. We are not aware of a way to reproduce it consistently, though.
zhaofengli
on 20 Jun 2019
Has anyone experienced this with only a single replica? Or more than two replicas?
We also attempted to replicate in another environment which was in an identical state as the one that lost data and were unable to.
Is anyone actively working on this issue currently?
phutchins
on 21 Jun 2019
We are experiencing this issue quite regularly with 3 replicas.
Philip Hutchins notifications@github.com schrieb am Fr., 21. Juni 2019,
14:56:
Has anyone experienced this with only a single replica? Or more than two
replicas?We also attempted to replicate in another environment which was in an
identical state as the one that lost data and were unable to.Is anyone actively working on this issue currently?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubedb/project/issues/555?email_source=notifications&email_token=ABKM3AMNIOA3DZUSHIBFIS3P3TFYXA5CNFSM4HZDIMW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYIMLEQ#issuecomment-504415634,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABKM3AKO32F65RHDDAXC353P3TFYXANCNFSM4HZDIMWQ
.
martin31821
on 21 Jun 2019
This issue just occurred with me. Updated leaderElection leaseDurationSeconds, and all production data got wiped. Will take a while before we will use KubeDB again.
Wassasin
on 28 Jun 2019
We are seeing this with a single replica. And I think I can see what happens. The context is rolling reboot of nodes so all the pods get moved and restarted a few time.
a database pod is restarted, declares itself as replica, manages to reach the primary, deletes its data directory - https://github.com/kubedb/postgres/blob/30937fd1c6fb8cf9bf21cf340231709b05a8fc13/hack/docker/postgres/10.6/scripts/replica/run.sh#L37 - and starts reading a backup from the primary
while the backup is in progress the primary is restarted,
pg_basebackupfails and we see
I0712 06:32:56.341677 1 [leaderelection.go:187] attempting to acquire leader lease database/FOO-leader-lock...
Running as Replica
Attempting pg_isready on primary
Attempting query on primary
pg_basebackup: could not read COPY data: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_basebackup: removing contents of data directory "/var/pv/data"
I0712 06:33:28.270940 1 logs.go:19] exit status 1
the same secondary is restarted once again (bad node scheduling luck) but this time wins the leader election. The primary now is a server with no data at all.
the original primary comes up as secondary, deletes its data.
This is related #385
The replica/run.sh script contains a race condition between checking it can reach the primary and starting the backup.
It might be a good idea to create the backup into a different directory and only delete the real data directory once the backup has succeeded, but this would only create a less likely race condition.
What I feel would be better was to mark the secondary as "I'm in no state to become a primary" before it deletes the data directory and only remove that marker once done with the backup (a lock file on the persistent volume that also holds the data?). If such a marker exists, the pod should not participate in leader elections after startup.
bodewig
on 12 Jul 2019
The problem with creating a full backup of the data is that you need twice the space for the same data, which can be fairly expensive when dealing with large databases. Imagine a 1TB database, so you need two PVCs with 1TB each...
martin31821
on 12 Jul 2019
Not having any data is not really a better option ;-)
But yes, understood.
bodewig
on 12 Jul 2019
We're also experiencing this problem and getting the database wiped. We're running with one primary and two replicas.
Excds
on 16 Jul 2019
the problem can be avoided by using a wal archiver (wal-g), unfortunatly that is not a solution for many private k8s clusters.
schmitch
on 16 Jul 2019
the problem can be avoided by using a wal archiver (wal-g), unfortunatly that is not a solution for many private k8s clusters.
Oh, thanks a million! I will test this out.
Excds
on 18 Jul 2019
Today we faced the very same issue.
There was one primary and one replica. Both were running in hot mode.
All we did is changed the nodeSelector in deployment configuration. And viola, when the both server got up, there is no more database!!! All data got wiped. Unfortunately the the backup was not working due to some mis-configuration. So yeah, we just lost last 2 days data.
mithucste30
on 24 Jul 2019
This problem is known since January: https://github.com/kubedb/project/issues/385
KubeDB team even started working on it, but the progress unfortunately got stuck by some reason: https://github.com/kubedb/postgres/pull/248
In fact it's a critical issue which makes PostgresDB engine completely unusable for us as soon as we run k8n on AWS EC2 spot instances with autoscaling. Hence individual EC2 instance can be terminated at any moment.
spa-87
on 24 Jul 2019
I can now confirm that using WAL-G archiving _does not_ prevent the data loss.
Setup: one primary and two replicas.
Excds
on 3 Sep 2019
This happened to me as well with the last 2 versions of the operator v0.12.0 and v0.13.0-rc.0. In both cases I had a cluster with a master and replica, and edited the pg resource for the db to change the backup configuration just as suggested here - https://kubedb.com/docs/0.8.0-beta.2/guides/postgres/snapshot/scheduled_backup/#create-postgres-with-backupschedule, to disable and enable the backups.
In each case, the replica was terminated and regenerated and wiped out it's data, then the other database pod was also terminated, restarted and wiped out it's data. This is a major flaw and it's a little surprising to me that it's not been addressed yet, if it was known since January. Anyway I will have to discontinue my use of KubeDB Postgres, it is not ready for production.
bluelion
on 18 Sep 2019
Just hit the same issue with a primary and one replica.
There should be a red note in the docs pointing here until this gets fixed.
tkuther
on 8 Oct 2019
We experienced a full data loss (database wiped)—but we did not change backup settings. Instead, monitoring was added to the Postgres CRD:
spec:
monitor:
agent: prometheus.io/builtin
While I was not able to reproduce it (so far) I'm wondering if all config changes that trigger a rolling restart can hit this issue? All other symptoms match what has been said above.
stephan2012
on 20 Nov 2019
+1 with one primary and 2 warm standby replicas.
Production (beta) cluster, so not great... I'm pretty surprised that something like this isn't line item one on the issues list. It's a bit like playing Russian roulette...
phi2039
on 5 Mar 2020
This issue has been resolved in KubeDB release v2021.03.17 . Please open a new bug if you still see this issue.
tamalsaha
on 19 Mar 2021
Related issues
LinAnt
·
5Comments
pjebs
·
3Comments
spa-87
·
3Comments
olksdr
·
6Comments
JamesLaverack
·
6Comments
Most helpful comment
Just hit the same issue with a primary and one replica.
There should be a red note in the docs pointing here until this gets fixed.