Postgres-operator: Incomplete deletion if resources upon delete postgresql

Created on 2 Feb 2018 · 3Comments · Source: zalando/postgres-operator

Under circumstances (for example, missing pod disruption budget) a kubectl delete postgresql {name} will return successfully, but leaving behind resources. The case of PDB leaves behind the service and the endpoint, for example.

The reason is the early return upon errors in https://github.com/zalando-incubator/postgres-operator/blob/bf4406d2a49deb73830eeb72cb1b35cdccd846e0/pkg/cluster/cluster.go#L568-L583

The logic has to be revised so that we don't leave behind stuff, at least not when the error stems from the nonexistence of the resource.

bug

Source

avaczi

👍2

Most helpful comment

As noted by @CyberDem0n , a proper fix would also include retries upon failed attempts to delete clusters, instead of simply discarding the error and moving on as written here:

https://github.com/zalando-incubator/postgres-operator/blob/master/pkg/controller/postgresql.go#L221

        if err := cl.Delete(); err != nil {
            lg.Errorf("could not delete cluster: %v", err)
        }

mgomezch on 2 Feb 2018

👍2

All 3 comments

Log output for the incomplete deletion:

time="2018-02-02T09:39:05Z" level=info msg="\"DELETE\" event has been queued" cluster-name=default/XXX pkg=controller worker=3
time="2018-02-02T09:39:05Z" level=info msg="deletion of the cluster started" cluster-name=default/XXX pkg=controller worker=3
time="2018-02-02T09:39:05Z" level=debug msg="deleting statefulset" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=info msg="statefulset \"default/XXX\" has been deleted" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=debug msg="deleting pods" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=debug msg="no pods to delete" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=debug msg="deleting PVCs" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=debug msg="no PVCs to delete" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=info msg="not removing secret \"standby.XXX.credentials.postgresql.YYY\" for the system user \"standby\"" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=debug msg="deleting secret \"default/ZZZ.XXX.credentials.postgresql.YYY\"" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=info msg="secret \"default/ZZZ.XXX.credentials.postgresql.YYY\" has been deleted" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=info msg="not removing secret \"postgres.XXX.credentials.postgresql.YYY\" for the system user \"postgres\"" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=debug msg="deleting pod disruption budget" cluster-name=default/XXX pkg=cluster worker=3
time="2018-02-02T09:39:05Z" level=error msg="could not delete cluster: could not delete pod disruption budget: there is no pod disruption budget in the cluster" cluster-name=default/XXX pkg=controller worker=3
time="2018-02-02T09:39:05Z" level=info msg="cluster has been deleted" cluster-name=default/XXX pkg=controller worker=3

mgomezch on 2 Feb 2018

As noted by @CyberDem0n , a proper fix would also include retries upon failed attempts to delete clusters, instead of simply discarding the error and moving on as written here:

https://github.com/zalando-incubator/postgres-operator/blob/master/pkg/controller/postgresql.go#L221

        if err := cl.Delete(); err != nil {
            lg.Errorf("could not delete cluster: %v", err)
        }

mgomezch on 2 Feb 2018

👍2

The delete logic has been revised with #295 and since #654 got merged the logs (especially no pods to delete) from the first answer should not occur anymore. Therefore, I'm closing the issue now.