Project: Snapshot stuck on "Running" causing new backups not to be scheduled

[mauritsvdvijgh]

Summary

Backups stopped working because the last backup was stuck in the "Running" state.
Solved it by removing the snapshot.kubedb.com/status=Running label from the snapshot.

KubeDB operator version: 0.12.0.

Complete version

kubectl get snapshots
mysql-shared-20190523-022837   mysql-shared   Succeeded   3d
mysql-shared-20190523-082837   mysql-shared   Succeeded   2d
mysql-shared-20190523-142837   mysql-shared   Running     2d

kubectl describe snapshot mysql-shared-20190523-142837
Name:         mysql-shared-20190523-142837
Namespace:    acceptance
Labels:       kubedb.com/kind=MySQL
              kubedb.com/name=mysql-shared
              snapshot.kubedb.com/status=Running
Annotations:  <none>
API Version:  kubedb.com/v1alpha1
Kind:         Snapshot
Metadata:
  Creation Timestamp:  2019-05-23T14:28:37Z
  Finalizers:
    kubedb.com
  Generation:        1
  Resource Version:  112938740
  Self Link:         /apis/kubedb.com/v1alpha1/namespaces/acceptance/snapshots/mysql-shared-20190523-142837
  UID:               0d6fff65-7d67-11e9-bfdd-00259066e7e4
Spec:
  Database Name:  mysql-shared
  Pod Template:
    Controller:
    Metadata:
    Spec:
      Resources:
...
Status:
  Phase:       Running
  Start Time:  2019-05-23T14:28:37Z
Events:        <none>

2 things of note here:

• Status.Phase says Running
• Label: snapshot.kubedb.com/status=Running

No pods or jobs concerning kubedb snapshots were to be seen in the relevant namespace.

I first tried restarting the operator which changed the status from Running into Failed:

kubectl get snapshots -n acceptance | tail -n3
mysql-shared-20190523-022837   mysql-shared   Succeeded   3d
mysql-shared-20190523-082837   mysql-shared   Succeeded   2d
mysql-shared-20190523-142837   mysql-shared   Failed      2d

New snapshots still were not being scheduled, seemingly because it did not clear the kubedb status label saying that the snapshot was running.

Manually deleteting the snapshot.kubedb.com/status=Running label using

kubectl edit snapshots -n production mysql-shared-20190523-142837

Did work to clear the "locking" mechanism of only running 1 backup at a time per database.
Seems to me that whenever kubedb decides that a backup is failed it should also remove the snapshot.kubedb.com/status=Running label, possibly after checking that no related pod/job is active.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.95. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[ThommyH]

any updates on this? we have the same issue frequently and have to manually fix it every week.

[istepanov]

Same issue here. A snapshot stuck in "Running" state, no new snapshots were created.

[HaveFun83]

Same here

[gb8may]

Same here!