Velero: Backup marked completed when disk snapshot call ignored by GCP
What steps did you take and what happened:
Created backup (restic is disabled):
$ velero backup create dummy-4325-2020-01-21-1579613072 --wait \
--include-namespaces dummy-4325 \
--include-resources pvc,pv \
--ttl 144h \
--labels transient=true \
--labels app.kubernetes.io/namespace=dummy-4325 \
--labels app.kubernetes.io/component=data \
--labels velero.io/backup=dummy-4325-2020-01-21-1579613072
Backup completed and here is the list of snapshots:
$ velero backup describe --details test-build-4325-data-2020-01-21-1579613072 | grep 'Snapshot ID'
Snapshot ID: restore-dfe96ed9-9a09-4886-c9ce1306-c33d-45f5-b5af-9d9884595644
Snapshot ID: restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3
Snapshot ID: restore-6acd21a8-56bc-49bc-0554d606-54b9-489c-9826-b8787eb3bb25
Snapshot ID: restore-7ecbed72-0fac-4be9-c16020c5-1f14-47fa-9016-9b65b2c8262c
Snapshot ID: restore-9c80eca0-9fe6-4c7b-dc76b78f-d867-44c1-b204-7a276b302818
Snapshot ID: restore-bf5a7c56-a726-40ee-c0d4e95d-f955-48ed-a618-5db32ccb1a44
Disk snapshots were note created for all disks:
$ for i in restore-dfe96ed9-9a09-4886-c9ce1306-c33d-45f5-b5af-9d9884595644 restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3 restore-6acd21a8-56bc-49bc-0554d606-54b9-489c-9826-b8787eb3bb25 restore-7ecbed72-0fac-4be9-c16020c5-1f14-47fa-9016-9b65b2c8262c restore-9c80eca0-9fe6-4c7b-dc76b78f-d867-44c1-b204-7a276b302818 restore-bf5a7c56-a726-40ee-c0d4e95d-f955-48ed-a618-5db32ccb1a44; do echo $i; gcloud compute snapshots list --filter="name=($i)"; done
restore-dfe96ed9-9a09-4886-c9ce1306-c33d-45f5-b5af-9d9884595644
NAME DISK_SIZE_GB SRC_DISK STATUS
restore-dfe96ed9-9a09-4886-c9ce1306-c33d-45f5-b5af-9d9884595644 15 europe-west1-b/disks/restore-dfe96ed9-9a09-4886-afc9-cff9d0cdf75f READY
restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3
NAME DISK_SIZE_GB SRC_DISK STATUS
restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3 5 europe-west1-b/disks/restore-a041c896-cd27-4542-b280-37a83b7d085e READY
restore-6acd21a8-56bc-49bc-0554d606-54b9-489c-9826-b8787eb3bb25
Listed 0 items.
restore-7ecbed72-0fac-4be9-c16020c5-1f14-47fa-9016-9b65b2c8262c
NAME DISK_SIZE_GB SRC_DISK STATUS
restore-7ecbed72-0fac-4be9-c16020c5-1f14-47fa-9016-9b65b2c8262c 40 europe-west1-b/disks/restore-7ecbed72-0fac-4be9-9d86-3f1c8aa3b337 READY
restore-9c80eca0-9fe6-4c7b-dc76b78f-d867-44c1-b204-7a276b302818
NAME DISK_SIZE_GB SRC_DISK STATUS
restore-9c80eca0-9fe6-4c7b-dc76b78f-d867-44c1-b204-7a276b302818 10 europe-west1-b/disks/restore-9c80eca0-9fe6-4c7b-941f-7fefcd42b8ba READY
restore-bf5a7c56-a726-40ee-c0d4e95d-f955-48ed-a618-5db32ccb1a44
Listed 0 items.
What did you expect to happen:
Snapshots for all disks were created.
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs deployment/velero -n velero
{"controller":"backup","key":"backup/dummy-4325-2020-01-21-1579613072","level":"debug","logSource":"pkg/controller/backup_controller.go:162","msg":"Running processBackup","time":"2020-01-21T13:24:34Z"}
{"controller":"backup","key":"backup/dummy-4325-2020-01-21-1579613072","level":"debug","logSource":"pkg/controller/backup_controller.go:169","msg":"Getting backup","time":"2020-01-21T13:24:34Z"}
{"controller":"backup","key":"backup/dummy-4325-2020-01-21-1579613072","level":"debug","logSource":"pkg/controller/backup_controller.go:194","msg":"Preparing backup request","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","controller":"gc-controller","expiration":null,"level":"debug","logSource":"pkg/controller/gc_controller.go:133","msg":"Backup has not expired yet, skipping","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","controller":"gc-controller","expiration":"2020-01-27T13:24:34Z","level":"debug","logSource":"pkg/controller/gc_controller.go:133","msg":"Backup has not expired yet, skipping","time":"2020-01-21T13:24:34Z"}
{"controller":"backup","key":"backup/dummy-4325-2020-01-21-1579613072","level":"debug","logSource":"pkg/controller/backup_controller.go:220","msg":"Running backup","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","controller":"backup","level":"info","logSource":"pkg/controller/backup_controller.go:440","msg":"Setting up backup log","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/controller/backup_controller.go:462","msg":"Setting up backup temp file","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/controller/backup_controller.go:469","msg":"Setting up plugin manager","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/controller/backup_controller.go:473","msg":"Getting backup item actions","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:99","msg":"looking for plugin in registry","name":"velero.io/pod","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","command":"/velero","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:114","msg":"creating new restartable plugin process","name":"velero.io/pod","time":"2020-01-21T13:24:34Z"}
{"args":["/velero","run-plugins","--log-level","debug"],"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"starting plugin","path":"/velero","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin started","path":"/velero","pid":147,"time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"waiting for RPC address","path":"/velero","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/framework/server.go:172","msg":"Setting log level to DEBUG","pluginName":"velero","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"using plugin","time":"2020-01-21T13:24:34Z","version":2}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:99","msg":"looking for plugin in registry","name":"velero.io/pv","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","command":"/velero","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:110","msg":"found preexisting restartable plugin process","name":"velero.io/pv","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:99","msg":"looking for plugin in registry","name":"velero.io/service-account","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","command":"/velero","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:110","msg":"found preexisting restartable plugin process","name":"velero.io/service-account","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/controller/backup_controller.go:479","msg":"Setting up backup store","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:99","msg":"looking for plugin in registry","name":"velero.io/gcp","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","command":"/plugins/velero-plugin-for-gcp","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:114","msg":"creating new restartable plugin process","name":"velero.io/gcp","time":"2020-01-21T13:24:34Z"}
{"address":"/tmp/plugin342398280","backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin address","network":"unix","pluginName":"velero","time":"2020-01-21T13:24:34Z"}
{"args":["/plugins/velero-plugin-for-gcp","--log-level","debug"],"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"starting plugin","path":"/plugins/velero-plugin-for-gcp","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin started","path":"/plugins/velero-plugin-for-gcp","pid":156,"time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"waiting for RPC address","path":"/plugins/velero-plugin-for-gcp","time":"2020-01-21T13:24:34Z"}
{"address":"/tmp/plugin328891762","backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin address","network":"unix","pluginName":"velero-plugin-for-gcp","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"using plugin","time":"2020-01-21T13:24:34Z","version":2}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/backup/backup.go:213","msg":"Writing backup version file","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/backup/backup.go:219","msg":"Including namespaces: dummy-4325","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/backup/backup.go:220","msg":"Excluding namespaces: \u003cnone\u003e","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/backup/backup.go:223","msg":"Including resources: persistentvolumeclaims, persistentvolumes","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","level":"info","logSource":"pkg/backup/backup.go:224","msg":"Excluding resources: \u003cnone\u003e","time":"2020-01-21T13:24:34Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"pods","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"pods","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"persistentvolumeclaims","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:227","msg":"Listing items","namespace":"dummy-4325","resource":"persistentvolumeclaims","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:241","msg":"Retrieved 6 items","namespace":"dummy-4325","resource":"persistentvolumeclaims","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"data-dummy-green-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"data-dummy-green-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-dummy-green-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:330","msg":"Executing custom action","name":"data-dummy-green-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"info","logSource":"pkg/backup/backup_pv_action.go:49","msg":"Executing PVCAction","pluginName":"velero","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:395","msg":"Executing takePVSnapshot","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:99","msg":"looking for plugin in registry","name":"velero.io/gcp","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","command":"/plugins/velero-plugin-for-gcp","kind":"ObjectStore","level":"debug","logSource":"pkg/plugin/clientmgmt/manager.go:110","msg":"found preexisting restartable plugin process","name":"velero.io/gcp","time":"2020-01-21T13:24:36Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:446","msg":"Got volume ID for persistent volume","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z","volumeSnapshotLocation":"default"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:466","msg":"Getting volume information","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z","volumeID":"restore-dfe96ed9-9a09-4886-afc9-cff9d0cdf75f"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:472","msg":"Snapshotting persistent volume","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:36Z","volumeID":"restore-dfe96ed9-9a09-4886-afc9-cff9d0cdf75f"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-dummy-green-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"data-dummy-green-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"data-fake-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"data-fake-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-fake-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:330","msg":"Executing custom action","name":"data-fake-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"info","logSource":"pkg/backup/backup_pv_action.go:49","msg":"Executing PVCAction","pluginName":"velero","time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:395","msg":"Executing takePVSnapshot","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:446","msg":"Got volume ID for persistent volume","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z","volumeSnapshotLocation":"default"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:466","msg":"Getting volume information","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:37Z","volumeID":"restore-a041c896-cd27-4542-b280-37a83b7d085e"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:472","msg":"Snapshotting persistent volume","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:38Z","volumeID":"restore-a041c896-cd27-4542-b280-37a83b7d085e"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-fake-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"data-fake-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"data-redis-dummy-green-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"data-redis-dummy-green-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-redis-dummy-green-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:330","msg":"Executing custom action","name":"data-redis-dummy-green-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"info","logSource":"pkg/backup/backup_pv_action.go:49","msg":"Executing PVCAction","pluginName":"velero","time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:395","msg":"Executing takePVSnapshot","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:446","msg":"Got volume ID for persistent volume","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z","volumeSnapshotLocation":"default"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:466","msg":"Getting volume information","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z","volumeID":"restore-6acd21a8-56bc-49bc-8f4b-54360c558681"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:472","msg":"Snapshotting persistent volume","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:39Z","volumeID":"restore-6acd21a8-56bc-49bc-8f4b-54360c558681"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"pvc-006fd481-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-redis-dummy-green-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"data-redis-dummy-green-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"data-redis-fake-green-master-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"data-redis-fake-green-master-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-redis-fake-green-master-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:330","msg":"Executing custom action","name":"data-redis-fake-green-master-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"info","logSource":"pkg/backup/backup_pv_action.go:49","msg":"Executing PVCAction","pluginName":"velero","time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:395","msg":"Executing takePVSnapshot","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:446","msg":"Got volume ID for persistent volume","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z","volumeSnapshotLocation":"default"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:466","msg":"Getting volume information","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:40Z","volumeID":"restore-7ecbed72-0fac-4be9-9d86-3f1c8aa3b337"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:472","msg":"Snapshotting persistent volume","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:41Z","volumeID":"restore-7ecbed72-0fac-4be9-9d86-3f1c8aa3b337"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-redis-fake-green-master-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"data-redis-fake-green-master-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"data-redis-boo-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"data-redis-boo-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-redis-boo-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:330","msg":"Executing custom action","name":"data-redis-boo-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"info","logSource":"pkg/backup/backup_pv_action.go:49","msg":"Executing PVCAction","pluginName":"velero","time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:395","msg":"Executing takePVSnapshot","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:446","msg":"Got volume ID for persistent volume","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z","volumeSnapshotLocation":"default"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:466","msg":"Getting volume information","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z","volumeID":"restore-9c80eca0-9fe6-4c7b-941f-7fefcd42b8ba"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:472","msg":"Snapshotting persistent volume","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","persistentVolume":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:42Z","volumeID":"restore-9c80eca0-9fe6-4c7b-941f-7fefcd42b8ba"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"pvc-00504355-3c35-11ea-bdc7-42010a8401fb","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"data-redis-boo-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"data-redis-boo-server-0","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"mysql","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"mysql","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"mysql","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:330","msg":"Executing custom action","name":"mysql","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"info","logSource":"pkg/backup/backup_pv_action.go:49","msg":"Executing PVCAction","pluginName":"velero","time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:169","msg":"Backing up item","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:171","msg":"Executing pre hooks","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:395","msg":"Executing takePVSnapshot","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:446","msg":"Got volume ID for persistent volume","name":"dummy-4325-mysql","namespace":"","persistentVolume":"dummy-4325-mysql","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z","volumeSnapshotLocation":"default"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:466","msg":"Getting volume information","name":"dummy-4325-mysql","namespace":"","persistentVolume":"dummy-4325-mysql","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z","volumeID":"restore-bf5a7c56-a726-40ee-98a5-8e4af9994119"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:472","msg":"Snapshotting persistent volume","name":"dummy-4325-mysql","namespace":"","persistentVolume":"dummy-4325-mysql","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:43Z","volumeID":"restore-bf5a7c56-a726-40ee-98a5-8e4af9994119"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"dummy-4325-mysql","namespace":"","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:311","msg":"Skipping action because it does not apply to this resource","name":"mysql","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"debug","logSource":"pkg/backup/item_backupper.go:247","msg":"Executing post hooks","name":"mysql","namespace":"dummy-4325","resource":{"Group":"","Resource":"persistentvolumeclaims"},"time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"persistentvolumes","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"persistentvolumes","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"serviceaccounts","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"serviceaccounts","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"limitranges","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"limitranges","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"replicationcontrollers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"replicationcontrollers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"namespaces","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"namespaces","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"events","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"events","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"resourcequotas","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"resourcequotas","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"configmaps","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"configmaps","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"endpoints","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"endpoints","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"nodes","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"nodes","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"services","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"services","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"podtemplates","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"podtemplates","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"secrets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"secrets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apiregistration.k8s.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apiregistration.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"apiservices","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apiregistration.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"apiservices","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"statefulsets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"statefulsets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"daemonsets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"daemonsets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"replicasets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"replicasets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"controllerrevisions","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"controllerrevisions","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"deployments","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apps/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"deployments","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"autoscaling/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"autoscaling/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"horizontalpodautoscalers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"autoscaling/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"horizontalpodautoscalers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"batch/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"batch/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"jobs","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"batch/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"jobs","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"batch/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"batch/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"cronjobs","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"batch/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"cronjobs","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"certificates.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"certificates.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"certificatesigningrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"certificates.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"certificatesigningrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.k8s.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"networkpolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"networkpolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"ingresses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"ingresses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"policy/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"policy/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"poddisruptionbudgets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"policy/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"poddisruptionbudgets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"policy/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"podsecuritypolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"policy/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"podsecuritypolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"clusterrolebindings","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"clusterrolebindings","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"clusterroles","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"clusterroles","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"rolebindings","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"rolebindings","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"roles","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"rbac.authorization.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"roles","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"volumeattachments","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"volumeattachments","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"storageclasses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"storageclasses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"csidrivers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"csidrivers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"csinodes","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"storage.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"csinodes","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"admissionregistration.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"admissionregistration.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"mutatingwebhookconfigurations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"admissionregistration.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"mutatingwebhookconfigurations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"admissionregistration.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"validatingwebhookconfigurations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"admissionregistration.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"validatingwebhookconfigurations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apiextensions.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apiextensions.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"customresourcedefinitions","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"apiextensions.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"customresourcedefinitions","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"scheduling.k8s.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"scheduling.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"priorityclasses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"scheduling.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"priorityclasses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"coordination.k8s.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"coordination.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"leases","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"coordination.k8s.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"leases","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"node.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"node.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"runtimeclasses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"node.k8s.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"runtimeclasses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"prometheuses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"prometheuses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"prometheusrules","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"prometheusrules","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"alertmanagers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"alertmanagers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"servicemonitors","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"servicemonitors","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"podmonitors","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"monitoring.coreos.com/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"podmonitors","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"downloadrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"downloadrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"restores","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"restores","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"resticrepositories","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"resticrepositories","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"podvolumebackups","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"podvolumebackups","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"deletebackuprequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"deletebackuprequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"serverstatusrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"serverstatusrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"backups","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"backups","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"volumesnapshotlocations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"volumesnapshotlocations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"backupstoragelocations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"backupstoragelocations","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"schedules","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"schedules","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"podvolumerestores","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"velero.io/v1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"podvolumerestores","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"internal.autoscaling.k8s.io/v1alpha1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"internal.autoscaling.k8s.io/v1alpha1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"capacityrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"internal.autoscaling.k8s.io/v1alpha1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"capacityrequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"nodemanagement.gke.io/v1alpha1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"nodemanagement.gke.io/v1alpha1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"updateinfos","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"nodemanagement.gke.io/v1alpha1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"updateinfos","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"scalingpolicy.kope.io/v1alpha1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"scalingpolicy.kope.io/v1alpha1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"scalingpolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"scalingpolicy.kope.io/v1alpha1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"scalingpolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"acme.cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"acme.cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"challenges","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"acme.cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"challenges","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"acme.cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"orders","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"acme.cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"orders","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"issuers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"issuers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"clusterissuers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"clusterissuers","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"certificaterequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"certificaterequests","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"certificates","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cert-manager.io/v1alpha2","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"certificates","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cloud.google.com/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cloud.google.com/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"backendconfigs","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"cloud.google.com/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"backendconfigs","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.gke.io/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.gke.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"managedcertificates","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"networking.gke.io/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"managedcertificates","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/group_backupper.go:101","msg":"Backing up group","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"replicasets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"replicasets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"deployments","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"deployments","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"networkpolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"networkpolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"podsecuritypolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:128","msg":"Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup","resource":"podsecuritypolicies","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"daemonsets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"daemonsets","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:106","msg":"Backing up resource","resource":"ingresses","time":"2020-01-21T13:24:44Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","group":"extensions/v1beta1","level":"info","logSource":"pkg/backup/resource_backupper.go:138","msg":"Skipping resource because it's excluded","resource":"ingresses","time":"2020-01-21T13:24:44Z"}
{"controller":"backup","level":"info","logSource":"pkg/controller/backup_controller.go:537","msg":"Backup completed","time":"2020-01-21T13:24:45Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin process exited","path":"/velero","pid":147,"time":"2020-01-21T13:24:45Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/velero","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin exited","time":"2020-01-21T13:24:45Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin process exited","path":"/plugins/velero-plugin-for-gcp","pid":156,"time":"2020-01-21T13:24:45Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","cmd":"/plugins/velero-plugin-for-gcp","level":"debug","logSource":"pkg/plugin/clientmgmt/logrus_adapter.go:74","msg":"plugin exited","time":"2020-01-21T13:24:45Z"}
{"controller":"backup","key":"backup/dummy-4325-2020-01-21-1579613072","level":"debug","logSource":"pkg/controller/backup_controller.go:246","msg":"Updating backup's final status","time":"2020-01-21T13:24:45Z"}
{"backup":"backup/dummy-4325-2020-01-21-1579613072","controller":"gc-controller","expiration":"2020-01-27T13:24:34Z","level":"debug","logSource":"pkg/controller/gc_controller.go:133","msg":"Backup has not expired yet, skipping","time":"2020-01-21T13:24:45Z"}
velero backup describe <backupname>orkubectl get backup/<backupname> -n velero -o yaml
Name: dummy-4325-2020-01-21-1579613072
Namespace: backup
Labels: app.kubernetes.io/component=data
app.kubernetes.io/namespace=dummy-4325
transient=true
velero.io/backup=dummy-4325-2020-01-21-1579613072
velero.io/pv=dummy-4325-mysql
velero.io/storage-location=default
Annotations: <none>
Phase: Completed
Namespaces:
Included: dummy-4325
Excluded: <none>
Resources:
Included: pvc, pv
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Snapshot PVs: auto
TTL: 144h0m0s
Hooks: <none>
Backup Format Version: 1
Started: 2020-01-21 15:24:34 +0200 EET
Completed: 2020-01-21 15:24:44 +0200 EET
Expiration: 2020-01-27 15:24:34 +0200 EET
Resource List:
v1/PersistentVolume:
- pvc-00504355-3c35-11ea-bdc7-42010a8401fb
- pvc-006fd481-3c35-11ea-bdc7-42010a8401fb
- pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb
- pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb
- pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb
- dummy-4325-mysql
v1/PersistentVolumeClaim:
- dummy-4325/data-dummy-green-0
- dummy-4325/data-fake-0
- dummy-4325/data-redis-dummy-green-server-0
- dummy-4325/data-redis-fake-green-master-0
- dummy-4325/data-redis-boo-server-0
- dummy-4325/mysql
Persistent Volumes:
pvc-ff4f5478-3c34-11ea-bdc7-42010a8401fb:
Snapshot ID: restore-dfe96ed9-9a09-4886-c9ce1306-c33d-45f5-b5af-9d9884595644
Type: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/diskTypes/pd-standard
Availability Zone: europe-west1-b
IOPS: <N/A>
pvc-4726adf2-3c35-11ea-bdc7-42010a8401fb:
Snapshot ID: restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3
Type: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/diskTypes/pd-standard
Availability Zone: europe-west1-b
IOPS: <N/A>
pvc-006fd481-3c35-11ea-bdc7-42010a8401fb:
Snapshot ID: restore-6acd21a8-56bc-49bc-0554d606-54b9-489c-9826-b8787eb3bb25
Type: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/diskTypes/pd-standard
Availability Zone: europe-west1-b
IOPS: <N/A>
pvc-ffa506b4-3c34-11ea-bdc7-42010a8401fb:
Snapshot ID: restore-7ecbed72-0fac-4be9-c16020c5-1f14-47fa-9016-9b65b2c8262c
Type: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/diskTypes/pd-standard
Availability Zone: europe-west1-b
IOPS: <N/A>
pvc-00504355-3c35-11ea-bdc7-42010a8401fb:
Snapshot ID: restore-9c80eca0-9fe6-4c7b-dc76b78f-d867-44c1-b204-7a276b302818
Type: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/diskTypes/pd-standard
Availability Zone: europe-west1-b
IOPS: <N/A>
dummy-4325-mysql:
Snapshot ID: restore-bf5a7c56-a726-40ee-c0d4e95d-f955-48ed-a618-5db32ccb1a44
Type: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/diskTypes/pd-ssd
Availability Zone: europe-west1-b
IOPS: <N/A>
Anything else you would like to add:
Deployment limits are the following:
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: "1"
memory: 1Gi
Pod memory usage is low:
GCP audit logs do not log the snapshots create events but only delete events which is why can not provide audit logs from the GCP snapshots API but have failed to replicate the same behaviour via gcloud compute disks shanshots command. With gcloud everything works as expected.
Environment:
- Velero version (use
velero version):v1.2.0 - Velero features (use
velero client config get features):none - Kubernetes version (use
kubectl version):v1.14.7 - Kubernetes installer & version:
v1.14.7-gke.23 - Cloud provider or hardware configuration:
GKE - OS (e.g. from
/etc/os-release):Darwin
dsuievalov
👍1
All 10 comments
hi @dsuievalov, I'm really not sure what could be going on here. From the logs and outputs you included, it sure looks like things are working as expected from the velero side.
It might be interesting to run gcloud compute snapshots list in a watch while the backup is executing, to see if those snapshots ever appear via gcloud. Perhaps that can give us some more information to go on.
skriss
on 24 Jan 2020
Hey @dsuievalov, has your issue been resolved?
carlisia
on 5 Feb 2020
Hey @dsuievalov, has your issue been resolved?
Hello, unfortunately no. Sorry, forget to respond in here after testing of the gcloud compute snapshots list during the back procedure. Eventually gcloud does not tell anything new and there are no errors about missing snapshots.
I would like to see the full responce from GCP side for each velero snapshot creation call. If GCP responce with successful creation - velero working as expected and it is a problem of GCP (some limitation maybe). But I am not very good with a golang so will appreciate if someone can point me to the correct way of changing the source code to archive that goal :) I will build the custom binary just for the test.
dsuievalov
on 6 Feb 2020
Was any issue logged with Google Support about this? Given that it may well be a problem on the Google side.
gorzek
on 10 Feb 2020
Was any issue logged with Google Support about this? Given that it may well be a problem on the Google side.
Did not contact Google Support since the issue happens only when using velero - direct calls via gcloud always report with snapshots creation/failure (stress testing it with ~200 disks (>50Gb) snapshots creation per hour). Which is why would like to debug it from velero side.
dsuievalov
on 11 Feb 2020
@dsuievalov I built a custom image with some additional logging that you could try out and see if you get any more info.
The branch with the code change is here: https://github.com/skriss/velero-plugin-for-gcp/commit/961d5025f088bc62b3001c8ae43d3e2addf6e6eb
The image you can use is steveheptio/velero-plugin-for-gcp:snapshot-logging
If you have velero up and running, you can swap this plugin image in by doing:
velero plugin remove velero/velero-plugin-for-gcp:v1.0.0
velero plugin add steveheptio/velero-plugin-for-gcp:snapshot-logging
Please try this out and let me know what you find in the logs! Hopefully the "Got response from CreateSnapshot call" log lines will provide some more info.
skriss
on 19 Feb 2020
@dsuievalov were you able to test this out?
skriss
on 21 Feb 2020
@skriss sorry, I was not able to test it this week. I will do that on monday.
dsuievalov
on 23 Feb 2020
👍1
@skriss Thank you very much for the extra debugging message, it helps to find the actual issue. Here is a velero log from gcloud for failed snapshot:
{"backup":"backup/test-build-5396","cmd":"/plugins/velero-plugin-for-gcp","createSnapshotResponse":{"id":"1593028888651089567","insertTime":"2020-02-26T05:53:52.115-08:00","kind":"compute#operation","name":"operation-1582725231622-59f7aef353a2f-6746de61-a55034dc","operationType":"createSnapshot","selfLink":"https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/operations/operation-1582725231622-59f7aef353a2f-6746de61-a55034dc","startTime":"2020-02-26T05:53:52.117-08:00","status":"RUNNING","targetId":"1125778073669331382","targetLink":"https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/disks/restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3","user":"[email protected]","zone":"https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b"},"level":"info","logSource":"/go/src/github.com/vmware-tanzu/velero-plugin-for-gcp/velero-plugin-for-gcp/volume_snapshotter.go:245","msg":"Got response from CreateSnapshot call","pluginName":"velero-plugin-for-gcp","time":"2020-02-26T13:53:52Z"}
And the detail information about snapshot creation operation:
$ gcloud compute operations describe https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/operations/operation-1582725231622-59f7aef353a2f-6746de61-a55034dc
endTime: '2020-02-26T05:54:01.343-08:00'
error:
errors:
- code: INVALID_USAGE
message: Disk attachment changed while trying to make a snapshot.
httpErrorMessage: BAD REQUEST
httpErrorStatusCode: 400
id: '1593028888651089567'
insertTime: '2020-02-26T05:53:52.115-08:00'
kind: compute#operation
name: operation-1582725231622-59f7aef353a2f-6746de61-a55034dc
operationType: createSnapshot
progress: 100
selfLink: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/operations/operation-1582725231622-59f7aef353a2f-6746de61-a55034dc
startTime: '2020-02-26T05:53:52.117-08:00'
status: DONE
targetId: '1125778073669331382'
targetLink: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b/disks/restore-a041c896-cd27-4542-b541bc85-f508-4990-b4f9-b9d76c3901a3
user: [email protected]
zone: https://www.googleapis.com/compute/v1/projects/dummy-project/zones/europe-west1-b
So the snapshot was not created because of gcloud internal issue. Since I have scaled all the pods in the namespace (and wait until all of them will be terminated) before velero backup call it looks during the snapshotting gcloud disk was detached and its information changed. But as I mentioned - it is not gcloud issue.
I think if @boxcee finishes with PR https://github.com/vmware-tanzu/velero/pull/2297 and velero backup with --wait argument wait until the snapshot creates successfully or alert in case of failures (like this one, for example) this issue should not happen again.
馃挴 kudos to you guys: @skriss / @boxcee
dsuievalov
on 26 Feb 2020
👍1
Closing because will be covered by #3533
eleanor-millman
on 10 May 2021
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
nrb
路
4Comments
my1990
路
3Comments
Marki4711
路
3Comments
debianmaster
路
3Comments
abh
路
4Comments