Velero: Removing backups with kubectl does not trigger velero-server backup removal

[invidian]

What steps did you take and what happened:

• create full backup with velero backup create test
• wait for backup to complete
• remove backup using kubectl: kubectl delete backups test -n velero

What did you expect to happen:

• Azure snapshot and data in blob storage should be removed

The output of the following commands will help us better understand what's going on:

Velero server does not produce any logs on delete operation

Anything else you would like to add:
Creating subsequent backups fails with following message:

time="2019-07-31T12:55:48Z" level=error msg="backup failed" controller=backup error="backup already exists in object storage" error.file="/go/src/github.com/heptio/velero/pkg/controller/backup_controller.go:484" error.function="github.com/heptio/velero/pkg/controller.(*backupController).runBackup" key=velero/test logSource="pkg/controller/backup_controller.go:230"

Snapshots are still visible in Azure, so manual cleanup is required.

I also wasn't able to find any information in the docs, that using kubectl for taking backups is not recommended.

Environment:

• Velero version (use velero version):

Client:
    Version: v1.0.0
    Git commit: 72f5cadc3a865019ab9dc043d4952c9bfd5f2ecb
Server:
    Version: v1.0.0
``
- Kubernetes version (use `kubectl version`):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.7", GitCommit:"4683545293d792934a7a7e12f2cc47d20b2dd01b", GitTreeState:"clean", BuildDate:"2019-06-06T01:39:30Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
```

• Kubernetes installer & version: AKS
• Cloud provider or hardware configuration: Azure

[skriss]

@invidian yes - if you want to delete the backup including all data in object/block storage, use velero backup delete. As you said, using kubectl delete will only delete the custom resource.

I agree this is probably not documented well, and we should probably add something to explain the hows and whys.

[invidian]

@invidian yes - if you want to delete the backup including all data in object/block storage, use velero backup delete. As you said, using kubectl delete will only delete the custom resource.

Thanks for confirmation @skriss. And do you know if there are any implications from creating backups, schedules etc using kubectl?

[skriss]

Creates via kubectl are fine. You can also velero backup create ... -o yaml to generate a YAML spec for a backup, which you can later apply using kubectl.

[turkenh]

What I understand from the source code, deletions are triggered via another crd called DeleteBackupRequest.

So, if you want to use kubectl (or go client) to delete a backup, you need to create DeleteBackupRequest instead of deleting Backup resource.

[framled]

there are no documentation about those resources.

[eleanor-millman]

@a-mccarthy Changing this to be a documentation issue.