Velero: Velero is not backing up volume using Azure & Restic

[michael-niemand]

What steps did you take and what happened:

• install velero using helm or velero install command, activate restic, use storage account access key authentication method
• install azure plugin version v1.1.0-beta.1
• annotate pod in question with backup.velero.io/backup-volumes: volume (Volume is in fact named "volume")
• run backup for particular namespace (schedule or one time)

The objects get backed up, but the volume is not. Phase of the backup is PartiallyFailed. I enabled debug output on Velero, which gave me a restic error message (not sure if related):
"Error checking repository for stale locks" controller=restic-repository error="error getting azure restic env vars: unable to get all required environment variables: the following keys do not have values: AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, A
Unfortunately, I can't use these values, I have to use the storage account credentials - which is supposed to be ok as long as I don't want to take Azure volume snapshots (I set snapshotsEnabled: false in helm).

What did you expect to happen:
Resources _and_ volume gets backed up

The output of the following commands will help us better understand what's going on:
output of various debugging commands:
https://gist.github.com/michael-niemand/d09e850732c9187f681e120a42ca7ee8

Environment:

• Velero version:

Client:
        Version: v1.3.2
        Git commit: 55a9914a3e4719fb1578529c45430a8c11c28145
Server:
        Version: v1.3.1

• Velero features:
  features: <NOT SET>
  Azure Plugin v1.1.0-beta.1 being used
• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Azure Container Engine, not AKS
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[skriss]

@michael-niemand you also need to use the Velero v1.4.0-beta.1 container image to be able to use static storage account keys with restic. You can run the following to update your container image tags:

kubectl -n velero set image deploy/velero velero=velero/velero:v1.4.0-beta.1
kubectl -n velero set image daemonset/restic restic=velero/velero:v1.4.0-beta.1

Please try that out (assuming you're OK running a beta release in your environment) and let us know how it goes!

[michael-niemand]

Thanks for the fast reply! It is highly appreciated. I updated the image versions as you said and it beautifully backs up all the resources - but again, no volume. Log here. https://gist.github.com/michael-niemand/395c50990bcd3990b4fd30e4fe84d507
I can't even see velero trying to back up a volume in the logs.
When I check the files uploaded to Azure, I see a "message": "volume was empty so no snapshot was taken", (I can provide the full json if necessary)
So I double checked the PVC and pod. It's definitely the PVC mounted by the pod in question and is definitely not empty. What am I missing?

[michael-niemand]

@skriss you tagged this with "Needs info" - is there something I can provide to help?

[skriss]

Could you provide:

• the YAML for the PV & PVC which are failing to back up?
• the YAML for the PodVolumeBackup created by the velero backup? (kubectl -n velero get podvolumebackups -l velero.io/backup-name=<VELERO-BACKUP-NAME> -o yaml should do it)

The other thing that would be helpful, if possible, is to SSH onto the node where the relevant pod is running, go to /var/lib/kubelet/pods/<pod-UID>/volumes/..., find the subdirectory for the relevant volume, and include here (a) the full path of that subdirectory, and (b) the results of ls -la in that subdirectory (or just an indication of if the expected files are there).

[michael-niemand]

Hi @skriss - sorry this took so long:

https://gist.github.com/michael-niemand/c04caa0858cb35611afb995e8258a7cf

unfortunately I can't ssh into the node myself, but I will try to get someone from Ops to do it.

[skriss]

@michael-niemand I see that you're using Azure File PVs. If you search on https://velero.io/docs/v1.4/restic/ for "Microsoft Azure", you'll find some info on how to set up your storage class properly to work with restic. If you haven't done that already, could you try it and let us know if that solves the problem?

[michael-niemand]

Highly appreciate your effort! We're currently in te process of moving from the old Container Engine to AKS, but I will soon start implementing the backup again so I can get back to you with a report if the "nouser_xattr" does the trick! Thanks again.

[michael-niemand]

was finally able to test patching the storage class and backup seems to work now, at least according to the size of the backup, since the restore doesnt work yet

But this one is a /close. Thanks for helping out a noob. Appreciate it.