Velero: Velero backup on AWS giving errors

Created on 19 Jul 2019 · 5Comments · Source: vmware-tanzu/velero

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
Creating Velero backup on AWS fails with Error: PartiallyFailed

Installed Velero 1.0.0 using the guide: https://velero.io/docs/v1.0.0/aws-config/

Velero Installation command
helm upgrade --install velero stable/velero \
--namespace kube-addons \
--set image.tag=v1.0.0 \
--set rbac.create=true \
--set configuration.provider=aws \
--set configuration.backupStorageLocation.name=aws \
--set configuration.backupStorageLocation.bucket=obscured \
--set configuration.backupStorageLocation.config.region=us-west-2 \
--set snapshotsEnabled=false \
--set credentials.useSecret=false \
--values kube2iam.yaml \
--dry-run=false --debug
Generated manifest
https://gist.github.com/rajakshay/7b0e93d5742c9a21278661dc1d2d008c
Tried creating a backup

velero backup create test-backup
Backup request "test-backup" submitted successfully.
Run `velero backup describe test-backup` or `velero backup logs test-backup` for more details.

Then, ran velero backup describe test-backup

Name:         test-backup
Namespace:    kube-addons
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  PartiallyFailed (run `velero backup logs test-backup` for more information)

Errors:    1
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2019-07-19 13:44:06 -0700 PDT
Completed:  2019-07-19 13:44:14 -0700 PDT

Expiration:  2019-08-18 13:44:06 -0700 PDT

Persistent Volumes: <none included>

What did you expect to happen:
Backup should have completed without an error.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

kubectl logs deployment/velero -n kube-addons
https://gist.github.com/rajakshay/5a320dad498c1fad90ed83073a84f3e4
velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

Name:         test-backup
Namespace:    kube-addons
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  PartiallyFailed (run `velero backup logs test-backup` for more information)

Errors:    1
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2019-07-19 13:44:06 -0700 PDT
Completed:  2019-07-19 13:44:14 -0700 PDT

Expiration:  2019-08-18 13:44:06 -0700 PDT

Persistent Volumes: <none included>

velero backup logs <backupname>
https://gist.github.com/rajakshay/eda6b04f54b6baa1ddfc067af5a26f0f

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Velero pod logs
https://gist.github.com/rajakshay/4fe13a7abe64944ad1adc15d52d54e09
Grepping error log from velero pod
kubectl logs -n kube-addons <velero-pod-name> | grep error

time="2019-07-19T20:16:06Z" level=error msg="reading plugin stderr" cmd=/velero controller=backup-sync error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:89" pluginName=velero
time="2019-07-19T20:17:06Z" level=error msg="reading plugin stderr" cmd=/velero controller=backup-sync error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:89" pluginName=velero
time="2019-07-19T20:20:06Z" level=error msg="reading plugin stderr" cmd=/velero controller=backup-sync error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:89" pluginName=velero
time="2019-07-19T20:39:10Z" level=error msg="reading plugin stderr" cmd=/velero controller=backup-sync error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/logrus_adapter.go:89" pluginName=velero

Grepping error log from backup logs:
velero backup logs test-backup | grep error

time="2019-07-19T20:44:07Z" level=error msg="Error getting volume snapshotter for volume snapshot location" backup=kube-addons/test-backup error="unable to locate VolumeSnapshotter plugin named velero.io/" group=v1 logSource="pkg/backup/item_backupper.go:410" name=pvc-12dca563-8331-11e9-9822-06e7522f5c62 namespace=kube-addons persistentVolume=pvc-12dca563-8331-11e9-9822-06e7522f5c62 resource=pods volumeSnapshotLocation=default

CRDs that got created (Please ignore the timestamps since I updated the ticket later with one of the terminal outputs. But, I have validated that these were the only CRDs existing on the cluster)
kubectl get crd -n kube-addons

NAME                                                  CREATED AT
backups.velero.io                                     2019-07-19T23:18:22Z
backupstoragelocations.velero.io                      2019-07-19T23:18:22Z
deletebackuprequests.velero.io                        2019-07-19T23:18:22Z
downloadrequests.velero.io                            2019-07-19T23:18:22Z
podvolumebackups.velero.io                            2019-07-19T23:18:22Z
podvolumerestores.velero.io                           2019-07-19T23:18:22Z
resticrepositories.velero.io                          2019-07-19T23:18:22Z
restores.velero.io                                    2019-07-19T23:18:22Z
schedules.velero.io                                   2019-07-19T23:18:22Z
serverstatusrequests.velero.io                        2019-07-19T23:18:22Z
volumesnapshotlocations.velero.io                     2019-07-19T23:18:22Z

Note I am deploying in custom namespace. But, even when I deployed to namespace velero, the result is no different.

Environment:

Velero version (use velero version):

Client:
    Version: v1.0.0
    Git commit: -
Server:
    Version: v1.0.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-19T22:12:47Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes installer & version:
kops & KOPS_BINARY_VERSION=1.14.0-alpha.1
Helm Version( use helm version)

Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.1", GitCommit:"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e", GitTreeState:"clean"}

Cloud provider or hardware configuration:
AWS. EC2/r4.large
OS (e.g. from /etc/os-release):

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"`

OS (from uname -a)
Linux ip-**obscured** 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

AreClouAWS Question

Source

rajakshay

All 5 comments

can you provide the output of velero snapshot-location get -o yaml?

skriss on 22 Jul 2019

🎉1

It looks like you're missing configuration for the volume snapshot location -- see the Values.yaml file around https://github.com/helm/charts/blob/master/stable/velero/values.yaml#L87

skriss on 22 Jul 2019

It looks like you're missing configuration for the volume snapshot location -- see the Values.yaml file around https://github.com/helm/charts/blob/master/stable/velero/values.yaml#L87

Thank you @skriss for helping me debug this issue.

For anyone following this issue, here was my problem:
I had a cached version velero-2.0.3 of this chart present on my local machine, and the feature that I was looking for (disabling PV snapshots) was included in the chart version 2.1.1 https://github.com/helm/charts/pull/15259/files

I never realized that I was using a cached copy until after your comment.
Since, I am setting snapshotsEnabled=false, I should not need a volume snapshot location configuration.

The only weird thing that I notice is that the CRD volumesnapshotlocations still gets created. Which should not be the case due to this line: https://github.com/helm/charts/blob/master/stable/velero/templates/volumesnapshotlocation.yaml#L1
Or at least the intention was to not create it (if not needed) as per the explanation on the chart README, and here: https://github.com/helm/charts/blob/master/stable/velero/values.yaml#L152

My understanding about "how this should work" is probably incorrect here though, or I am missing the idea behind it.

kubectl get crd -n kube-addons | grep velero

backups.velero.io                                     2019-07-22T21:07:57Z
backupstoragelocations.velero.io                      2019-07-22T21:07:57Z
deletebackuprequests.velero.io                        2019-07-22T21:07:57Z
downloadrequests.velero.io                            2019-07-22T21:07:57Z
podvolumebackups.velero.io                            2019-07-22T21:07:57Z
podvolumerestores.velero.io                           2019-07-22T21:07:57Z
resticrepositories.velero.io                          2019-07-22T21:07:57Z
restores.velero.io                                    2019-07-22T21:07:57Z
schedules.velero.io                                   2019-07-22T21:07:57Z
serverstatusrequests.velero.io                        2019-07-22T21:07:57Z
volumesnapshotlocations.velero.io                     2019-07-22T21:07:57Z

Here is my working output for creating backups:

velero backup describe backup99
Name:         backup99
Namespace:    kube-addons
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2019-07-22 13:53:42 -0700 PDT
Completed:  2019-07-22 13:53:50 -0700 PDT

Expiration:  2019-08-21 13:53:42 -0700 PDT

rajakshay on 22 Jul 2019

🎉1

can you provide the output of velero snapshot-location get -o yaml?

apiVersion: velero.io/v1
items: []
kind: VolumeSnapshotLocationList
metadata:
  resourceVersion: "13464800"
  selfLink: /apis/velero.io/v1/namespaces/kube-addons/volumesnapshotlocations

rajakshay on 22 Jul 2019

OK, glad you got this sorted out. If there's still an issue with the Helm chart, please file a bug against that repo. Thanks!

skriss on 22 Jul 2019

Was this page helpful?

0 / 5 - 0 ratings